A retrieval system for a large language model of legal documents for bidding

Abstract

The application relates to the field of text processing, and particularly relates to a retrieval system for a large language model of bidding and tendering legal documents. The retrieval system comprises a document preprocessing module, a semantic chunking module, an entity and relationship identification module, a knowledge graph construction module, a multi-strategy hybrid retrieval module and a retrieval result optimization module. In operation, the bidding and tendering legal documents are preprocessed, and are cut based on the number of words in the documents by adopting a chapter-level chunking or a clause-level chunking strategy. The LateChunking algorithm is used to determine the cutting points and extract semantic units. Entity elements are identified and a knowledge graph is constructed. The optimal retrieval result is obtained through multi-strategy hybrid retrieval and optimization processing. The limitations of traditional single retrieval methods in the processing of long bidding and tendering legal documents are overcome, the accuracy and context coherence of the clause retrieval are improved, the risk of hallucination of the large language model due to information loss or misunderstanding is greatly reduced, and the reliability and practicality of the legal intelligent question and answer result are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of text processing, and in particular to a retrieval system for a large language model of bidding and tender legal documents. BACKGROUND

[0002] Large language models commonly exhibit hallucination phenomena in text generation and question-answering tasks, i.e., the model may output seemingly reasonable but actually inconsistent with the facts or input information. This problem is particularly prominent in the legal field, which is highly professional. Since legal texts have high rigor, standardization, and logical dependency, if the model does not fully understand the semantic association and effectiveness hierarchy between provisions, it is easy to generate incorrect or misleading responses. Bidding and tender legal documents, as a specific branch of legal texts, not only have the general characteristics of legal language, but also exhibit significant domain-specificity. Such documents typically have a strict hierarchical structure, including chapter division, clause numbering, sub-item nesting, and a large number of provision reference relationships, and cover multiple business dimensions such as bidding procedures, tender requirements, legal responsibilities, etc. The text is long, dense in terminology, and complex in logical association.

[0003] Traditional RAG (Retrieval Augmented Generation) systems mostly use a single retrieval channel (such as vector retrieval), which is difficult to cover different types of knowledge. In complex scenarios, dynamic retrieval routing has become a research hotspot. Recently, some research has proposed introducing an intelligent router into the RAG framework, dynamically switching retrieval tools or data sources according to the query, known as Agentic RAG Router. For example, the SAGE framework uses an intelligent routing unit to adaptively switch between search engine retrieval and internal knowledge base retrieval, achieving dynamic adjustment of early routing, mid-route, and late routing. This multi-routing strategy can select different retrieval strategies based on query intent, significantly improving the coverage and accuracy of recall. On the other hand, how to evaluate the reliability of retrieval results is a weak link in existing systems. Currently, it usually relies on vector matching scores or cross-encoding reordering scores, but lacks intuitive confidence output. In 2024, some researchers proposed using Bayesian inference to verify the quality of RAG retrieval fragments: by estimating the relevance likelihood of the paragraph to the question by LLM, and combining the prior probability (such as the position of the paragraph in the document) to calculate the posterior confidence, to determine whether to include the context. This method proves the value of the Bayesian probability framework in improving the quality of retrieval content. Currently, there is no publicly available adjustable Bayesian dynamic routing scheme - i.e., dynamically allocating weights or paths for different retrieval methods (such as keyword retrieval, vector retrieval, knowledge graph query, etc.), and outputting a statistically significant confidence interval, which not only limits the retrieval effect, but further exacerbates the risk of hallucination by large language models due to information gaps or misinterpretation. SUMMARY

[0004] To solve the above problems, the application provides a retrieval system for a large language model of bidding legal documents, comprising:

[0005] A document preprocessing module is configured to clean and extract structural identifiers from the bidding legal documents;

[0006] A semantic chunking module is configured to cut the preprocessed documents based on the number of document words using a chapter-level chunking or clause-level chunking strategy, and determine the cutting points using a LateChunking algorithm during cutting, and output the semantic units obtained by cutting;

[0007] An entity and relationship recognition module is configured to identify entity elements from the semantic units, and label the entity elements with effectiveness level labels and business domain labels, and output the entity elements and label data, wherein the entity elements include regulation names;

[0008] A knowledge graph construction module is configured to take the regulation names as root nodes, and construct a bidding field knowledge graph based on the reference, cause and supplement relationships between entities;

[0009] A multi-strategy hybrid retrieval module is configured to receive user queries, extract legal entity keywords in the queries, and simultaneously perform semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, and keyword-based matching retrieval, and output multi-path recall results;

[0010] A retrieval result optimization module is configured to perform semantic similarity calculation and business label weighted sorting on the multi-path recall results, and select the optimal candidate answer set for generating results.

[0011] Specifically, when the document preprocessing module cleans and extracts structural identifiers from the bidding legal documents, it removes irrelevant characters in the header and footer, and marks the starting position and hierarchical relationship of the content in the title, chapter number, clause number, sub-clause, item, and / or preset important clause chapter library.

[0012] Specifically, when the semantic chunking module cuts the preprocessed documents based on the number of document words using a chapter-level chunking or clause-level chunking strategy, when the number of document words is less than 500 characters, the first cutting point is dynamically determined based on the title and chapter number;

[0013] When the number of document words is not less than 500 characters, the second cutting point is dynamically determined based on the clause number, and when the number of clause words exceeds 300 characters, further subdivision cutting is performed according to the sub-clause or the item.

[0014] Specifically, the semantic chunking module, when determining the cutting points using the LateChunking algorithm, combines the sentence boundary punctuation and semantic coherence analysis, and limits the cutting points at the period, semicolon or clause number by identifying the clause number sequence, title level and the pattern in the preset important clause chapter library.

[0015] Specifically, the entity and relationship identification module extracts entity elements including the publishing agency, the publishing and effective dates and the applicable scope.

[0016] The entity and relationship identification module, when identifying the entity elements from the semantic units, maps the text into a vector representation using an embedding model, and then inputs the vector into a large language model for entity and relationship extraction.

[0017] The large language model is fine-tuned through a reinforcement learning strategy, and during fine-tuning, the reward function is constructed based on the entity recognition accuracy, the relationship extraction accuracy or the matching degree with the existing entities in the knowledge graph.

[0018] Specifically, the effectiveness level label includes national bidding and tendering laws, departmental rules and local regulations, and the business field label includes comprehensive, engineering, finance and administrative management.

[0019] The entity and relationship identification module automatically classifies the entity elements based on the publishing agency and content theme of the bidding and tendering legal documents when labeling the effectiveness level label and the business field label.

[0020] Specifically, the knowledge graph construction module, when constructing the bidding and tendering field knowledge graph, takes the regulation name as the root node, and constructs the graph edge by identifying the reference relationship, causal relationship and supplementary explanation relationship between clauses.

[0021] Specifically, the multi-strategy hybrid retrieval module further includes a routing decision unit and a result reliability evaluation module. The routing decision unit is used to receive the vector representation of the user query, and dynamically select at least one retrieval path based on a pre-trained Bayesian decision network. The retrieval path includes vector similarity retrieval, keyword retrieval, knowledge graph query.

[0022] The Bayesian decision network outputs a path selection probability distribution by analyzing the relationship between query semantic features and historical performance data of each retrieval path.

[0023] The result reliability evaluation module is used to fuse the path selection probability distribution output by the Bayesian decision network and the quality score of the retrieval results returned by the path, calculate the overall confidence of the final answer through Bayesian inference, and output a quantitative confidence interval value or confidence level to represent the reliability of the final answer.

[0024] Specifically, when the retrieval result selection module selects the optimal candidate answer set, the posterior probability of the retrieval result satisfying the confidence interval requirement or the confidence level requirement is calculated. In calculating the posterior probability, the prior probability is determined based on the clause effectiveness level and the publication time. The effectiveness level includes national bidding and tendering laws, departmental regulations and local regulations. The prior probability weight of the national law is higher than that of the local regulations. The likelihood function is constructed based on the semantic similarity score. The top N results with the highest posterior probability are selected as the candidate answer set, and N is a preset constant.

[0025] Specifically, the retrieval system further comprises an illusion discrimination module for evaluating the credibility of the generated results according to a pre-trained fact discrimination model. The discrimination model outputs a confidence score by comparing the semantic consistency and factual consistency of the generated text with the real clauses in the knowledge base. Results below the threshold are filtered or marked as pending review.

[0026] The present application has the following technical effects:

[0027] Through multi-strategy hybrid retrieval and structured semantic processing, the limitations of traditional single retrieval method in long document processing of bidding and tendering laws are effectively overcome, and the accuracy and context coherence of clause retrieval are significantly improved, thereby greatly reducing the risk of hallucination of large language models due to information missing or misunderstanding, and enhancing the credibility and practicality of legal intelligent question and answer results. BRIEF DESCRIPTION OF DRAWINGS

[0028] The above and other objects, features and advantages of the example embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which several embodiments of the present application are shown by way of example, and in which the same or corresponding elements are referred to by the same or corresponding reference numerals. The present application will be described more fully hereinafter with reference to the accompanying drawings, in which several embodiments of the present application are shown by way of example, and in which the same or corresponding elements are referred to by the same or corresponding reference numerals.

[0029] Figure 1 is a schematic diagram of an architecture of a retrieval system for a large language model of bidding and tendering legal documents in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

[0031] The present application aims to solve the key problems in the intelligent retrieval and processing of bidding legal documents. In the prior art, the retrieval system based on large language models often faces the illusion phenomenon, that is, the generated content of the model deviates from the actual legal provisions, which is mainly due to the fact that the particularity of bidding legal texts has not been fully considered. Such documents have the characteristics of high structuring, frequent cross-referencing between clauses, strict terminology, and clear hierarchy of effectiveness. Traditional retrieval methods rely on a single strategy, such as vector similarity or keyword matching, which is difficult to capture deep semantic associations and logical structures. More notably, long documents are often mechanically chunked, disrupting the context continuity of the provisions, which prevents the model from fully understanding the legislative intent and the relationship between clauses, leading to misreading or information loss and exacerbating the illusion risk. This defect is particularly prominent in the bidding field, as the documents often involve a large number of cross-references, supplementary relationships, and differences in effectiveness levels, making it clear that a single retrieval strategy is insufficient to meet the complex query requirements.

[0032] Based on the above defects, the core motivation of the present application is to build a system that can deeply integrate the structural characteristics of bidding legal documents, support multi-dimensional retrieval and result optimization, and improve retrieval accuracy and suppress the illusion through collaborative multi-module processing. The following example illustrates a retrieval system for large language models of bidding legal documents, as shown in Figure 1

[0033] The document preprocessing module is used to clean and structure the bidding legal documents;

[0034] The semantic chunking module is used to cut the preprocessed documents based on the number of document words using chapter-level chunking or clause-level chunking strategies, and uses the LateChunking algorithm to determine the cutting points during cutting, and outputs the semantic units obtained by cutting;

[0035] The entity and relationship recognition module is used to identify entity elements from the semantic units and label the entity elements with effectiveness level labels and business domain labels, and output the entity elements and label data, including the names of regulations;

[0036] The knowledge graph construction module is used to construct a bidding domain knowledge graph based on the names of regulations as root nodes and the reference, causal, and supplementary relationships between entities;

[0037] The multi-strategy hybrid retrieval module is used to receive user queries, extract legal entity keywords in the queries, and simultaneously perform semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, and keyword-based matching retrieval, and output multi-path recall results;

[0038] ​The retrieval result optimization module is used for semantic similarity calculation and business label weighted sorting on the multi-channel retrieval results, and selects the optimal candidate answer set for generating results.

[0039] The retrieval system of the present application works as follows. First, the system cleans and extracts the structure of the original bidding legal documents through the document preprocessing module. The original documents often contain irrelevant characters such as headers and footers, and complex layout elements, which need to be removed to purify the text. At the same time, the module will identify and mark the starting position and hierarchical relationship of the contents in the title, chapter number, clause number, sub-clause, sub-item and other preset important clause section library. This process not only improves the text quality, but also provides key metadata for subsequent processing through structure extraction.

[0040] Subsequently, the semantic chunking module dynamically selects a chunking strategy based on the number of words in the preprocessed document. When the number of words in the document is small, for example, less than five hundred characters, a chapter-level chunking strategy is adopted, relying on the extracted title and chapter number to dynamically determine the cutting point to ensure content integrity. For longer documents, such as more than five hundred characters, switch to a clause-level chunking strategy, and divide according to the clause number. When the number of characters in a single clause exceeds three hundred characters, further subdivide according to the sub-clause or sub-item content to avoid information overload. During the chunking process, the system uses the LateChunking algorithm to determine the cutting point. This algorithm combines the analysis of sentence boundary punctuation symbols such as periods, semicolons and semantic coherence, while identifying clause number sequences, title levels and patterns in the preset important clause section library, ensuring that the cutting point is always located at the semantic boundary, thereby maintaining the coherence of the context of the provisions.

[0041] Next, the entity and relationship identification module extracts key entity elements from the semantic units obtained by the segmentation, including the name of the regulation, the issuing agency, the issuance and effective dates, and the scope of application. Unlike traditional static methods based on BERT fine-tuning, the system uses a joint optimization framework that combines embedding representation, large language models (LLM), and reinforcement learning for entity recognition and relationship extraction. Specifically, first, a domain-adapted embedding model is used to map the text semantic units into high-dimensional vector representations. Then, it is input into the LLM for deep semantic analysis and structured extraction to identify entities and their relationships. To further improve extraction quality and domain adaptability, a reinforcement learning mechanism is introduced to fine-tune the LLM. By designing reward functions, such as entity recognition accuracy, relationship extraction accuracy, and consistency with existing knowledge graphs, the model optimization strategy is guided to effectively reduce noise labeling interference and improve the understanding of complex semantics and logical structures in bidding legal texts. After recognition, the module automatically labels the entity elements with effectiveness level tags, such as national laws, departmental regulations, and local regulations, and business domain tags, such as comprehensive, engineering, finance, and administrative management. The label classification is based on the document issuing agency, content theme, and context semantics, providing structured support for subsequent knowledge graph construction and multi-strategy retrieval.

[0042] The knowledge graph construction module takes the regulation name as the root node and dynamically constructs the bidding domain knowledge graph based on the relationships between entities extracted from the semantic units, such as reference, causality, and supplementation. By analyzing explicit references between clauses (such as "see Article X") and implicit logical associations (such as causality and supplementary explanations), the edge relationships in the graph are constructed, forming a structured knowledge network with semantic connectivity. For example, the reference relationship between the "Regulations on the Implementation of the Bidding Law" and the "Bidding Law" will be extracted and represented as an edge in the graph, visually displaying the hierarchical structure and associated paths between legal provisions. Based on this, the system further introduces a Bayesian dynamic routing mechanism to achieve adaptive fusion and weight distribution of multi-source retrieval strategies, including vector retrieval, knowledge graph association retrieval, and keyword matching. This mechanism dynamically adjusts the contribution weights of each retrieval channel based on historical retrieval accuracy and recall rate, and calculates the confidence interval of the retrieval results based on the Bayesian inference framework, thereby achieving reliable fusion and ranking optimization of multi-path recall results, improving the coverage and answer reliability of the system under complex queries, and providing reliable structured knowledge support for subsequent multi-strategy retrieval.

[0043] The multi-strategy hybrid retrieval module is responsible for receiving user queries and extracting legal entity keywords in the queries. The module includes a routing decision unit and a result reliability evaluation module. The routing decision unit dynamically selects a retrieval path based on a pre-trained Bayesian decision network that analyzes the relationship between query semantic features and historical performance data of each retrieval path to output a path selection probability distribution to determine at least one retrieval strategy, such as semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, or keyword-based matching retrieval. The module synchronously executes the selected retrieval path: semantic retrieval calculates similarity by mapping queries and document blocks to vector space, association retrieval uses knowledge graphs to explore entity relationships to expand the retrieval scope, and keyword retrieval ensures accurate matching. The result reliability evaluation module fuses the path selection probability distribution and the quality score of the retrieval results, calculates the overall confidence of the final answer through Bayesian inference, and outputs the confidence interval or confidence level to represent reliability.

[0044] The retrieval result optimization module integrates and sorts multiple recall results. The module calculates the semantic similarity score of each result and performs weighted sorting in combination with business labels and effectiveness levels. Specifically, the Bayesian inference framework is used to calculate the posterior probability, where the prior probability is determined based on the clause effectiveness level and the release time, and the likelihood function is constructed based on the semantic similarity score. This sorting strategy ensures that the results are not only relevant, but also have high effectiveness and timeliness. Finally, the module selects the top N results with the highest posterior probability as the candidate answer set for generating the final output.

[0045] In addition, the system also includes an illusion discrimination module for reliability evaluation of the generated results. Based on a pre-trained fact discrimination model, the module compares the semantic consistency and factual consistency of the generated text with the real clauses in the knowledge base, and outputs a confidence score. Results below the threshold value will be filtered or marked as pending review, further reducing the illusion risk.

[0046] Obviously, the above-described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] It should be understood that when the claims, the specification, and the drawings of the present application use the terms "first", "second", etc., they are only used to distinguish different objects, and are not used to describe a specific order. The terms "include" and "contain" used in the specification and claims of the present application indicate the presence of the described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

Claims

1. A search system for a tendering legal document large language model, characterized by, The method comprises the following steps: a document preprocessing module for cleaning and structure identification extraction of bidding legal documents; a semantic segmentation module for cutting the preprocessed document based on the number of document words to determine the cutting point using the LateChunking algorithm, and outputting the semantic units obtained by cutting; an entity and relationship recognition module for identifying entity elements from the semantic units and labeling the entity elements with effectiveness level labels and business domain labels, and outputting entity elements and label data, the entity elements including regulation names; a knowledge graph construction module for constructing a bidding domain knowledge graph based on the regulation names as root nodes and the reference, causal, and supplementary relationships between entities; a multi-strategy hybrid retrieval module for receiving user queries, extracting legal entity keywords in the queries, and simultaneously performing semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, and keyword-based matching retrieval, and outputting multi-path recall results; a retrieval result optimization module for calculating semantic similarity and business label weighted sorting of the multi-path recall results, and selecting the optimal candidate answer set for generating results; The effectiveness level labels include national bidding and tendering laws, departmental regulations, and local regulations, and the business domain labels include comprehensive, engineering, finance, and administrative management; The entity and relationship recognition module automatically classifies the entity elements based on the publishing agencies and content themes of the bidding legal documents when labeling the entity elements with effectiveness level labels and business domain labels; The multi-strategy hybrid retrieval module further includes a routing decision unit and a result reliability evaluation module, the routing decision unit is used to receive vector representations of the user queries, and dynamically select at least one retrieval path based on a pre-trained Bayesian decision network, the retrieval path includes vector similarity retrieval, keyword retrieval, and knowledge graph query; The Bayesian decision network outputs a path selection probability distribution by analyzing the relationship between query semantic features and historical performance data of each retrieval path; The result reliability evaluation module is used to fuse the path selection probability distribution output by the Bayesian decision network and the quality score of the retrieval results returned by the path, calculate the overall confidence of the final answer through Bayesian inference, and output a quantitative confidence interval value or confidence level to represent the reliability of the final answer.

2. The system of claim 1, wherein, When the document preprocessing module cleans and extracts structure identification from bidding legal documents, it removes irrelevant characters in the header and footer and marks the starting position and hierarchical relationship of the title, chapter number, clause number, sub-clause, item, and / or content in the preset important clause chapter library.

3. The system of claim 2, wherein, When the semantic segmentation module determines the cutting point based on the number of document words to determine the cutting point using the LateChunking algorithm, and outputting the semantic units obtained by cutting, when the number of document words is less than 500 characters, the first cutting point is determined based on the title and chapter number. When the number of characters in the document is not less than 500, a second cutting point is dynamically determined based on the clause number, and when the number of characters in a clause exceeds 300, further subdivision cutting is performed according to the sub-clause or the sub-item.

4. The system of claim 1, wherein, When the semantic chunking module determines the cutting point using the LateChunking algorithm, it combines the analysis of sentence boundary punctuation and semantic coherence, and limits the cutting point to a period, a semicolon, or a clause number by identifying the sequence of clause numbers, the title level, and the patterns in the preset important clause chapter library.

5. The system of claim 1, wherein, The entity and relationship recognition module extracts entity elements including the issuing agency, the issuance and effective dates, and the applicable scope. When the entity and relationship recognition module identifies entity elements from the semantic units, it uses an embedding model to map the text into a vector representation, and then inputs the vector into a large language model for entity and relationship extraction. The large language model is fine-tuned through reinforcement learning strategies, and during fine-tuning, the reward function is based on entity recognition accuracy, relationship extraction accuracy, or the matching degree with existing entities in the knowledge graph.

6. The system of claim 1, wherein, When the knowledge graph construction module constructs the knowledge graph in the bidding field, it takes the name of the regulation as the root node and constructs the graph edges by identifying the reference relationship, causal relationship, and supplementary explanation relationship between clauses.

7. The system of claim 1, wherein, When the retrieval result selection module selects the optimal candidate answer set, it calculates the posterior probability of the retrieval results that meet the confidence interval requirement or the confidence level requirement. In calculating the posterior probability, the prior probability is determined based on the clause effectiveness level and the publication time. The effectiveness level includes national bidding and tendering laws, departmental regulations, and local regulations. The prior probability weight of national laws is higher than that of local regulations. The likelihood function is constructed based on the semantic similarity score. The top N results with the highest posterior probability are selected as the candidate answer set, and N is a preset constant.

8. The system of claim 1, wherein, The retrieval system also includes an illusion discrimination module for evaluating the credibility of the generated results based on a pre-trained factual discrimination model. The discrimination model outputs a confidence score by comparing the semantic consistency and factual consistency between the generated text and the real clauses in the knowledge base. Results below the threshold are filtered or marked as pending review.

Images