Bidding qualification review method and device based on large language model

Abstract

The present application relates to a bidding qualification review method and device based on a large language model. The method includes: extracting review points from historical bidding documents, constructing a review point set and constructing an expert library of negative cases of review points in the review point set; obtaining the bidding documents to be reviewed, and segmenting the bidding documents into multiple clauses; searching the multiple clauses in the review point set, extracting the scoring clauses that need to be reviewed and scoring criteria, and constructing a scoring clause library; using the scoring clauses in the scoring clause library as input to the large language model, and using the retrieval generation technology to retrieve the case result information in the expert library of negative cases of the review points; constructing a two-stage prompt template, using the case result information as an embedding, and outputting the bidding qualification review information of the bidding documents to be reviewed through the large language model. The use of this method can improve the efficiency of retrieval and generation.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a bidding qualification review method and device based on a large language model. Background Art

[0002] The bidding document is an important document in the process of project bidding and procurement, which stipulates the conditions and requirements for bidders to participate in bidding activities. It is a necessary step to conduct a fair review of the scoring criteria in the bidding documents. The legal review of the scoring criteria in the bidding documents mainly focuses on the fair review of the scoring clauses, judging whether there are discriminatory or unreasonable clauses in the scoring criteria. The review points are numerous and wide-ranging, such as the legality and compliance of the qualification conditions, the rationality of the company's systems and regulations, the consistency of the same clause in the bidding documents, and the completeness of the clauses. However, at present, in the review process of bidding documents, it mainly relies on experts for manual review. This labor-intensive method is highly dependent on the experience and knowledge of the experts themselves. When faced with a large number of review points and clauses in the bidding documents, it is easy to make mistakes or omissions, and it will consume a lot of human resources and time. Therefore, in order to improve the efficiency and accuracy of bidding qualification review, it is necessary to develop an efficient and intelligent automatic review technology.

[0003] Existing methods for using computers to assist in review mainly include rule-based natural language processing methods and machine learning methods. The rule-based natural language processing method requires the definition of comprehensive rules in advance according to the scenarios and needs to extract and compare the clauses or related information in the document. However, due to the complexity and changeability of legal terms and the involvement of a large amount of professional knowledge, it is impossible to cover all possible situations in advance, resulting in the lack of flexibility of this method. In addition, the process of defining rules is cumbersome and labor-intensive, and is not suitable for the large number of document processing needs in actual bidding. The machine learning method requires the preparation of a large amount of high-quality data to train the model, but the traditional machine learning model is difficult to accurately capture the complex relationship and logic between the terms due to its lack of semantic understanding and generation capabilities, and has great limitations in the qualification review of bidding.

[0004] In recent years, with the development of large language models (LLMs), the performance of language models in natural language processing tasks has been significantly improved. Large language models have powerful text understanding and generation capabilities, and can effectively handle complex language expressions in bidding documents. However, they still have certain limitations in the qualification review of bidding documents. First, due to the lack of relevant data for the review of bidding documents, it is impossible to provide sufficient high-quality training data for the language model; secondly, due to the high professional requirements of the qualification review and the wide source of the evaluation criteria, the language model is prone to misjudgment and omission. Therefore, there is an urgent need for a bidding qualification review method and system based on a large language model to achieve efficient, accurate and intelligent review of bidding qualifications and ensure that the bidding documents comply with relevant laws and regulations. Summary of the invention

[0005] Based on this, it is necessary to provide a bidding qualification review method and device based on a large language model to address the above technical problems.

[0006] A bidding qualification review method based on a large language model, the method comprising:

[0007] Extracting review points from historical bidding documents, constructing a review point set, and constructing a review point negative case expert database for the review points in the review point set;

[0008] Obtaining the tender document to be reviewed, and segmenting the tender document into multiple clauses;

[0009] Retrieving the multiple clauses in the review point set, extracting the scoring clauses that need to review the scoring criteria, and building a scoring clause library;

[0010] Using the scoring terms in the scoring terms library as input to the large language model, and using the retrieval generation technology to retrieve case result information from the negative case expert library at the review point;

[0011] A two-stage prompt template is constructed, the case result information is used as an embedding, and the bidding qualification review information of the bidding documents to be reviewed is output through a large language model.

[0012] In one of the embodiments, it also includes: collecting scoring standard data in historical bidding documents, and constructing review points of historical bidding documents according to the provisions of the bidding documents; extracting review points in all historical bidding documents to construct a review point set.

[0013] In one of the embodiments, it also includes: obtaining a legal and compliant review case data set; the fields of the review cases in the review case data set include: review point, case clause, whether it is legal, keywords, case analysis, relevant laws, and legal source; inputting the case clauses of the review cases in the review case data set into a vector coding model to obtain a case clause vector representation of each review case; storing the review point, case clause, case clause vector representation, whether it is legal, keywords, case analysis, relevant laws, and legal source of each review case into a database to obtain a review point negative case expert library.

[0014] In one of the embodiments, it also includes: obtaining vector representations of the multiple clauses and the vector representation of each review point in the review point set; calculating the similarity between the vector representation of each review point and the vector representation of the multiple clauses; sorting each clause from high to low according to the similarity, selecting multiple clauses with the highest ranking for splicing, and obtaining scoring clauses corresponding to the review points; and constructing a scoring clause library using the scoring clauses and the review points.

[0015] In one of the embodiments, it also includes: according to the review point text corresponding to the scoring clause in the scoring clause library, the review point field in the review point negative case expert library is filtered for retrieval review points containing the review point text, and a case clause set corresponding to the retrieval review point is obtained; the scoring clause is vectorized, and the similarity between the vectorized scoring clause and the case clause vector representation is calculated, and the case clauses are sorted from large to small according to the similarity, and multiple case clauses with the top rankings are selected, and the remaining fields thereof are queried in the review point negative case expert library to combine them into case result information.

[0016] In one of the embodiments, it also includes: constructing first-stage prompt words; the first-stage prompt words are used to select the best evaluation result from the case result information; constructing second-stage prompt words; the second-stage prompt words are used to evaluate the selected best evaluation result.

[0017] In one of the embodiments, it also includes: according to the constructed first-stage prompt words and second-stage prompt words, the case result information is used as an embedding, and the bidding qualification review information of the bidding documents to be reviewed is output through a large language model.

[0018] A bidding qualification review device based on a large language model, the device comprising:

[0019] An expert database construction module, used to extract review points from historical bidding documents, construct a review point set, and construct a review point negative case expert database for the review points in the review point set;

[0020] A word segmentation module, used to obtain the tender document to be reviewed and segment the tender document into multiple clauses;

[0021] A clause library building module, used to search the multiple clauses in the review point set, extract the scoring clauses that need to review the scoring criteria, and build a scoring clause library;

[0022] A case retrieval module, used to use the scoring terms in the scoring terms library as input to the large language model, and to retrieve case result information from the negative case expert library at the review point using a retrieval generation technology;

[0023] The evaluation module is used to construct a two-stage prompt template, use the case result information as embedding, and output the bidding qualification review information of the bidding documents to be reviewed through a large language model.

[0024] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0025] Extracting review points from historical bidding documents, constructing a review point set, and constructing a review point negative case expert database for the review points in the review point set;

[0026] Obtaining the tender document to be reviewed, and segmenting the tender document into multiple clauses;

[0027] Retrieving the multiple clauses in the review point set, extracting the scoring clauses that need to review the scoring criteria, and building a scoring clause library;

[0028] Using the scoring terms in the scoring terms library as input to the large language model, and using the retrieval generation technology to retrieve case result information from the negative case expert library at the review point;

[0029] A two-stage prompt template is constructed, the case result information is used as an embedding, and the bidding qualification review information of the bidding documents to be reviewed is output through a large language model.

[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0031] Extracting review points from historical bidding documents, constructing a review point set, and constructing a review point negative case expert database for the review points in the review point set;

[0032] Obtaining the tender document to be reviewed, and segmenting the tender document into multiple clauses;

[0033] Retrieving the multiple clauses in the review point set, extracting the scoring clauses that need to review the scoring criteria, and building a scoring clause library;

[0034] Using the scoring terms in the scoring terms library as input to the large language model, and using the retrieval generation technology to retrieve case result information from the negative case expert library at the review point;

[0035] A two-stage prompt template is constructed, the case result information is used as an embedding, and the bidding qualification review information of the bidding documents to be reviewed is output through a large language model.

[0036] The above-mentioned bidding qualification review method, device, computer equipment and storage medium based on the large language model, in order to avoid the problems of lack of data and high requirements in traditional technologies, the present invention can use a large number of external historical bidding documents as cases through the extraction of review points, extract review points from them to construct a review point set, and obtain a review point negative case expert library by combing the fields of the historical bidding documents. The review point negative case expert library contains all the information of a review point. In order to feedback the evaluation of the historical bidding documents to the bidding documents to be reviewed, it is first necessary to segment the bidding documents to be reviewed, and then search in the review point set to query the scoring terms contained in the bidding documents to be reviewed. Using the scoring terms for retrieval can significantly improve the retrieval efficiency and accuracy, and then use the scoring term library as the input of the large language model, obtain case result information through retrieval generation technology, and embed it into the large model to enhance the reasoning ability of the large model, so that it can generate more accurate bidding qualification review opinions, thereby achieving the improvement of the efficiency and accuracy of bidding qualification review. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 1 is a flow chart of a bidding qualification review method based on a large language model in one embodiment;

[0038] Figure 2 A flowchart of a bidding qualification review method based on a large language model in another embodiment;

[0039] Figure 3 A schematic diagram of a negative case expert database for review points in an embodiment;

[0040] Figure 4 A schematic diagram of a bidding document scoring clause library in one embodiment;

[0041] Figure 5 A schematic diagram of a self-consistent sampling method for generating answers multiple times in one embodiment;

[0042] Figure 6 It is a structural block diagram of a bidding qualification review device based on a large language model in one embodiment;

[0043] Figure 7FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In one embodiment, Figure 1 As shown, a bidding qualification review method based on a large language model is provided, comprising the following steps:

[0046] Step 102, extracting review points from historical bidding documents, constructing a review point set, and constructing a review point negative case expert database for the review points in the review point set.

[0047] The scenario for this example is the bidding qualification review scenario. Since the review points are numerous and wide-ranging, but the relevant data and corresponding bidding documents have not been accumulated, it is impossible to provide sufficient high-quality data for the large language model to train. Therefore, it is necessary to use the external knowledge source method to provide external knowledge for the large language model reasoning process. To facilitate subsequent retrieval, it is necessary to extract the review points of the requirements from the historical bidding documents. Currently, the scoring criteria in the bidding documents are subject to legal review, and the review content focuses on the fairness review of the scoring clauses to determine whether there are discriminatory or unreasonable clauses in the scoring criteria. The judgment is mainly based on: "Bidding Law of the People's Republic of China", "Implementation Regulations of Bidding of the People's Republic of China", "Government Procurement Law of the People's Republic of China", "Implementation Regulations of the Government Procurement Law of the People's Republic of China", etc.

[0048] Step 104, obtaining the tender document to be reviewed, and segmenting the tender document into multiple clauses.

[0049] Step 106, multiple clauses are searched in the review point set, scoring clauses that need to be reviewed for scoring criteria are extracted, and a scoring clause library is constructed.

[0050] Step 108, using the scoring terms in the scoring terms library as input to the large language model, and using the retrieval generation technology to retrieve the case result information in the review point negative case expert library.

[0051] Step 110, construct a two-stage prompt template, use the case result information as embedding, and output the bidding qualification review information of the bidding documents to be reviewed through the large language model.

[0052] In the above-mentioned bidding qualification review method based on a large language model, in order to avoid the problems of lack of data and high requirements in traditional technologies, the present invention can use a large number of external historical bidding documents as cases through the extraction of review points, extract review points from them to construct a review point set, and obtain a review point negative case expert library by combing the fields of historical bidding documents. The review point negative case expert library contains all the information of a review point. In order to feedback the evaluation of historical bidding documents to the bidding documents to be reviewed, it is first necessary to segment the bidding documents to be reviewed, and then search in the review point set to query the scoring terms contained in the bidding documents to be reviewed. Using the scoring terms for retrieval can significantly improve the retrieval efficiency and accuracy, and then use the scoring term library as the input of the large language model, obtain case result information through retrieval generation technology, and embed it into the large model to enhance the reasoning ability of the large model, so that it can generate more accurate bidding qualification review opinions, thereby achieving the improvement of the efficiency and accuracy of bidding qualification review.

[0053] In one of the embodiments, scoring standard data in historical bidding documents are collected, and review points of the historical bidding documents are constructed according to the provisions of the bidding documents; and review points in all historical bidding documents are extracted to construct a review point set.

[0054] Specifically, extracting review points mainly includes the following main steps:

[0055] (1) Collect historical bidding document scoring standard data;

[0056] (2) Based on relevant laws and regulations, including laws and regulations such as the Tendering and Bidding Law of the People's Republic of China and company regulations such as the Implementation Rules for Procurement Business of State Grid Corporation of China, sort out the review points in historical bidding documents;

[0057] (3) Use information extraction technology to extract the review points in the historical bidding documents and obtain the review point set , where each is a review point, and there are p review points in total.

[0058] Example of review point set:

[0059] {"Is the project name accurate?", "Is the project number repeated?", "Is the end time of obtaining the bidding documents correct?", ...}

[0060] In one of the embodiments, a legal and compliant review case data set is obtained; the fields of the review cases in the review case data set include: review point, case clause, whether it is legal, keywords, case analysis, relevant laws, and legal source; the case clauses of the review cases in the review case data set are input into a vector coding model to obtain a case clause vector representation of each review case; the review point, case clause, case clause vector representation, whether it is legal, keywords, case analysis, relevant laws, and legal source of each review case are stored in a database to obtain a review point negative case expert database.

[0061] Specifically, obtain a legal and compliant review case dataset; the fields of the review cases in the review case dataset include: review point, case clause, legality, keywords, case analysis, relevant laws, and legal sources; load the embedding model for encoding; input each case clause into the embedding model, call its encoding function, and convert the case clause into a vector representation:

[0062] ;

[0063] in is the vector representation of case terms, is the encoding function, The text representation of the case clauses. The review points, case clauses, vector representation of case clauses, legality, keywords, case analysis, relevant laws, and legal sources are stored in the database to build a review point negative case expert database.

[0064] In one of the embodiments, vector representations of multiple clauses and vector representations of each review point in a review point set are obtained; the similarity between the vector representation of each review point and the vector representations of multiple clauses is calculated; each clause is sorted from high to low according to the similarity, and multiple clauses with the highest ranking are selected for concatenation to obtain scoring clauses corresponding to the review points; and a scoring clause library is constructed using the scoring clauses and review points.

[0065] Specifically, the steps for constructing the scoring clause library of the bidding documents include:

[0066] Read the specified bidding document, load the embedding model used for encoding, split the bidding document, and obtain the clause set as follows:

[0067] ;

[0068] Where id is the clause index number, is the clause text content, and n is the total number of clauses.

[0069] Generate vectorized representations for each clause and review point in the review point set using the embedding model:

[0070] ;

[0071] in is the vectorized representation of the clause, is the vectorized representation of the review point, is the encoding function, is the textual representation of the clause, is the textual representation of the review point.

[0072] For review points, in the bidding documents, each review point may involve multiple clauses. Therefore, it is necessary to use vector retrieval to obtain the possible scoring clauses to improve the retrieval efficiency and accuracy; use cosine similarity as a similarity measurement method to calculate the vector representation of the review point Vector representation of clauses Similarities between :

[0073] ;

[0074] in, Representation vector The model, Representation vector The model, The larger the value of, the higher the similarity;

[0075] Sort the clauses according to their similarity to a certain review point from high to low, and select the first The clauses with the highest similarity to a certain review point are connected into a long sentence, which is a scoring clause. For example, the similarity between the sorted clauses and the review points is {0.92, 0.88, 0.86, 0.79, 0.74, 0.73, 0.54, ......}, and we set , then select clauses with similarity values ​​of 0.92, 0.88, 0.86, 0.79, and 0.74 to connect into long sentences. Use scoring clauses and review points to build a scoring clause library for bidding documents.

[0076] The following are examples of scoring terms and their corresponding review point data:

[0077] Scoring terms: "For procurement documents and bidding documents, the business evaluation items include "3 points for bid preparation", "2 points for staffing", and "10 points for overall strength".

[0078] Review point: "Technical conditions: The set technical conditions are consistent with the specific characteristics and actual needs of the procurement project or are related to the performance of the contract."

[0079] In one of the embodiments, based on the review point text corresponding to the scoring clause in the scoring clause library, the review point field in the review point negative case expert library is used to filter the retrieval review points containing the review point text, and a set of case clauses corresponding to the retrieval review points is obtained; the scoring clauses are vectorized, and the similarity between the vectorized scoring clauses and the case clause vector representation is calculated, and the case clauses are sorted from large to small according to the similarity, and multiple case clauses with the top rankings are selected, and their remaining fields are queried in the review point negative case expert library to combine them into case result information.

[0080] Specifically, load the Qwen1.5-14B model and the corresponding word segmenter;

[0081] Set model inference parameters, for example: max_new_tokens=512, do_sample=True, top_k=5, top_p=0.9.

[0082] For each scoring clause in the bidding document scoring clause library , taking it as input;

[0083] Using RAG technology, search for the input scoring terms Corresponding review point First, filter out the "examination point" field in the negative case expert database of the examination point to include The review points of the text, and obtain the relevant Case clauses, get the case clause collection:

[0084] ,

[0085] in For case clauses.

[0086] The scoring terms Vectorization :

[0087] ;

[0088] Vectorize the case terms to get :

[0089] ;

[0090] Calculate the similarity between the scoring terms and the case terms using cosine similarity :

[0091] ;

[0092] in, Representation vector The model, Representation vector The model, The larger the value of , the higher the similarity between the scoring clause and the case clause. Sort the case clauses by similarity from high to low, and select the first The case clauses that are most similar to a certain review point.

[0093] Extracted from the expert database of negative cases at the review point The legality, keywords, case analysis, relevant laws, and legal source information corresponding to each case clause are combined into a case review opinion.

[0094] An example of a case clause and its case review opinion is as follows:

[0095] {"Case Clause": "For procurement documents and bidding documents, the business evaluation items are set as "providing copies of the above certificates and affixing the original official seal 3 points", "bidding document preparation and binding 2 points", "financial status 1 point"",

[0096] "Case review opinion": {"Is it legal": "No",

[0097] "Keywords": "Compilation issues / irrelevant conditions",

[0098] "Case Analysis": "The qualifications, technology and business conditions set have nothing to do with the performance of the contract.",

[0099] "Relevant laws": ""A purchaser or purchasing agency shall be deemed to have adopted differential or discriminatory treatment against suppliers under unreasonable conditions if any of the following circumstances occurs: ... (ii) the qualifications, technical and commercial conditions set are not in line with the specific characteristics and actual needs of the bidding project or are irrelevant to the performance of the contract"",

[0100] "Legal source information": "Article 20, Item (2) of the Regulations for the Implementation of the Government Procurement Law of the People's Republic of China"}

[0101] }

[0102] In one of the embodiments, first-stage prompt words are constructed; the first-stage prompt words are used to select the best evaluation result from the case result information; second-stage prompt words are constructed; the second-stage prompt words are used to evaluate the selected best evaluation result.

[0103] Specifically, the first-stage prompt words are constructed, and the large language model is required to select the {case clause-case review opinion} with the most reference value for the scoring clause, for example:

[0104] First stage prompt words =“The following are review points, scoring clauses of the bidding documents, several related case clauses, and case review opinions. Please study them carefully, summarize them, and select the case clause and corresponding case review opinion that you think is most valuable for reference.”

[0105] For review points and scoring terms, compare them with the corresponding {Case Clauses-Case Review Opinions} and prompt words Embedded in a first-stage query, for example:

[0106] =“The following are the review points, scoring clauses of the bidding documents, several related case clauses, and case review opinions. Please analyze them carefully and select the case clause and corresponding case review opinion that you think is most valuable for reference.

[0107] Review points: ;

[0108] Rating Terms: ;

[0109] Case Clause 1: ;

[0110] Case review opinion 1: ; ......

[0111] Case Clause I: ;

[0112] Case review opinion i: ”

[0113] Use a large language model to generate answers to first-stage queries.

[0114] Using a self-consistent sampling method, the large model generates multiple choices of {case clauses-case review opinions} that are most valuable for the scoring clauses, thereby increasing the diversity of answers. For example, the large model generates 5 candidate answers:

[0115] ;

[0116] Use majority voting method and take the option with the most votes as the reference {Case Clause-Case Review Opinion} Each The voting score is calculated as follows:

[0117] ;

[0118] For example, in middle, , , Case Clause Case review opinions The most valuable reference is ;and and Consider the case clause Case review opinions The most valuable reference is , the scores of other case clauses and their case review opinions are all 0, then after a majority vote, the case clause will be selected Case review opinions As the final answer.

[0119] Construct the second-stage prompt words and ask the large language model to give review opinions on the scoring terms, for example:

[0120] Second stage prompt words ="The following are the review points, scoring clauses of the bidding documents, case clauses to be referenced and their case review opinions. Please analyze the case clauses to be referenced and their case review opinions, and consider whether the scoring clauses of the given bidding documents are legal and compliant based on the review points. Required answers: 1. Is it illegal? (No / Yes) 2. The degree of illegality? (Extremely serious, serious, general, minor) 3. Review opinion."

[0121] The prompt words, the review points and the scoring clauses and their referenced {case clauses-case review opinions} are embedded into the second-stage query, for example:

[0122] =“The following are the review points, scoring clauses of the bidding documents, case clauses to be referenced and their case review opinions. Please analyze the case clauses to be referenced and their case review opinions, and consider whether the scoring clauses of the bidding documents given are legal and compliant based on the review points. Required answers: 1. Is it legal? (No / Yes) 2. Degree of illegality? (Extremely serious, serious, general, minor) 3. Review opinion.

[0123] Review points: ;

[0124] Rating Terms: ;

[0125] Case clause: ;

[0126] Case review opinion: ”

[0127] Generate answers to the second-stage queries using a large language model;

[0128] Using a self-consistent sampling method, the large model can generate multiple judgments and review opinions on whether the scoring terms are legal and compliant, thereby improving the diversity of answers; using the majority voting method, the judgment with the most votes is adopted.

[0129] For example, the answer generated by the second-stage query is: "Is it legal: No; Degree of illegality: Serious; Review opinion: According to Article 20 (2) of the "Implementation Regulations of the Government Procurement Law of the People's Republic of China", "If the purchaser or the purchasing agency has any of the following circumstances, it shall be considered to have implemented differential or discriminatory treatment against suppliers under unreasonable conditions: ... (2) The qualifications, technical and commercial conditions set are not in line with the specific characteristics and actual needs of the bidding project or are irrelevant to the performance of the contract", while the qualifications, technical and commercial conditions set in the scoring terms are irrelevant to the performance of the contract."

[0130] Organize the answers and generate a legal and compliance review opinion for each review point and the scoring clauses described therein, including legal and regulatory review points, specific clauses in the bidding documents, review analysis, and the degree of illegality.

[0131] For example:

[0132] {Legal and regulatory review points: Technical conditions: The set technical conditions are consistent with the specific characteristics and actual needs of the procurement project or are related to the performance of the contract.

[0133] Specific terms in the bidding documents: In the procurement documents and bidding documents, the business evaluation items include “3 points for bid preparation”, “2 points for staffing” and “10 points for overall strength”.

[0134] Review and Analysis: According to Article 20(2) of the Regulations for the Implementation of the Government Procurement Law of the People's Republic of China, "If the purchaser or the purchasing agency has any of the following circumstances, it shall be deemed to have implemented differential or discriminatory treatment against suppliers under unreasonable conditions:... (2) The qualifications, technical and commercial conditions set forth are not in line with the specific characteristics and actual needs of the bidding project or are irrelevant to the performance of the contract." The qualifications, technical and commercial conditions set forth in the scoring clauses are irrelevant to the performance of the contract.

[0135] Violation level: Severe

[0136] }

[0137] The following is an explanation of the specific steps. Figure 2 As shown, the following steps are included:

[0138] S1: Collect historical bidding document scoring standard data and sort out the review points according to relevant laws and regulations to obtain a review point set.

[0139] S2: Build Figure 3The negative case expert database of the review points shown includes review points, case clauses, legality, keywords, case analysis, relevant laws, and legal sources.

[0140] S3: For a bidding document, split it into multiple clauses, perform vector retrieval with the review points, extract the scoring clauses that need to review the scoring criteria, and construct the following: Figure 4 The library of scoring clauses for tender documents is shown.

[0141] S4: The scoring terms are used as the input of the language model, and the retrieval-augmented generation (RAG) technology is used to retrieve the negative case expert database at the review point to obtain the case result information.

[0142] S5: Using the case result information as embedding, using two-stage prompts, and using a language model to generate a legal and compliance review opinion on the bidding document, including legal and regulatory review points, specific clauses in the bidding document, review analysis, and degree of illegality.

[0143] In this embodiment, the large language model includes but is not limited to GPT-3, Qwen, Chinese-LLaMA, etc.

[0144] Among them, an expert database of negative cases at the review point is constructed, a database of scoring clauses of the bidding documents is constructed, retrieval enhancement technology is used to improve retrieval efficiency and accuracy, case result information that can be used as a reference for scoring clauses is obtained and embedded into the big model, the reasoning ability of the big model is enhanced, so that it can generate more accurate bidding qualification review opinions, thereby improving the efficiency and accuracy of bidding qualification review. In addition, the big model in the present invention uses two-stage prompts, such as Figure 5 The self-consistent sampling method shown generates answers multiple times, for example, the score of Case 1 is equal to 3 / 5, and selects the answer with the highest number of votes as the result, which greatly enhances the robustness of the large model review results and realizes efficient, accurate and automated qualification review of bidding documents.

[0145] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0146] In one embodiment, Figure 6 As shown, a bidding qualification review device based on a large language model is provided, including: an expert database construction module 602, a word segmentation module 604, a clause database establishment module 606, a case retrieval module 608 and an evaluation module 610, wherein:

[0147] An expert database construction module 602 is used to extract review points from historical bidding documents, construct a review point set, and construct a review point negative case expert database of the review points in the review point set;

[0148] The word segmentation module 604 is used to obtain the tender document to be reviewed and segment the tender document into multiple clauses;

[0149] A clause library building module 606 is used to search the multiple clauses in the review point set, extract the scoring clauses that need to review the scoring criteria, and build a scoring clause library;

[0150] A case retrieval module 608 is used to use the scoring terms in the scoring terms library as inputs to the large language model, and to retrieve case result information from the review point negative case expert library using a retrieval generation technology;

[0151] The evaluation module 610 is used to construct a two-stage prompt template, use the case result information as an embedding, and output the bidding qualification review information of the bidding document to be reviewed through a large language model.

[0152] In one of the embodiments, the expert database construction module 602 is also used to collect scoring standard data in historical bidding documents, and construct review points of historical bidding documents according to the provisions of the bidding documents; extract review points from all historical bidding documents and construct a review point set.

[0153] In one embodiment, the expert database construction module 602 is also used to obtain a legal and compliant review case data set; the fields of the review cases in the review case data set include: review point, case clause, whether it is legal, keywords, case analysis, relevant laws, and legal source; the case clauses of the review cases in the review case data set are input into a vector coding model to obtain a case clause vector representation of each review case; the review point, case clause, case clause vector representation, whether it is legal, keywords, case analysis, relevant laws, and legal source of each review case are stored in a database to obtain a review point negative case expert database.

[0154] In one embodiment, the clause library establishment module 606 is also used to obtain the vector representation of the multiple clauses and the vector representation of each review point in the review point set; calculate the similarity between the vector representation of each review point and the vector representation of the multiple clauses; sort each clause from high to low according to the similarity, select the top-ranked clauses for splicing, and obtain the scoring clauses corresponding to the review points; and use the scoring clauses and the review points to construct a scoring clause library.

[0155] In one of the embodiments, the case retrieval module 608 is also used to filter the retrieval review points containing the review point text according to the review point text corresponding to the scoring clause in the scoring clause library, and obtain the case clause set corresponding to the retrieval review point based on the review point field in the review point negative case expert library; vectorize the scoring clause, calculate the similarity between the vectorized scoring clause and the case clause vector representation, sort the case clauses from large to small according to the similarity, select multiple case clauses with the top ranking, query their remaining fields in the review point negative case expert library, and combine them into case result information.

[0156] In one embodiment, the evaluation module 610 is also used to construct first-stage prompt words; the first-stage prompt words are used to select the best evaluation result from the case result information; construct second-stage prompt words; the second-stage prompt words are used to evaluate the selected best evaluation result.

[0157] In one embodiment, the evaluation module 610 is further used to embed the case result information according to the constructed first-stage prompt words and second-stage prompt words, and output the bidding qualification review information of the bidding documents to be reviewed through the large language model.

[0158] For the specific limitations of the bidding qualification review device based on the large language model, please refer to the limitations of the bidding qualification review method based on the large language model above, which will not be repeated here. Each module in the above-mentioned bidding qualification review device based on the large language model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0159] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a bidding qualification review method based on a large language model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0160] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0162] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0163] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A bidding qualification review method based on a large language model, characterized in that: The method comprises: Extracting review points from historical bidding documents, constructing a review point set, and constructing a review point negative case expert database for the review points in the review point set; Obtaining the tender document to be reviewed, and segmenting the tender document into multiple clauses; Retrieving the multiple clauses in the review point set, extracting the scoring clauses that need to review the scoring criteria, and building a scoring clause library; Using the scoring terms in the scoring terms library as input to the large language model, and using the retrieval generation technology to retrieve case result information from the negative case expert library at the review point; Construct a two-stage prompt template, use the case result information as embedding, and output the bidding qualification review information of the bidding documents to be reviewed through a large language model; Build a two-stage prompt template, including: Constructing a first-stage prompt word; the first-stage prompt word is used to select the best evaluation result from the case result information; Construct the second stage prompt words; the second stage prompt words are used to evaluate the selected optimal evaluation result.

2. The method according to claim 1, characterized in that Extract review points from historical bidding documents and construct a review point set, including: Collect scoring criteria data from historical bidding documents and construct review points for historical bidding documents according to the provisions of the bidding documents; Extract review points from all historical bidding documents and build a review point set.

3. The method according to claim 1, characterized in that Constructing an expert database of negative cases of the review points in the review point set, including: Obtain a legal and compliant review case data set; the fields of the review case in the review case data set include: review point, case clause, whether it is legal, keywords, case analysis, relevant laws, and legal source; Inputting the case clauses of the review cases in the review case data set into a vector encoding model to obtain a case clause vector representation of each review case; The review point, case clause, case clause vector representation, legality, keywords, case analysis, relevant laws, and legal sources of each review case are stored in the database to obtain an expert database of negative cases at the review point.

4. The method according to claim 1, characterized in that: The multiple clauses are searched in the review point set, scoring clauses that need to be reviewed for scoring criteria are extracted, and a scoring clause library is constructed, including: Obtaining vector representations of the plurality of clauses and a vector representation of each review point in the review point set; Calculating the similarity between the vector representation of each review point and the vector representations of the plurality of clauses; Sort each clause from high to low according to similarity, select multiple clauses with the highest ranking and concatenate them to obtain the scoring clause corresponding to the review point; A scoring clause library is constructed using the scoring clauses and the review points.

5. The method according to claim 4, characterized in that The scoring terms in the scoring terms library are used as inputs of the large language model, and the case result information is retrieved from the negative case expert library at the review point using the retrieval generation technology, including: According to the review point text corresponding to the scoring clause in the scoring clause library, the review point field in the review point negative case expert library is filtered for search review points containing the review point text, and a case clause set corresponding to the search review point is obtained; The scoring clauses are vectorized, the similarity between the vectorized scoring clauses and the case clause vector representation is calculated, the case clauses are sorted from large to small according to the similarity, a plurality of case clauses with the top rankings are selected, the remaining fields thereof are queried in the negative case expert database at the review point, and the result is combined into case result information.

6. The method according to claim 5, characterized in that A two-stage prompt template is constructed, the case result information is used as an embedding, and the bidding qualification review information of the bidding documents to be reviewed is output through a large language model, including: According to the constructed first-stage prompt words and second-stage prompt words, the case result information is used as embedding, and the bidding qualification review information of the bidding documents to be reviewed is output through the large language model.

7. A bidding qualification review device based on a large language model, used to implement the bidding qualification review method based on a large language model as described in any one of claims 1 to 6, characterized in that: The device comprises: An expert database construction module, used to extract review points from historical bidding documents, construct a review point set, and construct a review point negative case expert database for the review points in the review point set; A word segmentation module, used to obtain the tender document to be reviewed and segment the tender document into multiple clauses; A clause library building module, used to search the multiple clauses in the review point set, extract the scoring clauses that need to review the scoring criteria, and build a scoring clause library; A case retrieval module, used to use the scoring terms in the scoring terms library as input to the large language model, and to retrieve case result information from the negative case expert library at the review point using a retrieval generation technology; The evaluation module is used to construct a two-stage prompt template, use the case result information as embedding, and output the bidding qualification review information of the bidding documents to be reviewed through a large language model.

Images