Method and device for generating negative cases in bidding documents based on large language model

Abstract

The present application relates to a method and device for generating negative cases of bidding documents based on a large language model. The method comprises: firstly obtaining a dataset of negative cases of bidding documents, then constructing a prompt template for generating input data, a prompt template for generating output data, and a prompt template for fine-tuning input-output pairs, then constructing an example of input-output pairs of negative case data of bidding documents, and sequentially and iteratively generating prompts for generating input data, prompts for generating output data, and prompts for fine-tuning input-output pairs to generate negative cases of bidding documents. The use of this method can improve the comprehensiveness of the dataset of negative cases of bidding documents, thereby ensuring the accuracy of generating negative cases of bidding documents.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a method and device for generating negative cases in bidding documents based on a large language model. Background Art

[0002] With the continuous development of social economy, bidding plays a vital role in all walks of life. In the bidding process, relevant enterprises need to review the bidding documents. The preparation and review of bidding documents often involve a large number of laws, regulations and industry norms. It is particularly important for enterprises and regulatory authorities to ensure the compliance of documents. Traditional bidding document review usually relies on manual work. Reviewers manually read a large number of documents to identify content that does not comply with laws and regulations. Due to the huge number and complexity of bidding documents, manual review methods have problems such as large workload, low efficiency, and subjective judgment affecting accuracy. To meet this challenge, applying intelligent review technology to the field of bidding document review is a development direction. The automated review system based on generative artificial intelligence can effectively improve the review efficiency and reduce manual participation. However, in order for the language model to accurately understand and implement the review regulations, it is necessary to rely on a large amount of case data to train the model, especially negative cases involving violations or errors. Existing negative cases usually rely on manual annotation, which consumes a lot of manpower costs, and the amount of annotation data is limited, making it difficult to cover all possible errors or violation scenarios. Models trained based on these cases often have difficulty dealing with various types of illegal statements. Therefore, how to build a high-quality and comprehensive negative case library for bidding documents has become an urgent problem to be solved.

[0003] At present, in the field of data synthesis, thanks to the powerful semantic understanding and generation capabilities of large language models, large language models can have better performance than traditional deep learning methods. However, there are still some limitations in the synthesis of negative case data of bidding documents. First, the method of synthesizing data by simply writing prompts and providing some examples for large language models is often less accurate than the method of fine-tuning the model using reorganized training data, and the synthetic data depends largely on the model's ability to follow prompts and examples, making it difficult to synthesize complex and diverse bidding negative case data; second, the method of fine-tuning the pre-trained language model through supervised fine-tuning relies on sufficient high-quality training data, and data scarcity will significantly reduce the effectiveness of this method, so it is not suitable for generating bidding negative case data with scarce labeled data; in addition, recent studies have explored the potential of using effective "teacher" large language models to synthesize training data for specific tasks, thereby improving the capabilities of relatively poor "student" large language models, but the feasibility of this method is hindered by cost, scalability, and legal barriers, which are related to the continued reliance on more powerful "teacher" large language models. When there is no more powerful model to provide learning signals, this method is fundamentally unfeasible. It can be seen that there are currently problems in the synthesis of negative case data in bidding documents, such as scarcity of training data, poor quality of synthesized data, and high training costs. Summary of the invention

[0004] Based on this, it is necessary to provide a method and device for generating negative cases in bidding documents based on a large language model to address the above technical problems.

[0005] A method for generating negative cases of bidding documents based on a large language model, the method comprising:

[0006] Obtaining a negative case data set of bidding documents; the negative case data set of bidding documents includes negative cases of bidding documents;

[0007] Constructing input data generation prompt template, output data generation prompt template, and input-output pair fine-tuning prompt template respectively;

[0008] Extracting negative cases of bidding documents from the negative case data set of bidding documents, and constructing an input-output pair example of negative case data of bidding documents, and filling an input data generation prompt template according to the case input data of the input-output pair example to obtain an input data generation prompt;

[0009] Using a large language model to process the input data to generate prompts, obtain multiple input data, and store them in an input storage repository;

[0010] Extracting an input subset from the input repository and fusing it with the case input data to obtain an updated input data generation prompt and obtain an extended input set;

[0011] Extracting synthetic input data from the extended input set, filling the output data generation prompt template according to the synthetic input data and the input-output pair example, and obtaining the output data generation prompt;

[0012] Using a large language model to perform context learning processing on the output data generation prompt to generate a plurality of synthetic output data corresponding to the output data generation prompt;

[0013] An input-output pair fine-tuning hint template is populated according to the synthesized output data, the synthesized input data, and the input-output pair examples to fine-tune a large language model according to the input-output pair fine-tuning hint template.

[0014] In one of the embodiments, the construction of the input data generation prompt template, the output data generation prompt template, and the input-output pair fine-tuning prompt template all include: prompt words, text to be filled, and identifier descriptions; the prompt words are phrases used to guide the understanding and generation of the large language model; the text to be filled is used to fill in the original text; the identifier is a symbol or keyword that marks specific content or a specific structure.

[0015] In one embodiment, the method further includes: receiving an input word segmentation sequence , the large language model is based on the context in the Step to the next word Make a prediction and generate the probability distribution of the next word The calculation formula is:

[0016] ;

[0017] in The big language model comes first The sequence of all words that have been predicted. The large language model is The logits vector generated in the step is used to represent the original score value of the model for each word, and the probability conversion function is a function that transforms a logits vector into a probability distribution.

[0018] In one embodiment, the probability conversion function for:

[0019] ;

[0020] in is generated The score of the word vector, Get the probability corresponding to each word, is the temperature parameter.

[0021] In one of the embodiments, it further includes: filtering the input data or the synthesized output data through a preset noise filter and a length filter;

[0022] The noise filter is used to filter out noise items in input data or synthesized output data through a preset noise term list; and the length filter is used to filter the text length of input data or synthesized output data through a preset length range.

[0023] In one embodiment, it also includes: constructing a cross entropy loss function:

[0024] ;

[0025] in, is the sequence of all words that the model has predicted, is the value that should be generated currently, For a given parameter Under the premise that the current predicted value is the value that should be generated at present probability;

[0026] During the back-propagation process, the input-output loss value of the fine-tuned prompt template is calculated to perform gradient updates on the large language model for fine-tuning.

[0027] A device for generating negative cases of bidding documents based on a large language model, the device comprising:

[0028] A case acquisition module is used to acquire a negative case data set of bidding documents; the negative case data set of bidding documents includes negative cases of bidding documents;

[0029] A template construction module, used to respectively construct an input data generation prompt template, an output data generation prompt template, and an input-output pair fine-tuning prompt template;

[0030] An input data generation module is used to extract negative cases of bidding documents from the negative case data set of bidding documents, and construct an input-output pair example of negative case data of bidding documents, and fill in an input data generation prompt template according to the case input data of the input-output pair example to obtain an input data generation prompt;

[0031] An input data storage module, used to process the input data generation prompt using a large language model to obtain multiple input data and store them in an input storage repository;

[0032] An expansion module is used to extract an input subset from the input repository and fuse it with the case input data to obtain an updated input data generation prompt to obtain an expanded input set;

[0033] An output data generation module, used for extracting synthesized input data from the extended input set, filling the output data generation prompt template according to the synthesized input data and the input-output pair example, and obtaining the output data generation prompt;

[0034] an output data synthesis module, configured to perform context learning processing on the output data generation prompt using a large language model, and generate a plurality of synthesized output data corresponding to the output data generation prompt;

[0035] A fine-tuning module is used to fill an input-output pair fine-tuning prompt template according to the synthesized output data, the synthesized input data and the input-output pair examples, so as to fine-tune the large language model according to the input-output pair fine-tuning prompt template.

[0036] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Obtaining a negative case data set of bidding documents; the negative case data set of bidding documents includes negative cases of bidding documents;

[0038] Constructing input data generation prompt template, output data generation prompt template, and input-output pair fine-tuning prompt template respectively;

[0039] Extracting negative cases of bidding documents from the negative case data set of bidding documents, and constructing an input-output pair example of negative case data of bidding documents, and filling an input data generation prompt template according to the case input data of the input-output pair example to obtain an input data generation prompt;

[0040] Using a large language model to process the input data to generate prompts, obtain multiple input data, and store them in an input storage repository;

[0041] Extracting an input subset from the input repository and fusing it with the case input data to obtain an updated input data generation prompt and obtain an extended input set;

[0042] Extracting synthetic input data from the extended input set, filling the output data generation prompt template according to the synthetic input data and the input-output pair example, and obtaining the output data generation prompt;

[0043] Using a large language model to perform context learning processing on the output data generation prompt to generate a plurality of synthetic output data corresponding to the output data generation prompt;

[0044] An input-output pair fine-tuning hint template is populated according to the synthesized output data, the synthesized input data, and the input-output pair examples to fine-tune a large language model according to the input-output pair fine-tuning hint template.

[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0046] Obtaining a negative case data set of bidding documents; the negative case data set of bidding documents includes negative cases of bidding documents;

[0047] Constructing input data generation prompt template, output data generation prompt template, and input-output pair fine-tuning prompt template respectively;

[0048] Extracting negative cases of bidding documents from the negative case data set of bidding documents, and constructing an input-output pair example of negative case data of bidding documents, and filling an input data generation prompt template according to the case input data of the input-output pair example to obtain an input data generation prompt;

[0049] Using a large language model to process the input data to generate prompts, obtain multiple input data, and store them in an input storage repository;

[0050] Extracting an input subset from the input repository and fusing it with the case input data to obtain an updated input data generation prompt and obtain an extended input set;

[0051] Extracting synthetic input data from the extended input set, filling the output data generation prompt template according to the synthetic input data and the input-output pair example, and obtaining the output data generation prompt;

[0052] Using a large language model to perform context learning processing on the output data generation prompt to generate a plurality of synthetic output data corresponding to the output data generation prompt;

[0053] An input-output pair fine-tuning hint template is populated according to the synthesized output data, the synthesized input data, and the input-output pair examples to fine-tune a large language model according to the input-output pair fine-tuning hint template.

[0054] The above-mentioned method, device, computer equipment and storage medium for generating negative cases in bidding documents based on a large language model construct three types of prompt templates, namely, input data generation prompt template, output data generation prompt template and input-output pair fine-tuning prompt template, and then adopt a self-iterative method. First, based on the input-output pair example of the negative case data in the bidding document, the input data generation prompt template is filled to generate the input data generation prompt, and then the large language model is used to process the input data generation prompt to obtain multiple input data, and the synthetic input data is obtained after processing to complete the first stage of generation, and then the synthetic input data is transmitted to the output data generation prompt template to obtain the output data generation prompt, and then the large language model is used to perform context learning processing on the output data generation prompt to generate multiple synthetic output data corresponding to the output data generation prompt to complete the second stage of generation, and finally the synthetic output data and the original input-output pair example of the negative case data in the bidding document are used to synthesize the input-output pair of the negative case data in the bidding document, and finally the negative case fine-tuning prompt of the bidding document is generated to achieve the fine-tuning of the large language model, and finally the large language model is fine-tuned and updated to complete the third stage of generation. Through the above-mentioned method, the comprehensiveness of the case data set of bidding documents is improved, thereby ensuring the accuracy of the generation of negative cases in bidding documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a process of generating negative cases of bidding documents based on a large language model in one embodiment;

[0056] Figure 2 A framework diagram of a method for generating negative cases in bidding documents based on a large language model in another embodiment;

[0057] Figure 3 A schematic diagram of a negative case of a bidding document generated in one embodiment;

[0058] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] In one embodiment, Figure 1 As shown, a method for generating negative cases of bidding documents based on a large language model is provided, comprising the following steps:

[0061] Step 102, obtaining a negative case dataset of bidding documents.

[0062] The negative case data set of bidding documents includes negative cases of bidding documents. The present invention is aimed at the intelligent review scenario of bidding documents. The existing negative case data of bidding documents is in tabular form and needs to be preprocessed and filled into the corresponding prompt template to obtain input data generation prompt, output data generation prompt, and input-output pair fine-tuning prompt text data respectively.

[0063] Step 104 , constructing an input data generation prompt template, an output data generation prompt template, and an input-output pair fine-tuning prompt template respectively.

[0064] The scenario for this example is the intelligent review scenario of bidding documents. Since the accuracy of the model-based intelligent review method for bidding documents is greatly affected by the quantity and quality of the negative case data of bidding documents used to train the model, the goal of this invention is to synthesize the negative case data of bidding documents, and expand rich and high-quality negative case data of bidding documents from the scarce and incomplete negative cases of bidding documents, so as to improve the performance of the model of intelligent review of bidding documents trained with the negative case data of bidding documents. In order to guide the large language model to generate content that meets the requirements, this example constructs a prompt template for the generation of negative cases of bidding documents.

[0065] Step 106, extracting negative cases of bidding documents from the bidding document negative case data set, and constructing an input-output pair example of negative case data of bidding documents, filling in an input data generation prompt template according to the case input data of the input-output pair example, and obtaining an input data generation prompt.

[0066] Step 108, using a large language model to process the input data to generate prompts, obtain multiple input data, and store them in an input storage repository.

[0067] Step 110, extracting an input subset from the input repository and fusing it with the case input data to obtain an updated input data generation prompt and obtain an expanded input set.

[0068] Step 112, extracting synthetic input data from the extended input set, filling in an output data generation prompt template according to the synthetic input data and input-output pair examples, and obtaining an output data generation prompt.

[0069] Step 114 , using a large language model to perform context learning processing on the output data generation prompt, to generate a plurality of synthetic output data corresponding to the output data generation prompt.

[0070] Step 116, filling the input-output pair fine-tuning prompt template according to the synthesized output data, the synthesized input data and the input-output pair examples, so as to fine-tune the large language model according to the input-output pair fine-tuning prompt template.

[0071] In the above-mentioned negative case generation method for bidding documents based on a large language model, three types of prompt templates are constructed, namely, input data generation prompt template, output data generation prompt template, and input-output pair fine-tuning prompt template. Then, a self-iterative method is adopted. First, based on the input-output pair example of the negative case data of the bidding document, the input data generation prompt template is filled to generate the input data generation prompt. Then, the large language model is used to process the input data generation prompt to obtain multiple input data. After processing, synthetic input data is obtained to complete the first stage of generation. Then, the synthetic input data is transmitted to the output data generation prompt template to obtain the output data generation prompt. Then, the large language model is used to perform context learning processing on the output data generation prompt to generate multiple synthetic output data corresponding to the output data generation prompt to complete the second stage of generation. Finally, the synthetic output data and the original input-output pair example of the negative case data of the bidding document are used to synthesize the input-output pair of the negative case data of the bidding document, and finally the negative case fine-tuning prompt of the bidding document is generated to realize the fine-tuning of the large language model. Finally, the large language model is fine-tuned and updated to complete the third stage of generation. Through the above-mentioned method, the comprehensiveness of the case data set of bidding documents is improved, thereby ensuring the accuracy of the generation of negative cases in bidding documents.

[0072] In one of the embodiments, constructing an input data generation prompt template, an output data generation prompt template, and an input-output pair fine-tuning prompt template all include: prompt words, text to be filled, and identifier descriptions; prompt words are phrases used to guide the understanding and generation of a large language model; the text to be filled is used to fill in the original text; and identifiers are symbols or keywords that mark specific content or specific structures.

[0073] Specifically, the prompt word refers to a phrase used to guide the large language model to understand and generate, usually a task description. In this example, the prompt word part is represented by text, for example, when synthesizing input data, setting the task instruction = "Generate multiple new inputs based on <instructions> and example <input>; do your best to ensure that the new inputs you generate are different from the existing <input>, while ensuring that the new inputs are diverse, comprehensive, and of high quality."

[0074] The text to be filled refers to the user's original bidding document case text and other template inputs, which are represented by placeholders. For example, an input sample of a negative case in a bidding document = "The advanced technology of the smart blackboard system has won national science and technology awards."

[0075] Identifiers are symbols or keywords used to mark specific content or specific structures in the prompt template, which are used to help the large language model locate specific content through identifiers. For example, in the task description In the text, "<command>" and "<input>" are identifiers. Identifier description is a further explanation of a specific identifier. Indicates that the identifier description usually contains text to be filled in. For example, when synthesizing input data, the description of the identifier [instruction] is = "<Command>: Example <input> The incorrect keyword in the statement is { }, the detailed error reason is { }, the input you generate should have similar error keywords and error reasons"

[0076] This example constructs three types of templates, namely input data generation template, output data generation template, and input-output pair fine-tuning template. Each template is constructed as:

[0077] .

[0078] In one embodiment, in the data preprocessing part, the negative case table data of a certain enterprise's bidding document is obtained, including the fields of "statement", "legal", "keywords", and "case analysis". Among them, "statement" is the case statement in the bidding document, "legal" indicates whether the statement complies with the specification, the keyword indicates the key to judging the statement, and "case analysis" is the detailed judgment information of the statement. Extract the data with "legal" as no, and obtain the original data of negative cases in the bidding text. ,in is the number of samples, where .

[0079] In the prompt text generation section, use the prompt template ,The original data of negative bidding cases are used to generate input data generation prompts, output data generation prompts, and input-output pair fine-tuning prompt text data.

[0080] Input data to generate prompts Example:

[0081] ={"Generate multiple new inputs based on <instructions> and example <input>; do your best to ensure that the new inputs you generate are different from the existing <input>, while ensuring that the new inputs are diverse, comprehensive and of high quality",["<input>:{ }","<Command>: Example <input> The incorrect keyword in the statement is { }, the detailed error reason is {placeholder3}, the input you generate should have similar error keywords and error reasons"],["The advanced technology of the smart blackboard system has won national science and technology awards", awards, "The performance and awards of a specific administrative region or a specific industry are used as bonus conditions or conditions for winning the bid or closing the deal"]}.

[0082] Output data generation tips Example:

[0083] ={"Generate corresponding output based on user input according to specific <instructions>; for example; user: input={ }; Human helper: output = { }; Start now; User: Input = <input>; Human assistant: Output = ", ["<command>: For a given input statement, extract keywords and analyze the cause of the error", "<input>: { placeholder3}", ["The budget for equipment procurement in Section 2 is RMB 1 million, requiring the bidder to have a registered capital of no less than RMB 55 million; the budget for equipment procurement in Section 4 is RMB 2 million, requiring the bidder to have a registered capital of no less than RMB 60 million.","Keywords: Qualification review / registered capital; Cause of error: Blindly raising the threshold for registered capital, with the risk of less than three qualified bidders leading to a failed bid", "The budget for equipment procurement in Section 3 is RMB 2 million, requiring the bidder to have a registered capital of no less than RMB 75 million; the budget for equipment procurement in Section 6 is RMB 4 million, requiring the bidder to have a registered capital of no less than RMB 90 million"]

[0084] Input-Output Pair Fine-Tuning Tips Example:

[0085] ={"According to the specific <instruction>, for the given input, generate the corresponding output. Input={ }; output = { }",["<Command>: For a given input statement, extract keywords and analyze the cause of the error"],["For procurement documents and bidding documents, set "3 points for bid preparation", "2 points for staffing", and "10 points for overall strength" in the business evaluation items","Keywords: compilation issues / irrelevant conditions; Cause of error: The set qualifications, technology, and business conditions are irrelevant to contract performance."]}

[0086] In one embodiment, receiving an input word segmentation sequence , the large language model is based on the context in the Step to the next word Make a prediction and generate the probability distribution of the next word The calculation formula is:

[0087] ;

[0088] in The big language model comes first The sequence of all words that have been predicted. The large language model is The logits vector generated in the step is used to represent the original score value of the model for each word, and the probability conversion function is a function that transforms a logits vector into a probability distribution.

[0089] Specifically, the model base used in this example implementation is Vicuna-7b-1.5. The model base has been fine-tuned with the extensive and diverse user dialogue corpus from ShareGPT, and has a strong cross-task instruction following capability. It conforms to the negative case data generation scenario of bidding documents targeted by the present invention, and can handle input data generation tasks, output data generation tasks, etc. respectively.

[0090] Each sample of the input data text dataset and the output data text dataset All will be used as prompts to generate corresponding data. First, use the word segmenter to convert the input text data Processing word sequences Then, the autoregressive method is used to predict the distribution of the next word according to the instructions and prerequisites in the prompt text until the end symbol is generated, and the word segmentation sequence is obtained. Finally, the sub-sequences are merged to obtain the synthetic sample .

[0091] In another embodiment, the probability transfer function for:

[0092] ;

[0093] in is generated The score of the word vector, Get the probability corresponding to each word, is the temperature parameter.

[0094] Specifically, when When it is close to 1, the function This will make the calculated probabilities more evenly distributed, making the generated output more random and diverse. Using a higher temperature when generating input data makes the synthesized input data more diverse.

[0095] when When it approaches 0, the function The probability of the word vector with the highest score becomes very large, while the scores of other word vectors become very small, making the results generated by the model more certain. A lower temperature is used when generating output data to make the analysis of each case input more accurate.

[0096] After obtaining the probability distribution, the model uses the kernel sampling method to generate the next word. In kernel sampling, the model sorts the probabilities from high to low and selects a set of words with a cumulative probability of The words are used as candidate words, and then the next word is randomly sampled from them as the generated content. This method can avoid sampling some inappropriate or irrelevant words.

[0097] When synthesizing input data, set the temperature parameter Set the kernel sampling parameter to 0.7 is 0.5. In one example:

[0098] Input data: If the bidder has similar projects with a contract value of more than 1 million in the past two years (2014-2015), each performance will be scored 1 point, with a maximum of 2 points.

[0099] Model synthesis input data: If the bidder has similar project performance with a contract value of more than 2 million in the past three years (2020-2023), each performance will be awarded 2 points, with a maximum of 6 points

[0100] When generating output data, set the temperature parameter Set the kernel sampling parameter to 0.3 is 0.8. In one example:

[0101] Input data: "If the bidder has similar project performance with a contract value of more than 2 million in the past three years (2020-2023), each performance will be awarded 2 points, with a maximum of 6 points."

[0102] Model synthesis output data: "Keywords: operating performance; Error reason: similar project performance with a contract amount of more than 2 million, which has nothing to do with the project itself, and excludes or discriminates against potential suppliers with unreasonable conditions"

[0103] In one of the embodiments, input data or synthesized output data is filtered by a preset noise filter and a length filter; wherein the noise filter filters out noise items in the input data or synthesized output data by using a preset noise term list; and the length filter filters the text length of the input data or synthesized output data by using a preset length range.

[0104] Specifically, in this example, in the noise filter, a noise term list containing common greetings, hellos and noise characters is manually managed. If any noise term from this filter list appears in the input or output of a generated example, the entire example is discarded. And a list of noise terms is extracted from the discarded examples. Noise characters that are not in the list are added to the noise term list This filtering step ensures that the generated examples remain concise, focused, and free of irrelevant content. In this example, in the length filter, by analyzing the input and output data of negative cases in bidding documents, it is found that they conform to the normal distribution of the specific task. Therefore, the input and output lengths of the generated examples are set to be within the mean calculated by the examples. and variance Within the determined range, , and discard outputs that do not conform to this length range. By length filtering, we ensure that the length of the generated examples remains within a reasonable range, thereby improving the quality and effectiveness of the dataset.

[0105] In one embodiment, a cross entropy loss function is constructed:

[0106] ;

[0107] in, is the sequence of all words that the model has predicted, is the value that should be generated currently, For a given parameter Under the premise that the current predicted value is the value that should be generated at present The probability of fine-tuning is calculated during the back-propagation process by calculating the input-output loss value of the fine-tuning prompt template and performing gradient updates on the large language model for fine-tuning.

[0108] Specifically, although the base Vicuna-7b-1.5 used in this example has undergone a lot of training, its performance is still insufficient when facing the bidding document field. Therefore, this example uses the original and synthesized bidding document negative case data input-output pairs as input, fine-tunes the bidding text negative case model, completes the self-iteration of the model, and continuously enhances its ability to synthesize negative case data of bidding forms.

[0109] Specifically, the constructed bidding text negative case input-output pair is used for each sample in the dataset. As the input for generating fine-tuning. First, use the tokenizer to convert the fine-tuning text Processing word sequences , and then using the autoregressive method, the large model will start predicting from the first word, and then according to the method in step 2, the model predicts the probability distribution of the next word based on the predicted word, and selects the next word until the end symbol is generated and stops.

[0110] In summary, Figure 2 A framework diagram for generating negative cases in bidding documents is provided. The invention provides a self-iterative method for generating negative cases in bidding documents based on a large language model. By constructing a prompt template, negative case data of bidding documents are synthesized according to scenario requirements; a large language model pre-trained with a large amount of public text data is deployed to generate negative case input data, and the quality of input data is improved by combining a rule-based filtering strategy, thereby expanding and generating a high-quality bidding negative case input data set that meets the requirements; output data is generated for each input data by pre-training a large language model, and the quality of output data is improved by combining a rule-based filtering strategy, thereby realizing the construction of input-output pairs of negative case data of bidding documents. The scheme utilizes the ability of prompt templates to construct contextual cases to stimulate the large model, and utilizes the semantic understanding and generation ability of the large language model pre-trained with a large amount of public text data to perform effective data generation. It can generate data for scenario requirements, solve the problems of scarcity of training data, poor quality of synthesized data, high training cost, etc. in the synthesis of negative case data of bidding documents, and realizes the construction of low-cost, high-efficiency and high-quality data sets.

[0111] In addition, using large model fine-tuning technology, we specifically designed input-output data prompt templates, and used the labeled bidding document negative case data set to fine-tune the pre-trained large language model base to build a bidding document negative case model. By building a bidding document negative case generation model through fine-tuning technology, we can efficiently adapt the model to specific bidding document specification scenarios and reduce the resource requirements for model construction. Through the powerful semantic understanding and generation capabilities of the pre-trained large language model, we can expand the negative cases of bidding documents, thereby improving the comprehensiveness of the bidding document case data set, so that we can train a model that can automatically perform compliance checks on bidding documents through the bidding document case data set.

[0112] In data processing, two rounds of rule-based data filtering are used, one after generating input data and the other after generating the corresponding output for each input. By designing noise filters and length filters, the content of the generated input and output is controlled to ensure that the generated examples are concise and focused, without irrelevant content or patterns, and that the length distribution of the generated data is close to the original examples.

[0113] In a further scheme, the diversity and strategy of generating data are improved by controlling the temperature balance of the large language model, and using different relative temperatures in the input process and the process of generating output, respectively, to improve the diversity and quality of the generated data.

[0114] final, Figure 3 A schematic diagram of negative cases for generating bidding documents is given.

[0115] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0116] In one embodiment, a device for generating negative cases of bidding documents based on a large language model is provided, comprising:

[0117] A case acquisition module is used to acquire a negative case data set of bidding documents; the negative case data set of bidding documents includes negative cases of bidding documents;

[0118] A template construction module, used to respectively construct an input data generation prompt template, an output data generation prompt template, and an input-output pair fine-tuning prompt template;

[0119] An input data generation module is used to extract negative cases of bidding documents from the negative case data set of bidding documents, and construct an input-output pair example of negative case data of bidding documents, and fill in an input data generation prompt template according to the case input data of the input-output pair example to obtain an input data generation prompt;

[0120] An input data storage module, used to process the input data generation prompt using a large language model to obtain multiple input data and store them in an input storage repository;

[0121] An expansion module is used to extract an input subset from the input repository and fuse it with the case input data to obtain an updated input data generation prompt to obtain an expanded input set;

[0122] An output data generation module, used for extracting synthesized input data from the extended input set, filling the output data generation prompt template according to the synthesized input data and the input-output pair example, and obtaining the output data generation prompt;

[0123] an output data synthesis module, configured to perform context learning processing on the output data generation prompt using a large language model, and generate a plurality of synthesized output data corresponding to the output data generation prompt;

[0124] A fine-tuning module is used to fill an input-output pair fine-tuning prompt template according to the synthesized output data, the synthesized input data and the input-output pair examples, so as to fine-tune the large language model according to the input-output pair fine-tuning prompt template.

[0125] In one of the embodiments, constructing an input data generation prompt template, an output data generation prompt template, and an input-output pair fine-tuning prompt template all include: a prompt word, a text to be filled, and an identifier description; the prompt word is a phrase used to guide the understanding and generation of a large language model; the text to be filled is used to fill in the original text; the identifier is a symbol or keyword that marks specific content or a specific structure.

[0126] In one embodiment, receiving an input word segmentation sequence , the large language model is based on the context in the Step to the next word Make a prediction and generate the probability distribution of the next word The calculation formula is:

[0127] ;

[0128] in The big language model comes first The sequence of all words that have been predicted. The large language model is The logits vector generated in the step is used to represent the original score value of the model for each word, and the probability conversion function is a function that transforms a logits vector into a probability distribution.

[0129] In one embodiment, the probability transfer function for:

[0130] ;

[0131] in is generated The score of the word vector, Get the probability corresponding to each word, is the temperature parameter.

[0132] In one of the embodiments, input data or synthesized output data is filtered by a preset noise filter and a length filter; wherein the noise filter filters out noise items in the input data or synthesized output data by using a preset noise term list; and the length filter filters the text length of the input data or synthesized output data by using a preset length range.

[0133] In one embodiment, the fine-tuning module is also used to construct a cross entropy loss function:

[0134] ;

[0135] in, is the sequence of all words that the model has predicted, is the value that should be generated currently, For a given parameter Under the premise that the current predicted value is the value that should be generated at present The probability of fine-tuning is calculated during the back-propagation process by calculating the input-output loss value of the fine-tuning prompt template and performing gradient updates on the large language model for fine-tuning.

[0136] For the specific limitations of the device for generating negative cases in bidding documents based on a large language model, please refer to the limitations of the method for generating negative cases in bidding documents based on a large language model in the above text, which will not be repeated here. Each module in the above-mentioned device for generating negative cases in bidding documents based on a large language model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0137] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating negative cases of bidding documents based on a large language model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0138] Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0141] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating negative cases in bidding documents based on a large language model, characterized in that: The method comprises: Obtaining a negative case data set of bidding documents; the negative case data set of bidding documents includes negative cases of bidding documents; Constructing input data generation prompt template, output data generation prompt template, and input-output pair fine-tuning prompt template respectively; Extracting negative cases of bidding documents from the negative case data set of bidding documents, and constructing an input-output pair example of negative case data of bidding documents, and filling an input data generation prompt template according to the case input data of the input-output pair example to obtain an input data generation prompt; Using a large language model to process the input data to generate prompts, obtain multiple input data, and store them in an input storage repository; Extracting an input subset from the input repository and fusing it with the case input data to obtain an updated input data generation prompt and obtain an extended input set; Extracting synthetic input data from the extended input set, filling the output data generation prompt template according to the synthetic input data and the input-output pair example, and obtaining the output data generation prompt; Using a large language model to perform context learning processing on the output data generation prompt to generate a plurality of synthetic output data corresponding to the output data generation prompt; populating an input-output pair fine-tuning hint template according to the synthesized output data, the synthesized input data, and the input-output pair examples to fine-tune a large language model according to the input-output pair fine-tuning hint template; The steps of generating input data and output data of the large language model include: Receive input word sequence , the large language model is based on the context in the Step to the next word Make a prediction and generate the probability distribution of the next word The calculation formula is: in The big language model comes first The sequence of all words that have been predicted. The large language model is The logits vector generated in the step is used to represent the original score value of the model for each word, and the probability conversion function is a function that converts the logits vector into a probability distribution; Fine-tune the large language model based on the input-output pair fine-tuning prompt template, including: Construct the cross entropy loss function: in, is the sequence of all words that the model has predicted, is the value that should be generated currently, For a given parameter Under the premise that the current predicted value is the value that should be generated at present probability; During the back-propagation process, the input-output loss value of the fine-tuned prompt template is calculated to perform gradient updates on the large language model for fine-tuning.

2. The method according to claim 1, characterized in that The input data generation prompt template, the output data generation prompt template, and the input-output pair fine-tuning prompt template all include: prompt words, text to be filled, and identifier description; The prompt word is a phrase used to guide the understanding and generation of the large language model; The text to be filled is used to fill in the original text; The identifier is a symbol or a keyword that marks specific content or a specific structure.

3. The method according to claim 1, characterized in that The probability transfer function for: in is generated The score of the word vector, Get the probability corresponding to each word, is the temperature parameter.

4. The method according to claim 1, characterized in that: After obtaining a plurality of pieces of input data or generating a plurality of pieces of synthesized output data corresponding to the output data generation prompt, the method further includes: Filter the input data or synthesized output data through pre-set noise filters and length filters; The noise filter is used to filter out noise items in the input data or synthesized output data through a preset noise term list; and the length filter is used to filter the text length of the input data or synthesized output data through a preset length range.

5. A device for generating negative cases in bidding documents based on a large language model, used to implement the method for generating negative cases in bidding documents based on a large language model according to any one of claims 1 to 4, characterized in that: The device comprises: A case acquisition module is used to acquire a negative case data set of bidding documents; the negative case data set of bidding documents includes negative cases of bidding documents; A template construction module, used to respectively construct an input data generation prompt template, an output data generation prompt template, and an input-output pair fine-tuning prompt template; An input data generation module is used to extract negative cases of bidding documents from the negative case data set of bidding documents, and construct an input-output pair example of negative case data of bidding documents, and fill in an input data generation prompt template according to the case input data of the input-output pair example to obtain an input data generation prompt; An input data storage module, used to process the input data generation prompt using a large language model to obtain multiple input data and store them in an input storage repository; An expansion module is used to extract an input subset from the input repository and fuse it with the case input data to obtain an updated input data generation prompt to obtain an expanded input set; An output data generation module, used for extracting synthesized input data from the extended input set, filling the output data generation prompt template according to the synthesized input data and the input-output pair example, and obtaining the output data generation prompt; an output data synthesis module, configured to perform context learning processing on the output data generation prompt using a large language model, and generate a plurality of synthesized output data corresponding to the output data generation prompt; A fine-tuning module is used to fill an input-output pair fine-tuning prompt template according to the synthesized output data, the synthesized input data and the input-output pair examples, so as to fine-tune the large language model according to the input-output pair fine-tuning prompt template.

6. The device according to claim 5, characterized in that The construction of the input data generation prompt template, the output data generation prompt template, and the input-output pair fine-tuning prompt template all include: prompt words, text to be filled, and identifier descriptions; the prompt words are phrases used to guide the understanding and generation of the large language model; the text to be filled is used to fill in the original text; the identifier description is a symbol or keyword that marks specific content or specific structure.

Images