Bid invitation file semi-structure automatic generation method and device based on large language model prompt fine tuning

Through the tip fine-tuning method based on the large language model, combined with various technical means to preprocess and feature extraction of bidding documents, the problem of slow preparation speed and insufficient accuracy of bidding documents is solved, and efficient and accurate automatic generation of bidding documents is achieved. Improve file quality and model understanding ability.

CN118917281APending Publication Date: 2024-11-08曹琼瑶 +2
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410793037.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art has problems such as slow speed, untimely updates and poor management in the compilation of bidding documents, and it is difficult to effectively refine and summarize the text content with rich professional terms, resulting in insufficient file quality and accuracy.

Method used

The prompt fine-tuning method based on the large language model is adopted, combining directed ring-free word graphs, hidden Markov model, TextRank algorithm, P-tuning prompt fine-tuning, secondary MASK mechanism and filtering attention mechanism to carry out semi-structured bidding documents Automatic generation, through these technical means, pre-processing, key phrase extraction, feature extraction and model fine-tuning of bidding documents to generate high-quality bidding documents.

Benefits of technology

It improves the accuracy and efficiency of the generation of bidding documents, reduces human errors, ensures the integrity and relevance of the documents, enhances the model's ability to understand complex texts, and can quickly identify and extract core information content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917281A_ABST
    Figure CN118917281A_ABST
Patent Text Reader

Abstract

The invention discloses a bid invitation file semi-structure automatic generation method and device based on large language model prompt fine tuning, and the method comprises the steps: 1, carrying out the preprocessing of bid invitation file data based on a directed acyclic word graph and a hidden Markov model; 2, based on a TextRank unsupervised key phrase extraction method, key phrases are extracted from the preprocessed bid invitation file data, and annotation prompting is carried out on key content of the bid invitation file; 3, on the basis of a P-tuning prompt fine tuning method, modeling is conducted on the bid invitation file data and the key phrases through a large language model, and different initialization parameters are set; a secondary MASK mechanism is carried out on bid invitation file data to eliminate learning deviation in a model fine adjustment process and improve feature extraction accuracy, and feature extraction is still carried out on key phrases in a traditional mode; step 5, constructing a filtering attention mechanism, and performing certain filtering on a feature vector embedding result: removing useless interference information and retaining key information; and step 6, based on a maximum edge correlation algorithm, combining the basic bid invitation file subjected to semi-structured processing with the key information in the extracted bid invitation file, and outputting to obtain a final bid invitation file.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to extracting key contents of tender documents and automatically generating tender documents for specific industries in combination with relevant templates, and more specifically to a method and device for automatically generating semi-structured tender documents based on large language model prompt fine-tuning and text summaries. Background Art

[0002] Bidding is an important part of enterprise management, and the preparation of bidding documents is the key factor in the success of bidding. The quality of bidding documents directly determines the success or failure of bidding and procurement. How to improve the quality of bidding documents is one of the core issues in the bidding industry.

[0003] Since manually compiled standard bidding documents currently generally have problems such as slow compilation, untimely updates, and inaccurate management, with the development and maturity of artificial intelligence technologies such as natural language processing and machine learning, it has become possible to adopt intelligent compilation and updating methods.

[0004] In the modern bidding management process, the application of automatic document generation systems has gradually become popular. In particular, semi-structured bidding document automatic generation systems effectively combine the rigor of structured data with the flexibility of unstructured data. When creating documents, such systems can fix certain key fields, such as bidding numbers, project names, and deadlines, to ensure the accuracy and consistency of this information. At the same time, it also supports flexible text input, such as project descriptions and specific requirements, which can be personalized according to the specific needs of different projects. Through this semi-structured approach, the bidding document automatic generation system not only improves the efficiency of document preparation, but also maintains the customization and wide applicability of the content, greatly promoting the standardization and automation of bidding activities.

[0005] Automatic text generation refers to the use of natural language processing technology and language knowledge by computers to automatically generate text information such as reports, news, and summaries based on potential non-linguistic information. The technical parameters in the bidding documents will involve many professional terms, and their content is closely related to the industry field. How to effectively extract and summarize from many related texts is the first problem to be solved.

[0006] The large language model has been pre-trained on a wide range of text data and has learned rich language representations and contextual relationships, enabling it to understand complex tender document content, effectively extract key information from the text, and reduce the occurrence of text-related errors, with the advantages of high performance and high efficiency.

[0007] Text summarization technology based on large language models can quickly extract key information from a large amount of original documents and data, automatically and quickly identify core content, and generate content summaries. When combined with template content, it can automatically generate bidding documents, greatly reducing human resources and reducing the risks caused by errors. Summary of the invention

[0008] The present invention aims at the above-mentioned practical problems, solves the shortcomings of the prior art, and provides a method and device for automatically generating a semi-structured bidding document based on a large language model prompt fine-tuning. After carefully analyzing the technical field of the problem to which this type of problem belongs, the present invention can provide a theoretical basis and technical support for the automatic generation of bidding texts for relevant personnel.

[0009] The technical solution of the present invention is as follows:

[0010] The method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning includes the following steps:

[0011] Step 1: Use Directed Acyclic Word Graph (DAWG) and Hidden Markov Model (HMM) to pre-process the relevant bidding documents corresponding to the project category. The relevant documents include bidding document samples and related materials, which are accurately identified and irrelevant data are eliminated.

[0012] Step 2: Apply an unsupervised learning method based on TextRank to extract key phrases from the bidding documents. This can highlight the key content in the bidding documents, provide annotation hints for fine-tuning the large language model, and improve the model's ability to perceive key information.

[0013] Step 3: Use the P-tuning prompt fine-tuning method to deeply model the bidding documents and their key phrases using a large pre-trained language model. Set targeted initialization parameters so that the model can perform accurate feature extraction and content understanding in a wider context.

[0014] Step 4: The bidding document data is processed using a secondary MASK mechanism to eliminate possible learning biases that may occur during model fine-tuning, thereby improving the accuracy of feature extraction of the large language model. The key phrases obtained in step 3 are extracted using traditional methods to ensure comprehensiveness and depth of processing.

[0015] Step 5: Construct a filtering attention mechanism to finely filter the feature vector embedding results obtained in step 4. By removing irrelevant interference information while retaining key information, the output quality of the model is further optimized.

[0016] Step 6: Based on the maximum margin relevance algorithm, the key content about the tender obtained after the large language model is processed is effectively combined with the basic version of the tender document obtained by the large language model through semi-structured processing of the tender document data, and the final tender document is output. This ensures the integrity and correctness of the tender document, and also uses the advanced semantic understanding ability of the large language model to improve the relevance and innovation of the generated content.

[0017] Preferably, in step 1 described in the present invention, first, a directed acyclic word graph is used to quickly index the relevant bidding documents to achieve spelling checking and recognition of repeated patterns. This technology can effectively identify and remove repeated phrases and words, significantly reducing data redundancy. Subsequently, a hidden Markov model is used to identify and filter out noise data that does not conform to the expected language model, such as incorrect words or phrases. In addition, the model deeply understands the sentence context, effectively identifies and removes text fragments that are contextually irrelevant or logically incoherent, thereby improving the overall quality and consistency of the text.

[0018] Preferably, in step 2, the tender document data preprocessed in step 1 is converted into a graph structure based on the TextRank algorithm, the nodes in the graph represent words or phrases in the text, the edges represent the co-occurrence relationship between words or phrases, and the word or phrase corresponding to the node with the highest score is selected as the key phrase of the text by using an iterative score ranking method.

[0019] Preferably, in step 3, based on the P-tuning prompt fine-tuning method, different initialized prompt parameters are set for the text data of the tender document and the key phrase text generated in step 2 to optimize the large language model, so as to better establish the connection between the tender document and the key phrase and improve the encoding and decoding quality.

[0020] Preferably, in step 4, in order to improve the accuracy of feature extraction and the learning deviation in the fine-tuning process, for the bidding document data, a secondary MASK mechanism is used to randomly select 80% of the input data and replace them with special MASK tags, so as to obtain a more accurate feature vector. For key phrases, feature extraction is performed in a traditional way, and the two parts of feature extraction results are combined.

[0021] Preferably, in step 5, the feature vector embedding result is transformed to form a weight matrix, important content is set to a larger weight, and secondary content is set to a smaller weight. Through the activation function, the activated weight matrix is ​​obtained, and then combined with the input feature vector embedding result to obtain the filter factor. Finally, the filter factor is concatenated with the feature vector embedding result and linearly transformed to complete the entire filtering attention mechanism process.

[0022] Preferably, in step 6, the sentences are iteratively selected by the Maximum Marginal Relevance (MMR) algorithm, and the key information of the relevant bidding documents is combined with the basic version of the bidding documents obtained after semi-structured processing. In each iteration, the MMR value of each sentence to be selected is calculated, that is, the MMR value of each sentence to be selected is combined with its relevance to the original text and the information duplication with the selected sentences, and finally the sentence with the highest MMR value is added to the new bidding document to form the final version of the bidding document.

[0023] The second aspect of the present invention relates to a device for automatic generation of semi-structured tender documents based on large language model prompts and fine-tuning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for automatic generation of semi-structured tender documents based on large language model prompts and fine-tuning of the present invention.

[0024] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for automatic semi-structure generation of bidding documents based on large language model prompt fine-tuning of the present invention.

[0025] The beneficial effects of the present invention are as follows:

[0026] 1. Make full use of the large language model's ability to understand complex text data, and apply the large language model to the downstream task of automatically generating bidding documents through the prompt fine-tuning method. Compared with traditional manual editing methods, it improves accuracy and reduces human errors.

[0027] 2. By improving some modules in the large language model, the model can quickly identify and extract core information content from complex and diverse bidding documents, providing accurate input for generating the main part of the bidding documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an implementation flow chart of the method of the present invention. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0031] Implementation Example 1

[0032] like Figure 1 As shown, the method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning of the present invention comprises the following steps:

[0033] Step 1: Use directed acyclic word graph and hidden Markov model to pre-process the bidding documents;

[0034] In the process of automatic generation of bidding documents, accurate and efficient text preprocessing is a key step to ensure the quality of documents. This process involves two core technologies: Directed Acyclic Word Graph (DAWG) and Hidden Markov Model (HMM). First of all, the use of Directed Acyclic Word Graph (DAWG) greatly improves the efficiency of text processing. DAWG is an optimized data structure specially designed for performing fast search, spell checking and repeated pattern recognition. In the preprocessing stage of bidding documents, this structure is particularly helpful for quickly identifying and deleting repeated phrases and words, significantly reducing the redundancy of data. By effectively managing repeated elements in the text, DAWG not only improves the speed of data processing, but also ensures the refinement and clarity of the text by reducing unnecessary repetitive content.

[0035] Secondly, Hidden Markov Model (HMM) plays a central role in the further processing of text. HMM is a powerful statistical model used to identify and filter out noisy data that does not conform to the predefined language model, such as wrong words or phrases. In addition, the ability of HMM is not limited to simple error recognition. It is also able to understand the context in the sentence and further identify and eliminate text fragments that are contextually irrelevant or logically unreasonable. This is especially important when processing tender documents, because these documents usually require a high degree of logic and accuracy, and any contextual errors or inconsistencies may lead to misunderstandings or legal risks.

[0036] Step 2: Using the TextRank algorithm, extract key phrases from the preprocessed bidding document data;

[0037] The TextRank algorithm is a graph-based ranking algorithm that has good results in extracting key phrases. When applying the TextRank algorithm, you first need to segment the text into words or phrases, and identify the co-occurrence relationship between these elements to effectively capture the correlation between words. Secondly, after building the graph structure, the importance score of each node is calculated iteratively, and the iteration is stopped when the scores of all nodes converge to a stable value. Finally, the node with the highest score is selected, and the corresponding word or phrase is used as the key phrase of the text. In this way, key phrases are extracted from the bidding document data, the model's perception of key information is improved, and help is provided for the next step of model fine-tuning.

[0038] Step 3: Based on the P-Tuning prompt learning fine-tuning method, the bidding document data and key phrases are set with different initialization prompts to learn and tune the large language model;

[0039] Prompted fine-tuning methods use learnable prompts to guide large language models, so that the models can better adapt to downstream tasks.

[0040] In this method, prompt learning templates are set for bidding document data and key phrases respectively. These templates not only provide task-related contextual information, but also contain a small number of adjustable parameters, so that the model can efficiently capture the text features in the bidding documents with the least adjustment of parameters.

[0041] By applying optimizations separately on tender document data and key phrases, our approach enables large language models to extract the most relevant information for both types of text. This separate optimization strategy allows the model to deeply explore the potential correlation features between text and phrases while maintaining high efficiency. As a result, the model is able to generate higher quality latent vector representations, significantly improving the quality of the encoding and decoding processes.

[0042] Since the implicit association between the bidding document data and the prompt phrase mainly comes from the upper-level semantics after induction and abstraction, and the feature difference between the two is relatively small at the underlying word embedding layer, shared parameters are used at the bottom of the model to reduce the overall computational burden and video memory requirements. Shared parameters not only reduce the resource consumption of model training, but also speed up the training process, while maintaining the flexibility and efficiency of the model in processing different text types.

[0043] Step 4: The bidding document data is processed using the secondary MASK mechanism for feature extraction, and the key phrases are extracted using traditional methods;

[0044] In order to improve the accuracy of feature extraction and reduce learning bias during fine-tuning, this method uses an improved secondary MASK mechanism to process the bidding document data. Specifically, we randomly select 80% of the input data tags and replace them with a special [MASK] tag. This method can force the model to rely more on contextual information to predict masked words, thereby promoting the model to learn deeper and more accurate feature vectors.

[0045] For key phrase processing, we maintain the traditional feature extraction method, which includes using standard methods such as TF-IDF or word embedding technology to ensure that the feature vector of the key phrase can accurately reflect its semantic importance and the core concept in the text.

[0046] After feature extraction, the feature vector of the tender document data is combined with the feature vector of the key phrase. This combination is done through a specific algorithm that not only takes into account the respective feature weights, but also adjusts the contribution of each feature in the final representation through an optimization algorithm to achieve the best prediction performance.

[0047] By combining the secondary MASK mechanism with traditional feature extraction technology, we can greatly improve the richness and expressiveness of the bidding document data features while maintaining the accuracy of key phrase features. This not only helps the model to show better performance in subsequent applications, but also ensures the stability and reliability of the entire learning process.

[0048] Step 5: Construct a filtering attention mechanism to filter the feature vector embedding results: retain key information and remove interference content;

[0049] Although the bidding document data has been effectively preprocessed, the types of related documents are quite diverse, and their impact on the final bidding documents varies. Therefore, for the bidding document data information, correctly and efficiently distinguishing information of different importance and assigning appropriate weights is the key to improving model performance, and thus constructing a filtering attention mechanism can do this well. In this method, the second-class feature vector embedding result obtained in step 4 is first converted into a weight matrix. In this matrix, we assign different weights to the elements in each feature vector according to the predefined importance score. The elements of the core content receive larger weights, while the information less related to the topic is assigned smaller weights. This distinction ensures that the model can focus on the key information in the bidding documents.

[0050] Secondly, the weight matrix is ​​obtained by processing the weights through the ReLU activation function. The application of the activation function not only introduces nonlinear processing and enhances the expressiveness of the model, but also helps the model to more effectively distinguish important and unimportant information in subsequent processing.

[0051] Finally, the activated weight matrix is ​​combined with the input feature embedding vector to form a filter factor. In order to further remove the non-important information in the embedding vector, this method extracts information from the preliminary extraction results: the filter factor is concatenated with the feature embedding vector and linearly transformed to finally complete the feature information filtering.

[0052] Step 6: Using the maximum edge correlation algorithm, the key sentences are combined with the basic bidding documents obtained by semi-structured processing to form the final bidding documents;

[0053] The maximum margin relevance algorithm selects key sentences through effective iteration, balancing the relevance of the candidate sentences with the original text and the information repetition of the selected sentences. First, an empty text set and a pool of candidate sentences are set. In each iteration, each sentence is evaluated, the relevance of each candidate sentence with the original text is calculated, and the information repetition of the candidate sentence and all sentences in the selected sentence set is evaluated. Combining the two metrics, the MMR value is calculated for each sentence, and finally the sentence with the highest MMR value is selected from the candidate sentence pool and added to the final text. Repeat multiple times to generate the final optimal text.

[0054] Step 7: Apply the method of the present invention to the bidding document to realize automatic generation of the bidding document;

[0055] Relevant documents and materials are selected to form a text library, including the bidding templates of relevant industries, relevant laws and regulations, industry demand specifications, etc. The text library is first preprocessed and then input into the model in this method, and finally the bidding documents of the relevant industries are output. The relevant key information can be extracted more accurately from the content of the bidding documents, the text sentences are correct, and they comply with relevant laws and regulations, achieving good results.

[0056] Example 2

[0057] The present embodiment relates to a device for automatic generation of semi-structured tender documents based on large language model prompts and fine-tuning, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for automatic generation of semi-structured tender documents based on large language model prompts and fine-tuning of embodiment 1.

[0058] Example 3

[0059] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for automatic semi-structure generation of bidding documents based on large language model prompt fine-tuning of embodiment 1 is implemented.

[0060] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this implementation example, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for automatic semi-structured generation of bidding documents based on large language model prompt fine-tuning, characterized in that: The following steps are involved: Step 1: Use Directed Acyclic Word Graph (DAWG) and Hidden Markov Model (HMM) to pre-process the relevant bidding documents corresponding to the project category. The relevant documents include bidding document samples and related materials, and accurately identify and eliminate irrelevant data; Step 2: Apply an unsupervised learning method based on TextRank to extract key phrases; highlight the key content in the bidding documents, provide annotation prompts for fine-tuning the large language model, and improve the model's ability to perceive key information; Step 3: Use the P-tuning prompt fine-tuning method to deeply model the bidding documents and their key phrases using a large pre-trained language model; set targeted initialization parameters so that the model can perform accurate feature extraction and content understanding in a wider context; Step 4: The bidding document data is processed using a secondary MASK mechanism to eliminate the learning bias that occurs during model fine-tuning. The key phrases obtained in step 3 are extracted using traditional methods to ensure comprehensiveness and depth of processing. Step 5: Construct a filtering attention mechanism to finely filter the feature vector embedding results obtained in step 4; further optimize the output quality of the model by removing irrelevant interference information while retaining key information; Step 6: Based on the maximum marginal relevance algorithm, the key content about the bidding obtained after processing by the large language model is effectively combined with the basic version of the bidding document obtained by the large language model through semi-structured processing of the bidding document information, and the final bidding document is output; the integrity and correctness of the bidding document are ensured, and the advanced semantic understanding ability of the large language model is used to improve the relevance and innovation of the generated content.

2. According to claim 1, a method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning is characterized in that: Step 1 includes: first, using a directed acyclic word graph to quickly index relevant bidding documents to achieve spelling checking and recognition of repeated patterns; then, a hidden Markov model is used to identify and filter out noise data that does not conform to the expected language model, such as incorrect words or phrases; in addition, the hidden Markov model deeply understands the sentence context, effectively identifies and removes text fragments that are contextually irrelevant or logically incoherent, thereby improving the overall quality and consistency of the text.

3. The method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning according to claim 1 is characterized in that: In step 2, the tender document data preprocessed in step 1 is converted into a graph structure based on the TextRank algorithm. The nodes in the graph represent words or phrases in the text, and the edges represent the co-occurrence relationship between words or phrases. The word or phrase corresponding to the node with the highest score is selected as the key phrase of the text by using the iterative score ranking method.

4. According to claim 1, a method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning is characterized in that: In step 3, based on the P-tuning prompt fine-tuning method, different initialized prompt parameters are set for the text data of the tender document and the key phrase text generated in step 2 to optimize the large language model, establish the connection between the tender document and the key phrase, and improve the encoding and decoding quality.

5. The method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning according to claim 1 is characterized in that: In step 4, in order to improve the accuracy of feature extraction and the learning bias in the fine-tuning process, for the bidding document data, the secondary MASK mechanism is used to randomly select 80% of the input data and replace them with special MASK tags to obtain more accurate feature vectors; for key phrases, the traditional method is used for feature extraction, and the two parts of feature extraction results are combined.

6. The method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning according to claim 1 is characterized in that: In step 5, the feature vector embedding result is transformed to form a weight matrix, and the important content is set to a larger weight, while the secondary content is set to a smaller weight. Through the activation function, the activated weight matrix is ​​obtained, and then combined with the input feature vector embedding result to obtain the filter factor. Finally, the filter factor is concatenated with the feature vector embedding result and linearly transformed to complete the entire filtering attention mechanism process.

7. The method for automatically generating semi-structured bidding documents based on large language model prompt fine-tuning according to claim 1 is characterized in that: In step 6, the sentences are iteratively selected through the Maximum Marginal Relevance (MMR) algorithm, and the key information of the relevant bidding documents is combined with the basic version of the bidding documents obtained after semi-structured processing; In each iteration, the MMR value of each candidate sentence is calculated, that is, its relevance to the original text and the degree of information duplication with the selected sentences are combined, and finally the sentence with the highest MMR value is added to the new bidding document to form the final version of the bidding document.

8. A semi-structured automatic generation device for bidding documents based on large language model prompt fine-tuning, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the method for automatic semi-structure generation of bidding documents based on large language model prompt fine-tuning as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the method for automatic semi-structure generation of bidding documents based on large language model prompt fine-tuning as described in any one of claims 1-7 is implemented.

Citation Information

Cited By

  • Bidding document negative case generation method and device based on large language model

    CN119358530A

  • Method and device for generating negative cases in bidding documents based on large language model

    CN119358530B

  • Bid inviting and tendering file generation and automatic review method and system based on large language model

    CN121480470A