Bidding document data set construction method and device, electronic equipment and storage medium

Abstract

The invention discloses a bidding document data set construction method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps that a bidding document template and a historical bidding document data set are obtained, the bidding document template at least comprises N levels of titles, the historical bidding document data set at least comprises M historical bidding document files, N and M are integers larger than 1, and each historical bidding document file at least comprises multiple levels of titles; calculating semantic similarity between all titles in the bidding document template and all titles in the historical bidding document file; according to the semantic similarity, titles conforming to the bidding document template are screened out from the historical bidding document data set to be assembled, a target reference text is obtained, and the titles in the target reference text are arranged according to the title level sequence; and determining a target bidding document data set according to the target reference text. The technical problem that in the prior art, the effect of building a bidding document fine adjustment data set in the vertical field is poor is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for constructing a bid document data set. Background Art

[0002] In the context of the rapid development of artificial intelligence today, fine-tuning of large models for specific vertical fields has become a key step to improve the application effect of large models. However, in the existing technology, there are several challenges and limitations in the construction of bid fine-tuning datasets for large models in vertical fields.

[0003] First, traditional bid generation methods usually rely on manual annotation and rule matching, which is not only time-consuming and labor-intensive, but also difficult to ensure the diversity and coverage of the data set, and the consistency and accuracy of manual operations are difficult to guarantee, which directly affects the effect of subsequent model training. Secondly, the existing similarity calculation methods often fail to accurately capture the semantic similarity between titles when dealing with complex semantic bid generation tasks, which is crucial for generating high-quality bid texts that meet industry standards and actual needs. In the absence of effective semantic understanding capabilities, the generated bid texts may be greatly reduced in accuracy and relevance. In addition, there is currently a lack of a standardized process to build an efficient and high-quality fine-tuning dataset, which limits the application potential of generative artificial intelligence technology in areas such as rapid bid generation, resulting in poor application of the model in specific industries.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a method, device, electronic device and storage medium for constructing a bid document dataset, so as to at least solve the technical problem of poor effect of constructing a bid document fine-tuning dataset for vertical fields in the prior art.

[0006] According to one aspect of the present application, a method for constructing a bid document dataset is provided, comprising: obtaining a bid document template and a historical bid document dataset, wherein the bid document template includes at least N levels of titles, and the historical bid document dataset includes at least M historical bid document files, wherein N and M are integers greater than 1, and each historical bid document file includes at least multiple levels of titles; calculating the semantic similarity between all titles in the bid document template and all titles in the historical bid document files; selecting titles that conform to the bid document template from the historical bid document dataset according to the semantic similarity and assembling them to obtain a target reference text, wherein the titles in the target reference text are arranged in order of title levels; and determining a target bid document dataset according to the target reference text.

[0007] Optionally, the semantic similarity between all titles in the bid template and all titles in the historical bid files is calculated, including: obtaining the i-th level title in the bid template and the h-th level title in the j-th historical bid file in the historical bid data set, where i, j and h are integers greater than or equal to 1; calculating the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file in the historical bid data set.

[0008] Optionally, titles that conform to the bid template are screened out from the historical bid data set according to semantic similarity and assembled to obtain a target reference text, including: when the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file is less than or equal to a preset threshold, determining that the h-th level title in the j-th historical bid file does not conform to the structure of the bid template; when the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file is greater than a preset threshold, determining that the h-th level title in the j-th historical bid file conforms to the structure of the bid template; and determining the target reference text based on all titles in the j-th historical bid file that conform to the structure of the bid template.

[0009] Optionally, the target reference text is determined based on all titles in the jth historical bid file that conform to the structure of the bid template, including: obtaining content paragraphs corresponding to all titles in the jth historical bid file that conform to the structure of the bid template; assembling titles of different levels and content paragraphs corresponding to titles of different levels according to the structure of the bid template to obtain the target reference text.

[0010] Optionally, determining a target bid data set based on a target reference text comprises: in a case where the target reference text includes S levels of titles, extracting K target titles from the target reference text in order of title levels from low to high, wherein S is an integer greater than or equal to 1, and K is an integer less than or equal to S; obtaining all parent titles of the target title and content paragraphs corresponding to the target title; taking the target title and all parent titles of the target title as instruction names; taking the content paragraphs corresponding to the target title as instruction values; and determining the target bid data set based on the instruction name and the instruction value.

[0011] Optionally, after determining the target bid data set based on the target reference text, the method for constructing the bid data set also includes: proofreading and expanding the data processing of the target bid data set, wherein the proofreading is used to check the target bid data set for erroneous data, and the expanding data processing is used to construct a new data set based on the data in the target bid data set.

[0012] Optionally, after obtaining the bid template and the historical bid data set, the method for constructing the bid data set also includes: performing data cleaning and preprocessing operations on the historical bid data set, wherein data cleaning is used to delete preset characters in the historical bid data set, and preprocessing is used to perform word segmentation and vocabulary standardization on the historical bid data set, wherein vocabulary standardization is used to unify and normalize the vocabulary in the historical bid data set.

[0013] According to another aspect of an embodiment of the present application, a device for constructing a bid document data set is also provided, including: obtaining a bid document template and a historical bid document data set, wherein the bid document template includes at least N levels of titles, and the historical bid document data set includes at least M historical bid document files, wherein N and M are integers greater than 1, and each historical bid document file includes at least multiple levels of titles; a calculation unit, calculating the semantic similarity between all titles in the bid document template and all titles in the historical bid document files; a processing unit, screening out titles that meet the bid document template from the historical bid document data set according to the semantic similarity, and assembling them to obtain a target reference text, wherein the titles in the target reference text are arranged in order of title levels; a determination unit, determining the target bid document data set according to the target reference text.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the above-mentioned method for constructing a bid document data set.

[0015] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned method for constructing a bid document data set.

[0016] It can be seen from the above content that in the present application, firstly, a bid template and a historical bid data set of a specific vertical field are obtained, wherein the bid template includes at least N levels of titles, and the historical bid data set includes at least M historical bid files, wherein N and M are integers greater than 1, and each historical bid file includes at least multiple levels of titles. Then, the semantic similarity between all titles in each historical bid file and all titles in the bid template is calculated to determine the degree of match between the titles in each historical bid file and the bid template. Then, according to the calculated semantic similarity, the titles that meet the bid template are screened out from the historical bid data set for assembly to obtain the target reference text, thereby ensuring the relevance of the titles in the target reference text to the bid template, wherein the titles in the target reference text are arranged in the order of title levels. Finally, the target bid data set of the specific vertical field is determined based on the multiple target reference texts obtained, thereby ensuring the accuracy and relevance of the generated target bid data set.

[0017] Compared with the prior art of obtaining bid data sets in vertical fields through manual annotation, rule matching, similarity calculation, etc., the present application constructs a target bid data set in a specific vertical field through a standardized process. By obtaining bid templates and historical bid data sets in the specific vertical field, the semantic similarity between all titles in each historical bid file and all titles in the bid template is calculated, and finally the target bid data set in the specific vertical field is determined according to the semantic similarity, thereby achieving the purpose of automatically, quickly and accurately constructing the target bid data set in the specific vertical field, thereby achieving the technical effect of improving the effect of constructing bid fine-tuning data sets in vertical fields, and then solving the technical problem of poor effect of constructing bid fine-tuning data sets in vertical fields in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a flowchart of an optional method for constructing a bid document data set according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of an optional automated construction method of a bid fine-tuning data set according to an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of an optional device for constructing a bid document data set according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] It should also be noted that the information and data collected by this application are authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0025] According to an embodiment of the present application, an embodiment of a method for constructing a bid document data set is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] It should be noted that an intelligent generation system is used as the execution subject of the method for constructing a bid document data set in the embodiment of the present application. It is understandable that the method for constructing a bid document data set provided in the embodiment of the present application can also be executed by other systems or devices, and the embodiment of the present application does not specifically limit this.

[0027] Figure 1is a flowchart of an optional method for constructing a bid document data set according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0028] Step S101, obtaining a tender document template and a historical tender document dataset.

[0029] In step S101, the bid template includes at least N levels of titles, and the historical bid data set includes at least M historical bid files.

[0030] In step S101, N and M are integers greater than 1, and each historical bid file includes at least multiple levels of titles.

[0031] Optionally, the bid template also includes bid chapter information, bid clause information and bid structure information.

[0032] Optionally, the historical bid documents in the historical bid document data set are stored in PDF format (Portable Document Format), and files in PDF format are usually not directly editable. Therefore, a PDF parsing tool is required to convert the historical bid documents in PDF format into plain text format, and retain the format information of the historical bid documents during the format conversion process, such as chapter titles, list formats, paragraph structures, and font sizes.

[0033] Step S102, calculating the semantic similarity between all titles in the tender document template and all titles in the historical tender document file.

[0034] Optionally, the intelligent generation system calculates the semantic similarity between each historical tender document and all titles of the tender template.

[0035] Step S103, selecting titles that conform to the tender document template from the historical tender document data set according to semantic similarity and assembling them to obtain a target reference text.

[0036] In step S103, the titles in the target reference text are arranged in order of title levels.

[0037] Optionally, the intelligent generation system selects titles that conform to the tender template from each historical tender document in the historical tender document data set according to semantic similarity and assembles them to obtain multiple target reference texts.

[0038] Step S104, determining a target bid document data set according to the target reference text.

[0039] Optionally, the target bid document dataset may be in a JSON (JavaScript Object Notation) format or a CSV (Comma-Separated Values) format.

[0040] Optionally, the intelligent generation system determines a target bid data set based on multiple target reference texts obtained from each historical bid file.

[0041] It can be seen from the contents of steps S101 to S104 that in the present application, firstly, a bid template and a historical bid data set are obtained, wherein the bid template includes at least N levels of titles, and the historical bid data set includes at least M historical bid files, wherein N and M are integers greater than 1, and each historical bid file includes at least multiple levels of titles. Then, the semantic similarity between all titles in the bid template and all titles in the historical bid files is calculated, and then, based on the semantic similarity, titles that meet the bid template are screened from the historical bid data set for assembly to obtain a target reference text, wherein the titles in the target reference text are arranged in order of title levels, and finally, the target bid data set is determined based on the target reference text.

[0042] It can be seen from the above content that in the present application, firstly, a bid template and a historical bid data set of a specific vertical field are obtained, wherein the bid template includes at least N levels of titles, and the historical bid data set includes at least M historical bid files, wherein N and M are integers greater than 1, and each historical bid file includes at least multiple levels of titles. Then, the semantic similarity between all titles in each historical bid file and all titles in the bid template is calculated to determine the degree of match between the titles in each historical bid file and the bid template. Then, according to the calculated semantic similarity, the titles that meet the bid template are screened out from the historical bid data set for assembly to obtain the target reference text, thereby ensuring the relevance of the titles in the target reference text to the bid template, wherein the titles in the target reference text are arranged in the order of title levels. Finally, the target bid data set of the specific vertical field is determined based on the multiple target reference texts obtained, thereby ensuring the accuracy and relevance of the generated target bid data set.

[0043] Compared with the prior art of obtaining bid data sets in vertical fields through manual annotation, rule matching, similarity calculation, etc., the present application constructs a target bid data set in a specific vertical field through a standardized process. By obtaining bid templates and historical bid data sets in the specific vertical field, the semantic similarity between all titles in each historical bid file and all titles in the bid template is calculated, and finally the target bid data set in the specific vertical field is determined according to the semantic similarity, thereby achieving the purpose of automatically, quickly and accurately constructing the target bid data set in the specific vertical field, thereby achieving the technical effect of improving the effect of constructing bid fine-tuning data sets in vertical fields, and then solving the technical problem of poor effect of constructing bid fine-tuning data sets in vertical fields in the prior art.

[0044] In an optional embodiment, the intelligent generation system first obtains the i-th level title in the bid template and the h-th level title in the j-th historical bid file in the historical bid data set, where i, j and h are integers greater than or equal to 1, and then calculates the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file in the historical bid data set.

[0045] Optionally, the intelligent generation system first obtains all levels of titles in the template bid and all levels of titles of each historical bid file in the historical bid data set, and then calculates the semantic similarity between all titles of each historical bid file and the bid template.

[0046] Optionally, the intelligent generation system uses word embedding technology, such as Word2Vec (Word to Vector), GloVe (Global Vectors for Word Representation), or BERT (Bidirectional Encoder Representations from Transformers), to calculate the semantic similarity between the bid template title and all titles in historical bid documents.

[0047] Optionally, the intelligent generation system can also use a similarity calculation method combining cosine similarity and word order features, a comprehensive similarity calculation method combining part-of-speech information and semantic similarity, an end-to-end similarity calculation model based on a deep learning network, and a title similarity calculation method using a fuzzy matching library.

[0048] Alternatively, cosine similarity is a commonly used text similarity measurement method, which evaluates the similarity of two vectors by comparing their angles in a multidimensional space. In text processing, it usually involves converting text into word vectors and then calculating the cosine value between them. Combining word order features can enhance the accuracy of cosine similarity. Word order features take into account the position and order of words in a sentence or document, which is crucial for understanding sentence structure and semantic relationships. Combining these two features can more accurately capture the semantic content and structure of the text, thereby improving the accuracy of similarity calculation.

[0049] Optionally, part-of-speech information refers to the grammatical properties of words, such as nouns, verbs, etc., which helps to understand the structure of the text, and semantic similarity focuses on the degree of closeness in meaning between words or phrases. By combining these two types of information, similarity calculation can more accurately reflect the actual similarity between texts, making similarity calculation more comprehensive and accurate.

[0050] Optionally, an end-to-end deep learning model can learn features directly from raw text data and perform similarity calculations through deep learning technology. The end-to-end model usually uses neural networks such as convolutional neural networks and recurrent neural networks for analysis, which can automatically extract complex text features, learn deep relationships between texts, and capture the deep semantic information of the text.

[0051] From the above content, it can be seen that the intelligent generation system can generate a bid data set that is more in line with a specific vertical field by evaluating the semantic similarity between bid templates in different vertical fields and titles of historical bid data sets. Using the generated bid data set in the vertical field to train the bid generation model can improve the accuracy and generalization ability of the bid generation model, thereby improving the quality and relevance of bids in the vertical field. In addition, by adopting deep learning word embedding technology and comprehensive similarity strategies, it can more effectively capture the semantic similarity between texts, solving the limitations of traditional similarity calculation methods in dealing with complex semantics.

[0052] In an optional embodiment, the intelligent generation system determines that the h-th level title in the j-th historical bid file does not conform to the structure of the bid template when the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file is less than or equal to a preset threshold value; determines that the h-th level title in the j-th historical bid file conforms to the structure of the bid template when the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file is greater than a preset threshold value; finally, the target reference text is determined based on all titles in the j-th historical bid file that conform to the structure of the bid template.

[0053] Optionally, the intelligent generation system first obtains a preset threshold value of semantic similarity, and then determines the degree of match between the title in the bid template and the title in the historical bid file based on a comparison between the calculated semantic similarity and the preset threshold value.

[0054] Optionally, when the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file is less than or equal to a preset threshold, the intelligent generation system determines that the h-th level title in the j-th historical bid file does not conform to the structure of the bid template; when the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file is greater than a preset threshold, the intelligent generation system determines that the h-th level title in the j-th historical bid file conforms to the structure of the bid template, and then assembles all the titles in the j-th historical bid file that conform to the structure of the bid template in order of title levels to obtain a target reference text. It should be noted that the intelligent generation system will separately calculate the semantic similarity between the title in each historical bid file and the title of the corresponding level in the title template, that is, each historical bid file will eventually correspond to a target reference text.

[0055] Optionally, when setting the preset threshold of semantic similarity, the intelligent generation system can set it according to different similarity threshold setting strategies, for example, setting a fixed similarity threshold, setting a dynamically adjusted similarity threshold, setting a similarity threshold determined based on a statistical confidence interval, and setting a similarity threshold determined experimentally for a specific text task.

[0056] From the above content, it can be seen that by setting a preset threshold for semantic similarity, the intelligent generation system can accurately identify and match structurally similar or consistent titles, thereby ensuring that the target reference text obtained conforms to the bid template in structure. At the same time, the automated semantic similarity evaluation reduces the need for manual review, improves efficiency, and can consistently apply the same evaluation criteria, ensuring the quality of the target reference text, providing a reliable data foundation for the subsequent generation of the target bid data set, and allowing the preset threshold for semantic similarity to be set according to specific needs or specific industry standards, providing flexible adjustment space to meet the specific requirements of different vertical fields.

[0057] In an optional embodiment, the intelligent generation system obtains the content paragraphs corresponding to all titles in the jth historical bidding document that conform to the structure of the bidding document template, assembles titles of different levels and content paragraphs corresponding to titles of different levels according to the structure of the bidding document template, and obtains the target reference text.

[0058] Optionally, after determining the titles in the jth historical bid file that conform to the structure of the bid template, the intelligent generation system obtains the content paragraphs corresponding to all the titles in the jth historical bid file that conform to the structure of the bid template, and then assembles the titles of different levels and the content paragraphs corresponding to the titles of different levels according to the structure of the bid template to obtain the target reference text corresponding to the jth historical bid file.

[0059] From the above content, it can be seen that the intelligent generation system obtains the content paragraphs corresponding to all the titles in the historical bid documents that conform to the structure of the bid template, and finally generates a structured target reference text. The reference text is consistent with the bid template in structure and content, which improves the overall quality of the reference text and provides an accurate data source for the subsequent construction of the target bid data set. In addition, the system automatically extracts and assembles content from historical bid documents, reduces the workload of manual editing and typesetting, improves the efficiency of building bid data sets, and reduces the cost of manual annotation and dataset construction, thereby reducing the operating costs of enterprises in the bid generation process.

[0060] In an optional embodiment, when the target reference text includes S levels of titles, the intelligent generation system extracts K target titles from the target reference text in order of title levels from low to high, where S is an integer greater than or equal to 1 and K is an integer less than or equal to S. Then, all parent titles of the target title and the content paragraphs corresponding to the target title are obtained, and then the target title and all parent titles of the target title are used as instruction names, and the content paragraphs corresponding to the target title are used as instruction values. Finally, the target bid data set is determined based on the instruction name and the instruction value.

[0061] Optionally, when the target reference text includes S levels of titles, the intelligent generation system extracts K target titles from the S levels of titles, where the K target titles are lower-level titles under each chapter in the target reference text, and then obtains all the parent titles of the target title and the content paragraphs corresponding to the target title, and uses the target title and all the parent titles of the target title as instruction names, and the content paragraphs corresponding to the target title as instruction values, and finally determines the target bid data set based on all the instruction names and instruction values. For example, when each chapter in the target reference text includes four levels of titles, obtain all the parent titles of the third-level title and the fourth-level title under each chapter, and simultaneously obtain the content paragraphs corresponding to the third-level title and the fourth-level title respectively, and then use the third-level title and all the parent titles of the third-level title as the instruction name, and the content paragraph corresponding to the third-level title as the instruction value corresponding to the instruction name. Similarly, use the fourth-level title and all the parent titles of the fourth-level title as the instruction name, and the content paragraph corresponding to the fourth-level title as the instruction value corresponding to the instruction name, that is, the instruction name and the instruction value are in a one-to-one correspondence, and then determine the target bid data set based on the instruction name and instruction value obtained from each target reference text.

[0062] Optionally, the intelligent generation system may select only the smaller level titles among the multiple level titles in the target reference text as the instruction name, or may select all the level titles as the instruction name. For example, in the case where one of the chapters in the target reference text includes 4 level titles, the intelligent generation system may only obtain the fourth level title, and use the fourth level title and all the upper level titles of the fourth level title as the instruction name, and use the content paragraph corresponding to the fourth level title as the instruction value corresponding to the instruction name. It may also obtain the first level title, the second level title, the third level title, and the fourth level title at the same time, and use the first level title as the instruction name, and the content paragraph corresponding to the first level title as the instruction value corresponding to the instruction name, and use the second level title and all the upper level titles of the second level title (i.e., the first level title) as the instruction name, and use the content paragraph corresponding to the second level title as the instruction value corresponding to the instruction name, and so on, and finally determine the target bid document data set according to all the instruction names and instruction values ​​obtained.

[0063] Optionally, the intelligent generation system can use the target bid dataset as fine-tuning corpus for the base language large model, for example: for training a bid generation model based on the Transformer architecture (a model architecture applied to natural language processing tasks), for training a bid summary generation model based on a recurrent neural network, and for training a bid content generation model based on a generative adversarial network.

[0064] From the above content, it can be seen that the intelligent generation system generates a target bid document data set with higher quality, richer and more accurate information through structured and hierarchical processing. The target bid document data set obtained based on precise instruction names and instruction values ​​is used to train the bid generation model, which can make the generated bid content more in line with the requirements and improve the relevance and accuracy of the generated content. In this way, key information can be effectively extracted from existing documents and a structured and hierarchical target bid document data set can be constructed to meet the business needs of specific vertical fields.

[0065] In an optional embodiment, the intelligent generation system performs proofreading and data expansion processing on the target bid data set, wherein the proofreading is used to check for erroneous data in the target bid data set, and the data expansion processing is used to construct a new data set based on the data in the target bid data set.

[0066] Optionally, after obtaining the target bid document data set, the intelligent generation system will perform proofreading and data expansion processing on the target bid document data set to ensure the accuracy and relevance of the data set.

[0067] Optionally, the intelligent generation system uses machine learning algorithms to automatically detect and correct errors in the data set, and can be combined with management review. Proofreading checks can identify and correct errors or inconsistent data in the bid data set, such as spelling errors, formatting errors, or inaccurate data.

[0068] Optionally, the intelligent generation system increases the diversity and coverage of the data set through data augmentation technology, wherein data augmentation technology is a commonly used method in machine learning and deep learning, which aims to increase the diversity and coverage of the training data set by generating new and modified data samples. The intelligent generation system can obtain a new data set by performing synonym replacement, random deletion, back translation, word form change, etc. on the target bid data set, thereby increasing the diversity of the training data set.

[0069] From the above content, it can be seen that through proofreading, the intelligent generation system can discover and fill in the missing or omitted information in the data set, thereby improving the completeness and accuracy of the data and improving the quality of the data set. By expanding data processing, it helps to expand the coverage of the data set, cover more scenarios and conditions, and provide a solid foundation for generating high-quality tender content.

[0070] In an optional embodiment, the intelligent generation system performs data cleaning and preprocessing operations on the historical bid document data set, wherein data cleaning is used to delete preset characters in the historical bid document data set, and preprocessing is used to perform word segmentation and vocabulary standardization on the historical bid document data set, wherein vocabulary standardization is used to unify and normalize the vocabulary in the historical bid document data set.

[0071] Optionally, after obtaining the historical tender document dataset, the intelligent generation system performs data cleaning and preprocessing operations on the historical tender document dataset to ensure the accuracy and operability of the data.

[0072] Optionally, data cleaning at least includes text cleaning and stop word filtering. Among them, text cleaning is used to remove special characters, redundant spaces, punctuation marks, and line breaks in plain text to ensure the neatness of the text. Stop word filtering is used to remove common stop words in the text, such as "de", "he", "shi", etc. These words usually do not carry important information in similarity calculation.

[0073] Optionally, the preprocessing operations at least include word segmentation and part-of-speech tagging processing and lexical standardization processing. Among them, word segmentation and part-of-speech tagging processing are used to perform word segmentation on the text and tag the part of speech for each word to better understand the text content. Lexical standardization processing is used to unify synonyms, convert the vocabulary in the text into a standard form, and reduce the diversity of vocabulary.

[0074] Optionally, Figure 2 is a schematic diagram of an optional automated construction method for a tender document fine-tuning dataset according to an embodiment of the present application. As Figure 2 shown, first, input the tender document template and the historical tender document dataset in the vertical domain. Then, perform preprocessing operations on the historical tender document dataset. According to the format and structure information of the historical tender documents in the historical tender document dataset, locate the title information and corresponding content paragraphs of the historical tender documents, and identify the level of each title, such as first-level title, second-level title, etc. Then, filter out the valid information in the historical tender documents according to the tender document template, that is, by calculating the semantic similarity between the tender document template and the historical document title, screening out relevant titles according to the semantic similarity, and assembling the relevant titles in the order of the level from high to low according to the title level of the relevant titles. After that, retrieve the lowest-level title (the title with the smallest title level) from the assembled text, and at the same time extract the content paragraph corresponding to the lowest-level title. Use the lowest-level title and all its superior titles as the instruction name, and use the content paragraph corresponding to the lowest-level title as the answer, that is, the instruction value, corresponding to the instruction name. Construct a fine-tuning dataset according to the instruction name and the instruction value. Finally, output the fine-tuning dataset for the vertical domain, and the tender document generation large model can be trained according to this fine-tuning dataset.

[0075] As can be seen from the above, by performing data cleaning and preprocessing on the historical tender document dataset, the intelligent generation system not only improves the quality and consistency of the data, but also provides a solid foundation for generating high-quality tender document content, thereby improving the output quality of the system.

[0076] According to an embodiment of the present application, an embodiment of a construction device for a tender document dataset is also provided. Figure 3is a schematic diagram of an optional device for constructing a bid document data set according to an embodiment of the present application, such as Figure 3 As shown, the device for constructing a bid document data set includes: an acquisition unit 301 , a calculation unit 302 , a processing unit 303 , and a determination unit 304 .

[0077] Optionally, the acquisition unit 301 is used to acquire a bid template and a historical bid data set, wherein the bid template includes at least N levels of titles, and the historical bid data set includes at least M historical bid files, wherein N and M are integers greater than 1, and each historical bid file includes at least multiple levels of titles; the calculation unit 302 is used to calculate the semantic similarity between all titles in the bid template and all titles in the historical bid files; the processing unit 303 is used to filter out titles that meet the bid template from the historical bid data set according to the semantic similarity and assemble them to obtain a target reference text, wherein the titles in the target reference text are arranged in order of title levels; the determination unit 304 is used to determine the target bid data set according to the target reference text.

[0078] Optionally, the acquisition unit 301 includes: a first processing subunit, used to perform data cleaning and preprocessing operations on the historical bid document data set, wherein data cleaning is used to delete preset characters in the historical bid document data set, and preprocessing is used to perform word segmentation and vocabulary standardization on the historical bid document data set, wherein vocabulary standardization is used to unify and normalize the vocabulary in the historical bid document data set.

[0079] Optionally, the calculation unit 302 includes: a first acquisition subunit and a first calculation subunit. The first acquisition subunit is used to acquire the i-th level title in the bid template and the h-th level title in the j-th historical bid file in the historical bid data set, wherein i, j and h are integers greater than or equal to 1; the first calculation subunit is used to calculate the semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file in the historical bid data set.

[0080] Optionally, the processing unit 303 includes: a first determining subunit, a second determining subunit, and a third determining subunit. The first determining subunit is used to determine that the h-th level title in the j-th historical bidding document does not conform to the structure of the bidding document template when the semantic similarity between the i-th level title in the bidding document template and the h-th level title in the j-th historical bidding document file is less than or equal to a preset threshold; the second determining subunit is used to determine that the h-th level title in the j-th historical bidding document file conforms to the structure of the bidding document template when the semantic similarity between the i-th level title in the bidding document template and the h-th level title in the j-th historical bidding document file is greater than a preset threshold; the third determining subunit is used to determine the target reference text according to all titles in the j-th historical bidding document file that conform to the structure of the bidding document template.

[0081] Optionally, the third determination subunit includes: a first acquisition module and a first processing module. The first acquisition module is used to acquire the content paragraphs corresponding to all titles that conform to the structure of the tender template in the jth historical tender document file; the first processing module is used to assemble the titles of different levels and the content paragraphs corresponding to the titles of different levels according to the structure of the tender template to obtain the target reference text.

[0082] Optionally, the determination unit 304 includes: a first extraction subunit, a second acquisition subunit, a second processing subunit, a third processing subunit, and a fourth determination subunit. The first extraction subunit is used to extract K target titles from the target reference text in the order of title levels from low to high, when the target reference text includes titles of S levels, wherein S is an integer greater than or equal to 1, and K is an integer less than or equal to S; the second acquisition subunit is used to acquire all parent titles of the target title and the content paragraphs corresponding to the target title; the second processing subunit is used to use the target title and all parent titles of the target title as instruction names; the third processing subunit is used to use the content paragraphs corresponding to the target title as instruction values; the fourth determination subunit is used to determine the target bid data set according to the instruction name and the instruction value.

[0083] Optionally, the determination unit 304 includes: a fourth processing subunit, used for proofreading and expanding the data processing of the target bid document data set, wherein the proofreading is used to check the target bid document data set for erroneous data, and the expanding data processing is used to construct a new data set based on the data in the target bid document data set.

[0084] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the above-mentioned method for constructing a bid document data set.

[0085] According to another aspect of an embodiment of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned method for constructing a bid data set.

[0086] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0087] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0090] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0091] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0092] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for constructing a bid document dataset, characterized in that: include: Acquire a tender document template and a historical tender document data set, wherein the tender document template includes at least N levels of titles, and the historical tender document data set includes at least M historical tender document files, wherein N and M are integers greater than 1, and each historical tender document file includes at least multiple levels of titles; Calculating the semantic similarity between all titles in the bid template and all titles in the historical bid file; According to the semantic similarity, titles that match the bid template are selected from the historical bid document data set for assembly to obtain a target reference text, wherein the titles in the target reference text are arranged in order of title levels; A target bid document data set is determined according to the target reference text.

2. The method for constructing a bid document dataset according to claim 1, characterized in that: Calculating the semantic similarity between all titles in the bid template and all titles in the historical bid file, including: Obtaining the title of the i-th level in the bid template and the title of the h-th level in the j-th historical bid file in the historical bid data set, wherein i, j and h are integers greater than or equal to 1; The semantic similarity between the i-th level title in the bid template and the h-th level title in the j-th historical bid file in the historical bid data set is calculated.

3. The method for constructing a bid document dataset according to claim 2, characterized in that: According to the semantic similarity, titles that match the bid template are selected from the historical bid document data set for assembly to obtain a target reference text, including: When the semantic similarity between the i-th level title in the bid document template and the h-th level title in the j-th historical bid document file is less than or equal to a preset threshold, determining that the h-th level title in the j-th historical bid document file does not conform to the structure of the bid document template; When the semantic similarity between the i-th level title in the bid document template and the h-th level title in the j-th historical bid document file is greater than the preset threshold, determining that the h-th level title in the j-th historical bid document file conforms to the structure of the bid document template; The target reference text is determined according to all titles in the j-th historical bidding document that conform to the structure of the bidding document template.

4. The method for constructing a bid document data set according to claim 3, characterized in that: Determining the target reference text according to all titles in the j-th historical bidding document that conform to the structure of the bidding document template includes: Obtaining content paragraphs corresponding to all titles in the j-th historical bidding document that conform to the structure of the bidding document template; The titles at different levels and the content paragraphs corresponding to the titles at different levels are assembled according to the structure of the bid template to obtain the target reference text.

5. The method for constructing a bid document dataset according to claim 1, characterized in that: Determining a target bid document data set according to the target reference text includes: In the case where the target reference text includes S levels of titles, extracting K target titles from the target reference text in order of title levels from low to high, where S is an integer greater than or equal to 1, and K is an integer less than or equal to S; Obtain all parent titles of the target title and content paragraphs corresponding to the target title; Taking the target title and all the parent titles of the target title as instruction names; Using the content paragraph corresponding to the target title as the instruction value; The target bid data set is determined according to the instruction name and the instruction value.

6. The method for constructing a bid document dataset according to claim 1, characterized in that: After determining the target bid document dataset according to the target reference text, the method for constructing the bid document dataset further includes: The target bid document data set is proofread and expanded, wherein the proofread is used to check the target bid document data set for erroneous data, and the expanded data processing is used to construct a new data set based on the data in the target bid document data set.

7. The method for constructing a bid document dataset according to claim 1, characterized in that: After obtaining the tender document template and the historical tender document dataset, the method for constructing the tender document dataset further includes: The historical bid document data set is subjected to data cleaning and preprocessing operations, wherein the data cleaning is used to delete preset characters in the historical bid document data set, and the preprocessing operation is used to perform word segmentation and vocabulary standardization on the historical bid document data set, wherein the vocabulary standardization is used to unify and normalize the vocabulary in the historical bid document data set.

8. A device for constructing a bid document data set, characterized in that: include: An acquisition unit is configured to acquire a bid document template and a historical bid document data set, wherein the bid document template includes at least N levels of titles, and the historical bid document data set includes at least M historical bid document files, wherein N and M are integers greater than 1, and each historical bid document file includes at least multiple levels of titles; A calculation unit, calculating the semantic similarity between all titles in the bid template and all titles in the historical bid file; A processing unit, which selects titles that match the bid template from the historical bid data set according to the semantic similarity and assembles them to obtain a target reference text, wherein the titles in the target reference text are arranged in order of title levels; A determination unit is configured to determine a target bid document data set according to the target reference text.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the method for constructing a bid document dataset according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method for constructing a bid document data set as described in any one of claims 1 to 7.

Images