Data set construction method, equipment and medium

Abstract

The invention discloses a data set construction method and device and a medium, and the method comprises the steps: carrying out the processing of original data through a standardized logic program and a big language model cue word, and obtaining a standardized processing text; extracting a field label of the standardized processing text through a large language model cue word, and constructing an initial training sample set according to the field label; performing quality evaluation on the initial training sample set through a verification set evaluation function to obtain a quality evaluation result; performing question extraction on the quality evaluation result through a large language model, and generating prompt word modification suggestions; and re-inputting the adjusted cue word into the large language model to generate a new training sample set, and carrying out loop iteration on the processes of quality evaluation, question extraction and cue word adjustment until the sample quality reaches a preset quality standard. Through combination of large language model cue words and an adaptive optimization algorithm, the data set construction efficiency is improved, and the pain points which depend on manpower and are difficult to adapt to large-scale complex data processing are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data set construction method, device, and medium. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, the amount of data in various industries has exploded. How to efficiently extract value from massive data and realize AI intelligent applications has become the core requirement of current data set construction.

[0003] Most existing dataset construction methods directly summarize the raw data to generate datasets. However, the raw data generally has problems such as messy format and lack of effective annotation, making it difficult to use directly for AI model training. It needs to be processed accordingly, resulting in low construction efficiency. Secondly, data processing is highly dependent on manual operations, which consumes a lot of manpower and material resources. When faced with large-scale and complex structured data, the dataset construction efficiency is low and the adaptability is poor, which cannot meet the urgent needs of AI models for high-quality training data. Summary of the Invention

[0004] The embodiments of the present application provide a data set construction method, device, and medium for solving the problems of poor data quality and low construction efficiency in existing data set construction methods.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a data set construction method, which includes: processing the original data through a standardized logic program and a large language model prompt word to obtain a standardized processed text; extracting the domain label of the standardized processed text through the large language model prompt word, and constructing an initial training sample set based on the domain label; performing a quality assessment on the initial training sample set through a validation set evaluation function to obtain a quality assessment result; refining the quality assessment result through a large language model to generate prompt word modification suggestions; re-inputting the adjusted prompt word into the large language model to generate a new training sample set, and iterating the quality assessment, problem refinement and prompt word adjustment process until the sample quality reaches the preset quality standard.

[0007] In one example, the original data is format converted, multi-format parsed, and logically segmented through standardized logic programs and large language model prompts to obtain standardized processed text, specifically including: using a large language model prompt word driving tool to identify and parse original data in various formats, and converting the original data in various formats into text data in a unified format; according to the text segmentation logic, the text data is segmented with the guidance of large language model prompts to obtain standard text fragments; JSON format data and the standard text fragments are integrated to obtain standardized processed text.

[0008] In one example, the domain labels of the standardized text are extracted through the large language model prompt words, and question-answer pairs and multi-genre question-answer samples are generated based on the domain labels to construct an initial training sample set, specifically including: refining the domain labels according to preset classification rules, and constructing a domain knowledge tree based on the refined domain labels; matching the large language model prompt words with each refined domain in the domain knowledge tree to generate question-answer pairs corresponding to the covered domain knowledge points; performing genre conversion on the standardized text according to the preset genre and the question-answer pairs to generate multi-genre question-answer samples; integrating the multi-genre question-answer samples and the question-answer pairs to obtain the initial training sample set.

[0009] In one example, a large language model is used to extract problems from the quality assessment results, determine the problem pattern, and generate prompt word modification suggestions based on the problem pattern, a preset policy library, and preset heuristic rules. Specifically, the following steps are used: according to preset evaluation indicators, the quality assessment results are classified to obtain qualified results and unqualified results; the qualified results are identified and classified as sample defects by the large language model, and the problem pattern is determined to be empty; the unqualified results are identified and classified as sample defects by the large language model, and the problem pattern type is determined; according to the preset policy library and preset heuristic rules, the problem pattern type is parsed by the large language model instruction generation capability to generate prompt word modification suggestions.

[0010] In one example, based on a preset policy library and preset heuristic rules, the question pattern type is parsed through the large language model instruction generation capability to generate prompt word modification suggestions, specifically including: if the number of words in the question and answer pair exceeds a preset limit, generating a constraint instruction for the preset word limit for the question and answer pair; if the label granularity is too coarse, generating an instruction for the third-level subcategories of the question and answer pair label; if the question and answer pair does not match the original data, generating an instruction for the question and answer pair based on the original content constraints.

[0011] In one example, the method further includes: performing quality assessment on the initial training sample set through a lightweight discriminant model to obtain a quality assessment result; the lightweight discriminant model is trained through model distillation technology and learns from the evaluation capabilities of a large language model and manually annotated data.

[0012] In one instance, after the adjusted prompt words are re-input into the large language model to generate a new training sample set, the method also includes: sending the new training sample set to the client for manual review to obtain a manual review result; determining the received manual review result as a review feedback suggestion to iteratively update the preset strategy library and the preset heuristic rules.

[0013] In one example, the quality assessment, problem refinement, and prompt word adjustment processes are iterated until the sample quality reaches the preset quality standard and the optimized training data set is output. The method also includes: determining the data set format to be exported based on the preset task requirements and the format matching table; and converting the optimized training data set format based on the data set format to export the converted optimized training data set.

[0014] On the other hand, an embodiment of the present application provides a dataset construction device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the above-mentioned dataset construction methods.

[0015] On the other hand, an embodiment of the present application provides a data set construction non-volatile computer storage medium storing computer executable instructions capable of executing any one of the above-mentioned data set construction methods.

[0016] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0017] This application significantly improves the automation level and efficiency of data set construction through the intelligent construction method of the large language model prompt word project combined with the prompt word adaptive optimization algorithm, while ensuring the high quality and accuracy of the output data; at the same time, the large prediction prompt word project directly drives the large language model API to perform automatic construction work, realizing highly automated, intelligent and controllable data set construction; the iterative process of the prompt word adaptive optimization algorithm aims to minimize manual intervention, maximize the effectiveness of prompt words, and realize the transformation of data generation from static to dynamic, from manual to adaptive. This not only greatly reduces the cost and threshold of data preparation, but also provides a solid data foundation for the rapid iteration of AI models and the implementation of various intelligent applications, and effectively promotes the process of digital transformation in various industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, some embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:

[0019] Figure 1 A flowchart of a method for constructing a data set provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of the specific steps of a data set construction method provided in an embodiment of the present application;

[0021] Figure 3A schematic diagram of the structure of a data set construction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] Some embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is a flow chart of a dataset construction method provided in an embodiment of the present application. This method can be applied to different business areas. Certain input parameters or intermediate results in this process can be manually adjusted to help improve accuracy.

[0025] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For ease of understanding and description, the following embodiments are described in detail using a controller as an example.

[0026] Based on this, Figure 1 The process in may include the following steps:

[0027] S101: Through standardized logic programs and large language model prompt words, the original data is format converted, multi-format parsed, and logically segmented to obtain standardized processed text.

[0028] In some embodiments of the present application, after the solution is started, preparations for data set construction begin. First, for structured data such as tables in the original data that are inconvenient for the large language model to read directly, a preset logic program is called to convert them into standardized formats such as JSON to adapt to the large language model API processing requirements.

[0029] Furthermore, the large language model prompt word-driven tool is used to intelligently identify and parse raw data in various formats, such as text and image OCR results, and convert the raw data in various formats into text data in a unified format.

[0030] Furthermore, guided by the large language model's prompt words, the long text is segmented into independent segments based on structural logic, generating standardized text segments. Finally, the JSON formatted data and the standardized text segments are integrated to obtain standardized processed text for subsequent refined processing.

[0031] S102: Extracting domain labels of the standardized text through large language model prompt words, and generating question-answer pairs and multi-genre question-answer samples based on the domain labels to construct an initial training sample set.

[0032] In some embodiments of the present application, for standardized text processing, key domain labels are extracted from the text using large language model prompt words, and then according to preset classification rules, detailed labels under the domain are further extracted based on the extracted labels, and a domain knowledge tree is automatically constructed in a hierarchical relationship based on the refined domain labels.

[0033] Furthermore, for each sub-domain in the domain knowledge tree, questions and answers are automatically generated by matching them with the prompt words of the large language model to ensure that the generated questions and answers (question-answer pairs) can widely cover the domain knowledge points and build a high-quality question-answer data set; and this application also supports the generation of texts of different genres based on an original text, that is, based on the same original text, question and answer samples of different genres such as summary, analysis, and expansion are generated to construct a multi-genre generation (MGA) sample data set to enrich the diversity of training samples; finally, the multi-genre question and answer samples and question and answer pairs are integrated to obtain the initial training sample set.

[0034] S103: Performing quality evaluation on the initial training sample set using a validation set evaluation function to obtain a quality evaluation result.

[0035] In some embodiments of the present application, regarding the evaluation of the initial training sample set, the system uses a validation set evaluation function to perform a preliminary quality evaluation on the automatically generated initial training sample set (including question-answer pairs and domain labels) to obtain a quality evaluation result.

[0036] In particular, this application can also combine model distillation technology to train a lightweight discriminant model as an auxiliary evaluator. This lightweight model learns from the evaluation capabilities of the original large model or manually annotated data, and can efficiently judge the quality of generated samples at a lower computational cost, and quantitatively judge based on preset quality indicators (such as coverage and precision).

[0037] S104: extracting problems from the quality assessment results through a large language model, determining a problem pattern, and generating prompt word modification suggestions based on the problem pattern, a preset strategy library, and preset heuristic rules to adjust the prompt words.

[0038] In some embodiments of the present application, after obtaining the quality assessment results, the quality assessment results are input into the prompt word adaptive optimization algorithm, and the quality assessment results are classified according to preset evaluation indicators to obtain qualified results and unqualified results.

[0039] Furthermore, the large language model is used to identify and classify sample defects in the compliance results, and determine that the problem pattern is empty; when the evaluation results do not meet the standards or the evaluation model (including the lightweight discriminant model) identifies specific problems (such as poor question and answer quality, repeated content), the system automatically identifies and classifies the defects, and can use the semantic understanding ability of the large model to refine the problem pattern and determine the problem pattern type.

[0040] Based on the identified question type, the system leverages the large model's inherent instruction-following and generation capabilities, combining it with a predefined policy library and heuristic rules to automatically generate targeted prompt word modification suggestions. Specifically, if the word count of a question-answer pair exceeds a preset limit, instructions are generated to constrain the question-answer pair to the preset word limit. If the label granularity is too coarse, instructions are generated to classify the question-answer pair into three subcategories. If the question-answer pair does not match the original data, instructions are generated to constrain the question-answer pair based on the original text content. This process aims to maximize the large model's self-optimization potential.

[0041] S105: The adjusted prompt words are re-input into the large language model to generate a new training sample set, and the quality assessment, question refinement, and prompt word adjustment processes are iterated repeatedly until the sample quality reaches the preset quality standard, and the optimized training data set is output.

[0042] In some embodiments of the present application, the prompt words are optimized based on the generated prompt word modification suggestions, the prompt words optimized by the large model are re-input into the large language model to generate a new training sample set, and the above-mentioned quality assessment, problem refinement and prompt word adjustment process are repeated.

[0043] This cycle continues until the quality of the generated samples reaches the preset standard (such as accuracy ≥ 90%, coverage ≥ 85%), and the optimized training data set is output, aiming to achieve full automation of prompt words and ultimately achieve automated optimization of prompt words.

[0044] It should be noted that to address potential issues with the adaptive optimization algorithm and limitations of the evaluation function, all automatically generated and annotated results can be efficiently manually reviewed through the interactive interface provided by intelligent tools. Reviewers can correct and optimize question-answer pairs, domain labels, domain trees, and other generated content.

[0045] Specifically, after each automated iteration, new training samples are sent to the client. Reviewers use the interactive interface to correct any remaining issues (such as incorrect labels and unreasonable question-answer pairs). This manual feedback is fed back into the prompt word adaptive optimization algorithm and serves as new optimization instructions for further iterative adjustments to the prompt words, addressing the limitations of automated evaluation. This creates a closed loop of automated iteration plus manual feedback iteration, thereby continuously improving the accuracy and quality of automated generation and annotation.

[0046] It should also be noted that in some embodiments of the present application, the format of the data set to be exported will be determined based on the preset task requirements and the format matching table, and the optimized training data set will be converted into a format (such as JSON, JSONL and other standard formats) according to the data set format, and then the converted optimized training data set will be exported.

[0047] The exported dataset format is compatible with mainstream AI model training frameworks and API interfaces, and is also used to train and fine-tune large language models and other AI models, thereby enabling the development and performance improvement of downstream AI applications (such as automatic report generation systems, document structured extraction systems, etc.).

[0048] It should be noted that although the embodiments of this application are based on Figure 1 Steps S101 to S105 are described in sequence, but this does not mean that steps S101 to S105 must be performed in a strict order. Figure 1 The order shown in FIG1 is to introduce and explain steps S101 to S105 in order to facilitate those skilled in the art to understand the technical solutions of the embodiments of the present application. In other words, in the embodiments of the present application, the order of steps S101 to S105 can be appropriately adjusted according to actual needs.

[0049] pass Figure 1 This application uses a large language model prompt word project combined with an intelligent construction method of a prompt word adaptive optimization algorithm to significantly improve the automation level and efficiency of data set construction, while ensuring the high quality and accuracy of the output data; at the same time, the large prediction prompt word project directly drives the large language model API to perform automatic construction work, realizing a highly automated, intelligent and controllable data set construction; the iterative process of the prompt word adaptive optimization algorithm aims to minimize manual intervention, maximize the effectiveness of prompt words, and realize the transformation of data generation from static to dynamic, and from manual to adaptive. This not only greatly reduces the cost and threshold of data preparation, but also provides a solid data foundation for the rapid iteration of AI models and the implementation of various intelligent applications, and effectively promotes the process of digital transformation in various industries.

[0050] Figure 2 A flowchart of the specific steps of a dataset construction method provided in an embodiment of the present application.

[0051] exist Figure 2The overall process of dataset construction is demonstrated, starting from the program, through various processing of the original data, generating high-quality samples (initial training sample set), and then through iterative cycle quality assessment, problem refinement and prompt word adjustment process, the high-quality samples are processed to obtain an optimized dataset, which is also manually reviewed and evaluated again, and the qualified dataset is exported in the required format.

[0052] Figure 3 A schematic diagram of a data set construction device provided in an embodiment of the present application includes:

[0053] at least one processor; and,

[0054] a memory communicatively connected to at least one processor; wherein,

[0055] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any one of the above-mentioned data set construction methods.

[0056] Some embodiments of the present application provide a data set construction non-volatile computer storage medium storing computer executable instructions capable of executing any one of the above-mentioned data set construction methods.

[0057] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0058] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0059] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0063] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0064] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM), and non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0065] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0066] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0067] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the technical principles of the present application should fall within the scope of protection of the present application.

Claims

1. A method for constructing a data set, characterized in that: The method comprises: Through standardized logic programs and large language model prompts, the original data is format converted, multi-format parsed, and logically segmented to obtain standardized processed text; Extracting domain labels of the standardized text using large language model prompt words, and generating question-answer pairs and multi-genre question-answer samples based on the domain labels to construct an initial training sample set; Performing a quality assessment on the initial training sample set using a validation set evaluation function to obtain a quality assessment result; Refining the quality assessment results through a large language model to determine the problem pattern, and generating prompt word modification suggestions based on the problem pattern, a preset strategy library, and preset heuristic rules to adjust the prompt word; The adjusted prompt words are re-input into the large language model to generate a new training sample set. The quality assessment, problem refinement, and prompt word adjustment process are iterated repeatedly until the sample quality reaches the preset quality standard and the optimized training dataset is output.

2. The method according to claim 1, characterized in that The standardized logic program and the large language model prompt words are used to convert the original data into a format, parse multiple formats, and perform logical segmentation to obtain standardized processed text, specifically including: Converting the structured data in the original data through a standardized logic program to obtain data in JSON format; Using a large language model prompt word-driven tool to identify and parse raw data in various formats, and convert the raw data in various formats into text data in a unified format; According to the text segmentation logic, the text data is segmented by using the large language model prompt words to obtain standard text segments; The JSON format data and the standard text fragment are integrated to obtain a standardized processed text.

3. The method according to claim 1, characterized in that The process of extracting domain labels from the standardized text using the large language model prompt words and generating question-answer pairs and multi-genre question-answer samples based on the domain labels to construct an initial training sample set specifically includes: Refining the domain labels according to preset classification rules, and constructing a domain knowledge tree based on the refined domain labels; Match the large language model prompt words with each refined domain in the domain knowledge tree to generate question-answer pairs corresponding to the covered domain knowledge points; Performing genre conversion on the standardized text according to the preset genre and the question-answer pair to generate multi-genre question-answer samples; The multi-genre question-answer samples and the question-answer pairs are integrated to obtain an initial training sample set.

4. The method according to claim 1, wherein The method uses a large language model to extract problems from the quality assessment results, determine problem patterns, and generate prompt word modification suggestions based on the problem patterns, a preset strategy library, and preset heuristic rules, specifically including: Classify the quality assessment results according to the preset assessment indicators to obtain qualified results and unqualified results; Identify the compliance results and classify sample defects using a large language model to determine that the problem pattern is empty; Identify the substandard results and classify sample defects through a large language model to determine the problem pattern type; According to the preset strategy library and preset heuristic rules, the large language model instruction generation capability is used to parse the question pattern type and generate prompt word modification suggestions.

5. The method according to claim 4, characterized in that The method parses the question pattern type based on the preset strategy library and preset heuristic rules and generates prompt word modification suggestions through the large language model instruction generation capability, specifically including: If the number of words in the question and answer pair exceeds the preset limit, a constraint instruction for the preset limit of the number of words in the question and answer pair is generated; If the label granularity is too coarse, generate instructions for the three subcategories of the question-answer pair labels; If the question-answer pair does not match the original data, generate instructions for the question-answer pair based on the original content constraints.

6. The method according to claim 1, characterized in that The method further comprises: The initial training sample set is quality evaluated by a lightweight discriminant model to obtain a quality evaluation result; the lightweight discriminant model is trained by model distillation technology and learns from the evaluation capabilities of a large language model and manually annotated data.

7. The method according to claim 1, characterized in that After re-inputting the adjusted prompt words into the large language model to generate a new training sample set, the method further includes: Sending the new training sample set to the client for manual review to obtain a manual review result; The received manual review results are determined as review feedback suggestions to iteratively update the preset policy library and preset heuristic rules.

8. The method according to claim 1, characterized in that The quality assessment, question refinement, and prompt word adjustment processes are iterated until the sample quality reaches a preset quality standard and an optimized training data set is output. The method further includes: Determine the format of the dataset to be exported based on the preset task requirements and the format matching table; The optimized training dataset is format-converted according to the dataset format, so as to export the converted optimized training dataset.

9. A data set construction device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the data set construction method described in any one of claims 1 to 8.

10. A data set construction storage medium storing computer executable instructions, characterized in that: The computer-executable instructions can execute the data set construction method described in any one of claims 1 to 8.

Images