Method and system for generating and labeling data samples

Abstract

The application relates to a data sample generation and labeling method and system, which comprises the following steps: checking received seed data samples and analyzing received natural language description information; performing an expansion operation on the seed data samples by using a preset generative large model to generate a basic data set, wherein the basic data set comprises multiple sample data; analyzing a task corresponding to the natural language description information by using the preset generative large model to determine a task type, and recommending a corresponding model training paradigm to a user based on the task type; performing data labeling on each sample data in the basic data set according to the model training paradigm selected by the user, and outputting a labeled data set. The scheme can realize an end-to-end automatic and interactive data generation and labeling process, and solve the technical defects of the prior art, such as fragmentation, interaction absence and high use threshold.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for generating and labeling data samples. Background Technology

[0002] Large Language Models (LLMs), with their powerful semantic understanding and content generation capabilities, have rapidly moved from academic research to AI production across various industries. The instruction-following ability, complex task execution capabilities, and domain knowledge reasoning performance of large models are highly dependent on the scale and quality of the training dataset. Currently, mainstream training paradigms for large models include Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO). Each paradigm has different requirements for labeled data: SFT training relies on standardized instruction-response sample pairs, while RLHF and DPO training require preference-based labeled datasets with discriminative power. Therefore, large-scale, high-quality labeled data has become a core production factor for the optimization and deployment of large models.

[0003] Traditional manual annotation methods suffer from several inherent drawbacks in large-scale dataset production scenarios. Firstly, manual annotation is costly and inefficient. Building high-quality datasets covering multiple business domains and containing tens of thousands to millions of samples requires continuous investment of significant human, financial, and time resources. For vertical domain tasks, annotators also need to possess relevant industry expertise, further raising the implementation threshold for data collection and annotation. Secondly, the consistency and diversity of manually annotated data are difficult to guarantee. Differences in the subjective understanding of annotators can easily lead to discrepancies in annotation results for similar instructions, with annotation discrepancies being particularly prominent when facing ambiguous or complex instructions. The limited coverage of manually designed instructions and samples makes it difficult to match the numerous long-tail scenarios in real-world business, directly weakening the generalization performance of large models facing unknown tasks. Existing traditional data augmentation methods can only perform superficial formatting and text rewriting transformations on existing samples, failing to generate novel and challenging training samples from a semantic dimension, thus failing to fundamentally expand the semantic coverage of the dataset.

[0004] To address the shortcomings of manual annotation methods, a growing trend in the industry is towards technologies that leverage the native capabilities of large models to generate synthetic data. Self-Instruct technology proposes a model bootstrapping generation scheme, relying on only a small number of manually written seed instructions. Through iterative task generation, validity verification, and sample generation processes, it produces new instructions and accompanying samples in batches, significantly lowering the data supply threshold for manual annotation. Building on this, Evol-Instruct technology introduces an instruction evolution enhancement mechanism. Through constraint stacking, content visualization, and logic deepening, iterative optimization of original instructions effectively improves the complexity and scenario diversity of synthetic data. Meanwhile, modular data processing frameworks such as DataFlow and Data-Juicer attempt to automate data cleaning, filtering, enhancement, and quality assessment processes through standardized data operators, improving the reproducibility and execution efficiency of data engineering work. Summary of the Invention

[0005] The inventors discovered that while the aforementioned synthetic data generation and automated processing technologies alleviate the pain points of traditional manual annotation to some extent, the existing technical system still has significant shortcomings and cannot meet the needs of one-stop labeled data production. First, the related technical tools are independent of each other, and the technology stack is fragmented. Data generation modules, data processing frameworks, and data formatting tools adapted to different training paradigms such as SFT / DPO / GRPO belong to independent systems. Users need to manually connect multiple tool chains, resulting in a lengthy overall operation process and requiring high engineering and technical capabilities from operators. Second, existing automated pipelines primarily rely on offline one-click execution throughout the entire process, lacking a human-computer interaction mechanism. Users cannot intervene and control the process at key nodes; they cannot dynamically adjust the generation strategy based on the data quality produced in intermediate stages. If the system outputs a recommended training scheme, there is no interactive channel for users to confirm and modify the adaptation scheme, resulting in poor process controllability. Finally, the existing toolchain is not well-suited for non-professional business personnel. Even if only a small number of seed samples and task descriptions are required to start data generation, the complete process of data augmentation, training paradigm selection, and generation of dedicated labeled datasets still requires users to have complete professional knowledge of the underlying artificial intelligence, making independent operation quite difficult.

[0006] In summary, the industry urgently needs an integrated intelligent system that combines cutting-edge synthetic data generation algorithms with human-computer interaction and control capabilities. This system would streamline the entire process from inputting a small amount of seed data to outputting massive, multi-format, high-quality labeled datasets, enabling end-to-end automated and interactive data generation and labeling workflows. This would address the technical shortcomings of existing tools, such as fragmentation, lack of interaction, and high barriers to entry.

[0007] According to a first aspect of this application, a method for generating and labeling data samples is provided, characterized by comprising: Verify the received seed data samples and parse the received natural language description information; The seed data samples are expanded using a pre-defined generative large model to generate a basic dataset, wherein the basic dataset includes multiple sample data. The preset generative large model analyzes the task corresponding to the natural language description information to determine the task type, and recommends the corresponding model training paradigm to the user based on the task type; and Data annotation is performed on each sample data in the base dataset according to the model training paradigm selected by the user, and the labeled dataset is output.

[0008] According to a second aspect of this application, a data sample generation and annotation system is provided, characterized in that it includes: The interactive task definition module is used to verify the received seed data samples and parse the received natural language description information. The data augmentation module is used to expand the seed data samples using a preset generative large model to generate a basic dataset, wherein the basic dataset includes multiple sample data. The training method recommendation module is used to analyze the task corresponding to the natural language description information using the preset generative large model to determine the task type, and recommend the corresponding model training paradigm to the user based on the task type; and The annotation generation module is used to perform data annotation on each sample data in the base dataset according to the model training paradigm selected by the user, and output the annotated dataset.

[0009] According to a third aspect of this application, an electronic device is provided, comprising: Processor; and A memory storing computer instructions that, when executed by the processor, cause the processor to perform the method described in the first aspect.

[0010] According to a fourth aspect of this application, a non-transitory computer storage medium is provided, which stores a computer program that, when executed by a plurality of processors, causes the processors to perform the method described in the first aspect.

[0011] Based on the data sample generation and annotation method and system provided in this application, an interactive, automated generation process from seed data to multi-format labeled data is constructed. This process requires only a small amount of seed data to automatically generate massive amounts of labeled data, eliminating the need for deep AI expertise, significantly reducing manual annotation input, and lowering the barrier to entry and data production costs. It leverages an instruction evolution mechanism to generate complex, multi-scenario samples, coupled with multi-dimensional quality filtering, to address the issues of insufficient coverage and inconsistent bias in manual samples, thereby improving dataset quality and diversity. The process integrates the entire chain of data augmentation, paradigm recommendation, and annotation formatting, with each module decoupled and scalable, compatible with existing and future new training algorithms. The solution is integrated and easily expandable. Furthermore, interactive nodes can be set in key processes, allowing users to intervene and adjust generation strategies and training schemes, balancing automation efficiency with human controllability. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0013] Figure 1 This is a structural diagram of the data sample generation and annotation system according to an embodiment of this application.

[0014] Figure 2 This is an overall flowchart of the data sample generation and annotation method according to an embodiment of this application.

[0015] Figure 3 This is a flowchart of a method for generating and labeling data samples according to an embodiment of this application.

[0016] Figure 4 This is a flowchart of a method for generating and labeling data samples according to another embodiment of this application.

[0017] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0019] In summary, addressing the issues of high cost and low efficiency of manual annotation in existing technologies, this application provides an integrated interactive intelligent data generation and annotation system based on a large model. It employs a three-stage modular pipeline with a unified scheduling and human-computer interaction mechanism. In the first stage, the interactive task definition module receives a small amount of seed data and a natural language task description from the user. After parsing and verification, it is sent to the intelligent data augmentation module. This module can perform seed analysis, instruction augmentation, sample generation, and quality filtering using multiple operators, outputting an unlabeled basic dataset. In the second stage, the training method recommendation module automatically analyzes task features based on the basic dataset, intelligently recommending suitable schemes such as SFT / DPO / GRPO, and setting mandatory human-computer interaction nodes for user confirmation or adjustment of the training paradigm. In the third stage, the annotation generation module automatically converts the basic dataset into corresponding standard format labeled data according to the user-selected training paradigm. Furthermore, a unified system control and interaction module is added to coordinate the entire process, linking with the large model service to complete inference generation, recording process logs throughout, and outputting data reports.

[0020] Figure 1 This is a structural diagram of a data sample generation and annotation system according to an embodiment of this application. The system aims to provide users with an end-to-end automated generation solution for data from a small amount of seed data to massive amounts of multi-format, high-quality annotated data. Figure 1 As shown, the system includes an interactive task definition module, a data augmentation module, a training method recommendation module, a label generation module, and a unified system control and interaction module.

[0021] In one embodiment, the interactive task definition module serves as the entry point for user interaction with the system, undertaking functions such as receiving requirements, verifying data, and parsing the semantic structure of the task. Users submit two types of core input information through the front-end interface: the first type is a small number of representative, high-quality seed data samples, ranging from 3 to 10, whose content intuitively represents the target task paradigm that the user expects the large model to learn; the second type is a natural language task description, used to fully define the task execution objectives, application scenarios, and specific requirements (such as output rules and constraints).

[0022] In one specific embodiment, if a user needs to construct a fine-tuned dataset for automatically summarizing listed company financial reports, they can upload multiple sets of paired "original financial report - summary text" samples as seed data, along with a natural language task description, such as "Generate a summary of approximately 200 words for the annual financial reports of listed companies, with the summary content focusing on the company's revenue, net profit, and future business development expectations." After the interactive task definition module receives the input information, it drives the built-in data verification unit and semantic parsing unit to perform processing. The core execution logic is to verify the received seed data samples and parse the received natural language description information. First, the data verification unit conducts compliance verification on all seed data samples, which can uniformly check the standardization of sample format and the completeness of sample content, and screen out invalid samples that are incomplete, have disordered format, or deviate significantly from the task requirements, thus ensuring the reliability of the basic data source for the subsequent data amplification process. Secondly, the semantic parsing unit calls the pre-trained natural language understanding model to perform deep semantic extraction on the task description text input by the user, extracting multi-dimensional structured task features from the unstructured natural language. For example, it can include the business domain to which the task belongs (finance, medical care, code development, etc.), task intent (text summarization, sentiment classification, code generation, etc.), and output constraints (text word limit, JSON standard output format, etc.).

[0023] The standardized task structured information obtained after verification and parsing can be uniformly pushed to the unified system control and interaction module, serving as the core guiding basis for each stage of the entire process scheduling, data augmentation, training paradigm recommendation, and annotation generation. The structured information can include task description information and seed data samples. The task description information can include task objectives, application scenarios, and format constraints for input and output requirements.

[0024] In one embodiment, the data augmentation module is the core unit of the first stage of the system's data production process. Its core function is to call a pre-defined generative large model to perform augmentation operations on the limited input seed data samples, producing a sufficient number of semantically diverse basic datasets. These basic datasets contain multiple sample data points, which share the same feature distribution as the seed data, and have not undergone annotation and formatting processing for specific training paradigms such as SFT, DPO, and GRPO. The core objective of this stage is to expand the total number of samples and enrich the diversity of task scenarios. The complete execution flow of the module is as follows: The first step involves activating the seed data analysis operator to perform deep feature analysis on the seed data samples validated by the interactive task definition module, outputting standardized seed data analysis results. The operator can identify and extract implicit instruction patterns, data structures, and semantic features from the seed samples. Instruction patterns can be the way the task is expressed, such as "Please summarize…", "Please classify…", "Please extract…"; data structures can be the input and output organization of the samples, for example, in a financial statement summary task, the input is the full text of the financial statement, and the output is a summary text; semantic features can be the semantic relationship between the input and output, such as the relationship between the full text of the financial statement and the summary being "a long text summarized into a short text". This step can be implemented based on a method of "preset parsing prompts calling a large model + rule validation". The specific implementation process is as follows: First, the system performs format verification on the seed samples submitted by the user to confirm that each sample contains input and output content; then, the seed samples and task descriptions are input into the generative large model, and the generative large model is required to output the task analysis results according to preset fields, which may include task type, input structure, output structure, constraints, and input-output semantic relationships; finally, the rule verification unit checks whether the above analysis results are consistent with the seed samples.

[0025] In a specific embodiment, taking the financial statement summary task as an example, the operator can identify that the sample input is a long text financial statement and the output is a concise summary text. There is a text summary mapping relationship between the two, and the analysis result can be used as the underlying basis for the generation of subsequent instructions.

[0026] The second step involves scheduling instruction generation and evolution operators. Based on the aforementioned seed data analysis results, a hybrid generation strategy is used to complete the generation and iterative enhancement of new instructions. Firstly, the instruction generation and evolution operators can employ a method similar to Self-Instruct, using several seed samples as context input to pre-set a high-capacity generative large-scale model (e.g., GPT-4 level). This drives the model to generalize and generate multiple new instructions with different sentence structures, perspectives, and expressive focuses, but with a unified core task objective.

[0027] In one specific embodiment, taking the financial statement summary task as an example, the basic instruction "generate a summary for the financial statement" can be extended to several differentiated instructions such as "summarize the core financial indicators of this financial statement" and "explain the performance highlights of this financial statement in plain language".

[0028] In an optional embodiment, after the original new instruction is generated, a front-end human-computer interaction node can be added, which can receive adjustment instructions issued by the user and perform one or more adjustment operations such as filtering, editing, and adding supplements to the original new instruction according to the user's needs, so as to guide the overall direction of subsequent data generation from the source of the instruction.

[0029] In one embodiment, the instruction generation and evolution operator can introduce the Evol-Instruct instruction evolution mechanism to perform iterative enhancement operations on new instructions. This can be achieved by increasing the difficulty and richness of instructions through methods such as task complexity, constraint superposition, and content visualization. For example, the basic instruction "summarize financial report" can be enhanced to generate "summarize this financial report and compare the operating performance of the previous year"; or constraint superposition enhancement can be performed to generate "summarize the financial report content in the form of sub-items and within 150 words", ultimately resulting in multiple instructions after iterative enhancement.

[0030] The third step is to run the sample generation operator and push all the iteratively enhanced instructions one by one to the preset generative large model. The large model then generates complete input-output paired data samples that match the task logic of the instructions, resulting in a batch of original generated sample sets.

[0031] Fourth, after generating all the original samples, the module can execute data filtering and evaluation operators to construct a compliant basic dataset based on the batch of original data samples. In a specific embodiment, the operator can independently score each original sample according to preset multi-dimensional evaluation indicators (e.g., instruction matching degree, factual accuracy, logical consistency, text fluency, etc.) to determine the evaluation score corresponding to each sample; it can retain only qualified samples with evaluation scores higher than a preset quality threshold, and remove invalid samples that are low-quality, logically contradictory, or deviate from the task requirements. The set of qualified samples retained after screening becomes the basic dataset for use in the next stage.

[0032] In one embodiment, the training method recommendation module is the core unit of the second stage of the entire data production process, and a key carrier for realizing intelligent decision-making and human-computer collaborative interaction. The module receives the high-quality unlabeled basic dataset output by the first-stage intelligent data augmentation module, calls a pre-defined generative large model, and performs comprehensive task analysis by combining the natural language description information parsed by the preceding interactive task definition module. It accurately determines the task type corresponding to the current task and, based on task type matching logic, pushes a suitable model training paradigm to the user, completing the intelligent recommendation and triggering the human-computer interaction confirmation process.

[0033] The specific task type determination logic relies on a large model to deeply analyze all instructions and input / output sample structures within the basic dataset, enabling it to distinguish between different task types such as open-ended generation, classification, and preference selection. Based on the determined task type, the system matches the corresponding training paradigm, provides targeted recommendations for different scenarios, and simultaneously displays the required labeled data structures for each paradigm.

[0034] In a specific embodiment, if the task is determined to be a general generation task such as text summarization, machine translation, or open-ended content generation, the module can recommend the SFT (Supervised Fine-Tuning) training paradigm and simultaneously inform the user that the standard data format of this paradigm is {"instruction": "...", "input": "...", "output": "..."}; If the task is determined to be one that requires aligning the model output with human values ​​and making trade-offs among multiple options, the module can recommend the DPO (Direct Preference Optimization) training paradigm, while explaining that this paradigm needs to construct preference sample pairs that include high-quality answers chosen and low-quality answers rejected. If the task is determined to be complex, such as mathematical operations, coding, or complex logical reasoning, the module can recommend the GRPO (Group Relative Policy Optimization) training paradigm. It is specified that this paradigm should generate multiple candidate answers for a single instruction, and form structured data with the quality scores corresponding to each answer.

[0035] The training method recommendation module can push all training paradigm recommendation results to the front-end interactive interface in the form of visual cards or lists, clearly listing the suitable scenarios, technical advantages and application limitations of each recommended paradigm, and fully presenting the data preparation requirements of each paradigm. Then, the entire process scheduling is paused, waiting for the user to complete the operation instructions to confirm, modify or change the training paradigm, and the user's decision results are used as the basis for the execution of the next stage of the automated annotation generation module.

[0036] In one embodiment, the annotation generation module is the third-stage execution unit in the entire data production process, activated after the user completes interactive confirmation and selects the target model training paradigm in the training method recommendation module. The annotation generation module is essentially a dedicated data format conversion and processing unit. Its core execution logic is to automatically perform standardized annotation processing on each sample data in the general basic dataset produced in the first stage, according to the model training paradigm selected by the user, and finally output a dedicated annotation dataset adapted to the corresponding training process.

[0037] In one specific embodiment, the differentiated processing logic for the three mainstream training paradigms is as follows: When the user selects SFT (Supervised Fine-Tuning) as the model training paradigm, the annotation generation module can directly encapsulate each sample data in the basic dataset into a structured form, encapsulating each sample into an independent single-line JSON (JavaScript Object Notation) object. The object uniformly contains three types of standard fields: instruction, input, and output. After all encapsulation is completed, the standard JSONL (JavaScript Object Notation Lines) format annotation files are generated in batches without the need to call the generative large model to supplement the generated content.

[0038] When the user selects DPO (Direct Preference Optimization) as the model training paradigm, the module iterates through the instructions corresponding to each sample in the basic dataset and repeatedly calls the preset generative large model to complete two rounds of differential generation: the first round uses standard positive prompt words to generate high-quality chosen answers that meet the standards of completeness, accuracy, and fluency; the second round adjusts the constraints of the prompt words, such as issuing a generation instruction to the model that "the output content is not complete or accurate enough", generating defective rejected answers; the instruction prompt, high-quality chosen answers, and rejected answers are bound and assembled to form the standard preference sample pair data structure {"prompt": "...", "chosen": "...", "rejected": "..."}.

[0039] When the user selects GRPO (Group Relative Policy Optimization) as the model training paradigm, the module calls the generative large model for each instruction in the sample to generate a preset number of N groups of differentiated candidate answers (e.g., N=4), forming a complete answer group. Then, through the reward model or the generative large model acting as a referee, the quality of all candidate answers in the answer group is scored one by one, and finally, composite structured labeled data that simultaneously carries multiple candidate answer groups and corresponding scores is constructed.

[0040] After all samples have completed paradigm adaptation and annotation processing, the module summarizes all structured samples and generates a complete labeled dataset, along with a final data quality review and statistical report, which users can download directly for subsequent training of large models.

[0041] In one embodiment, the unified system control and interaction module serves as the global scheduling and interaction hub for the entire system, undertaking core functions such as overall business process coordination and driving, process node control, human-machine interaction scheduling, operation log retention, and data production report generation. The complete system workflow can be correspondingly attached. Figure 2 The logic is shown.

[0042] The entire process begins with the user inputting seed data and task description. The module automatically schedules the interactive task definition module and the intelligent data augmentation module to sequentially complete requirement analysis and basic dataset generation. After the training method recommendation module outputs the training paradigm recommendation result, the global process is temporarily interrupted. The module initiates forced human-computer interaction through the front-end interface, waiting for the user to complete the confirmation, modification, or change of the training paradigm. This interaction node is the core implementation form of the applicant's human-computer collaborative controllable design. After receiving the user's confirmation instruction, the module continues to schedule the automated annotation generation module to perform standardized annotation processing until a complete annotation dataset that can be directly exported and used is generated.

[0043] The unified system control and interaction module continuously collects and retains detailed end-to-end operation logs at every execution step of the process, fully recording information such as operator execution parameters of each module, large model call records, sample selection rules, and manual interaction operation records. After the entire data generation and annotation process is completed, the module automatically generates a complete data sample generation and annotation report. The report includes a list of final output documents, the overall expansion scale of the basic dataset relative to the original seed data sample, and multi-dimensional information such as data quality assessment statistics at each stage, supporting users to conduct complete traceability, effect verification, and compliance auditing of the entire data production process.

[0044] In one optional embodiment, in the data filtering and evaluation stage of the intelligent data augmentation module, the evaluation operator used to complete the generation of sample quality scoring and low-quality sample removal is not limited to a single judge-style large-scale generation model. Multiple equivalent alternatives can also be used to achieve the same data quality control function, and each alternative has a clear technical implementation logic: Deterministic rule filtering component combination scheme: Configure multiple sets of rigid verification rules to form a rule filter cluster, set exclusive judgment criteria for different task types, such as verifying the compileability and executability of code samples, verifying the legality of JSON format for structured output samples, and filtering invalid samples containing illegal sensitive words for general text samples. Relying on fixed logic, it can quickly screen out obviously unqualified generated samples without calling large model inference resources.

[0045] Multi-dimensional dedicated classification model integration scoring scheme: Deploy multiple sets of lightweight dedicated classification models, each model corresponding to a single quality evaluation dimension, including text fluency, instruction relevance, and content factual accuracy. The dimensional scores output by each model are weighted and fused to obtain a comprehensive sample score. Sample selection is completed based on the multi-model integration judgment results. Compared with a single large judge model, this can reduce the computational overhead caused by the high frequency of large model calls.

[0046] Embedding Space Distance Matching Verification Scheme: For business tasks with fixed sample distribution requirements, vector embedding representations are extracted from seed data samples and amplified generated samples respectively. By calculating the vector distance between the two types of samples in the embedding space, the consistency of data distribution between the generated samples and the original seed samples is quantitatively determined. Abnormal samples with serious distribution deviations and deviating from the task characteristics are eliminated, ensuring the overall homogeneity of the basic dataset from the data distribution level.

[0047] The aforementioned quality assessment alternatives can be deployed and used independently or combined with the large-scale evaluation mechanism of the referee model to achieve graded scoring and threshold screening of batch-generated samples, thereby achieving the technical goal of ensuring the overall quality of the basic dataset.

[0048] In one optional embodiment, the core human-computer interaction node of this application is set after the paradigm recommendation result is output by the training method recommendation module, so as to realize the manual confirmation and adjustment of the training scheme. Without changing the overall three-stage data generation and annotation process of this application or the core function of the system, the placement position of the interaction node can be added or replaced to form multiple equivalent interaction implementation schemes.

[0049] One typical alternative interaction node can be deployed at the end of the entire process: after the automated annotation generation module outputs a complete labeled dataset, the unified system control and interaction module loads a visual data browsing interface, supporting users to perform random sampling verification and fine-tuning of local content correction on batch labeled samples. The system collects sample defects and optimization requirements reported by users during manual verification and correction, and feeds back the corrected high-quality samples to the front-end intelligent data augmentation module as new seed data to supplement the data augmentation process of subsequent batches, thus constructing a closed-loop iterative link for active learning.

[0050] The terminal interaction node scheme and the original paradigm recommendation process interaction mechanism can be used independently or deployed in combination; both can grant users the right to manually intervene in the data production process, achieving the core effect of human-machine collaborative regulation of data generation quality, and thus fall within the protection scope of this application.

[0051] exist Figure 1 and Figure 2 Based on the scheme shown, according to one aspect of this application, a method for generating and labeling samples is provided. Figure 3 This is a flowchart illustrating a method for generating and labeling data samples according to an embodiment of this application. Figure 3 As shown, the method includes the following steps: Step S301: Verify the received seed data sample and parse the received natural language description information; Step S302: The seed data samples are expanded using a preset generative large model to generate a basic dataset, wherein the basic dataset includes multiple sample data. Step S303: Analyze the task corresponding to the natural language description information using the preset generative large model to determine the task type, and recommend the corresponding model training paradigm to the user based on the task type; and Step S304: Perform data annotation on each sample data in the basic dataset according to the model training paradigm selected by the user, and output the annotated dataset.

[0052] In an optional embodiment, step S302 may include: The seed data sample is analyzed to obtain the analysis results of the seed data sample; Based on the seed data samples, multiple different new instructions are generated using the preset generative large model. Perform iterative enhancement operations on the multiple different new instructions respectively to generate multiple instructions with iterative enhancement; Multiple data samples are generated based on the iteratively enhanced multiple instructions using the preset generative large model; and The base dataset is generated based on the multiple data samples.

[0053] In an optional embodiment, generating the base dataset based on the plurality of data samples may include: The multiple data samples are evaluated according to preset evaluation dimensions to determine the evaluation scores of the multiple data samples; and The data samples whose evaluation scores are greater than a preset threshold are retained from the plurality of data samples to generate the basic dataset.

[0054] In an optional embodiment, after generating multiple different new instructions based on the seed data sample using the preset generative large model, the step of expanding the seed data sample using the preset generative large model to generate the basic dataset further includes: Based on the adjustment instructions input by the user, adjustment operations are performed on the plurality of different new instructions, wherein the adjustment operations include one or more of row filtering, editing, and adding.

[0055] In an optional embodiment, step S304 may include: When the model training paradigm is supervised fine-tuning, each sample data in the base dataset is encapsulated into a single line of JSON and then concatenated into a standard JSONL file. When the model training paradigm is direct preference optimization, the generative large model is invoked to generate superior and inferior responses to the instructions in each sample data, and these responses are assembled to form preference pairs; and When the model training paradigm is group relative strategy optimization, the generative large model is invoked to generate an answer group containing a preset number of different candidate answers for each instruction in the sample data, and the preset number of different candidate answers are scored to generate a data structure containing the answer group and the corresponding score.

[0056] Figure 4 This is a flowchart illustrating a method for generating and labeling data samples according to another embodiment of this application. Figure 3 compared to, Figure 4 Steps S401 to S404 in the method shown are Figure 3 Steps S301 to S304 of the method shown are the same, except that, Figure 4 The method shown may also include: Step S405: Record the logs for each step and generate a data sample generation and annotation report. The report includes a file list and the amount of augmented data in the base dataset relative to the seed data sample.

[0057] The data sample generation and annotation method and system provided in this application, compared with existing decentralized data generation and annotation toolchains, rely on an integrated multi-module collaborative architecture and human-computer interaction design, and have the following significant technical advantages: I. Significantly reduce the threshold for training dataset production and overall implementation costs Existing technologies require the manual integration of multiple independent tools and demand operators with comprehensive expertise in artificial intelligence and data engineering. Furthermore, large-scale datasets heavily rely on manual annotation, resulting in high costs in terms of manpower, time, and capital. This invention highly integrates and automates the entire data engineering process, from seed data parsing and data augmentation to training paradigm recommendation and multi-format annotation generation. Users only need to provide 3 to 10 seed data samples and a natural language task description to automatically generate massive amounts of high-quality training datasets. This design reduces the rigid requirements for operators' professional backgrounds, allowing business personnel without deep AI expertise to independently complete the data production work required for model fine-tuning. It replaces large-scale manual annotation work from the source, significantly reducing overall costs related to manpower, computing power scheduling, and process debugging.

[0058] II. Effectively improve the sample diversity and overall data quality of the generated dataset. Existing manually labeled samples have limited scenario coverage and are prone to subjective bias. Traditional data augmentation only performs superficial text rewriting and cannot achieve semantic-level sample innovation. Samples generated by a single referee model are prone to homogenization. This invention deeply integrates Self-Instruct bootstrapping generation logic and Evol-Instruct instruction evolution enhancement mechanism in the intelligent data augmentation module. Based on the seed data analysis results, it generates differentiated instructions in batches and iteratively optimizes the instructions through complexity, constraint superposition, and scenario visualization to generate new samples with rich semantics and higher task complexity. At the same time, it is equipped with multi-dimensional data evaluation operators to complete sample screening, retaining only qualified samples with evaluation scores higher than a preset threshold. This ensures that the augmented samples are distributed in the same source as the original seed data and makes up for the shortcomings of manual annotation and traditional data augmentation in terms of scenario diversity. The high-quality dataset constructed in this way can support downstream training to obtain large models with better robustness and generalization ability.

[0059] III. Construct a three-stage integrated automated production line to achieve one-stop data production. In existing technologies, data generation, data cleaning and processing, and formatted annotation for different training paradigms such as SFT / DPO / GRPO are fragmented. Users need to manually connect multiple independent tools, resulting in lengthy processes, numerous manual interventions, and a high risk of human errors such as formatting errors and process interruptions. This invention integrates four core processes—interactive task definition, intelligent data augmentation, intelligent training method recommendation, and automated annotation generation—into a logically coherent and seamlessly connected standardized automated pipeline. It streamlines the entire process from initial user input to output of labeled datasets directly usable for model training, forming a one-stop data production solution that avoids the cumbersome operation and high error rate associated with manually connecting multiple tools.

[0060] IV. A human-centered human-computer collaborative interaction architecture enhances system controllability and usability. Existing automated data generation processes are mostly offline, one-click, black-box executions with no human intervention. Users cannot control the generation logic or modify training schemes, resulting in low transparency and poor controllability. This invention establishes a mandatory human-computer interaction mechanism at key nodes throughout the process. After the training method recommendation module outputs the appropriate paradigm, the process pauses, visually demonstrating the applicable scenarios, advantages, disadvantages, and data format requirements of each training paradigm to the user. Users can then independently confirm, modify, and change training paradigms. Simultaneously, it supports adding interactive nodes at the instruction generation stage and the end of the process, allowing users to filter and edit instructions and fine-tune the final dataset through sampling and verification, forming an active learning loop. This human-computer collaborative model combines the high-efficiency output advantages of automated pipelines with entrusting core decision-making power to users, eliminating the uncertainty of data production caused by purely black-box automation, and enhancing the system's practical value and user trust.

[0061] V. The overall modular and decoupled design provides excellent scalability and long-term adaptability. All functional units of this system adopt an independent and decoupled modular design. Components such as seed data analysis operators, instruction evolution operators, training paradigm recommendation units, and various annotation generators are independent and have standardized interfaces. Based on this architecture, the system can easily integrate newly developed data generation and sample evaluation algorithms, and is compatible with future industry additions to reinforcement learning and fine-tuning training paradigms. Simultaneously, it can customize dedicated data processing operators for vertical business domains such as finance, healthcare, and coding, without reconstructing the entire system's underlying logic. This allows for long-term adaptation to technological iterations in the field of artificial intelligence data engineering, ensuring the solution maintains its technological advancement.

[0062] VI. Full-process log retention and standardized report output, supporting complete traceability and auditing. The unified system control and interaction module continuously collects and stores complete operation logs at each execution step. After the process is completed, it automatically generates a special report on the generation and annotation of data samples. The report includes a list of final output files, the total amount of augmentation of the basic dataset relative to the original seed data, and statistical indicators for the overall data quality assessment. All production stages are fully traceable and verifiable, meeting the business needs of enterprises for data production compliance audits and performance reviews.

[0063] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0064] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be an electrical connection or other forms.

[0066] See Figure 5 , Figure 5An electronic device is provided, including a processor and a memory. The memory stores computer instructions or one or more programs, which, when executed by the processor, cause the processor to execute the computer instructions to achieve the following: Figure 3 and Figure 4 The method and its detailed scheme are shown.

[0067] It should be understood that the above-described device embodiments are merely illustrative, and the device disclosed in this invention can be implemented in other ways. For example, the division of units / modules described in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, integrated into another system, or some features may be ignored or not executed.

[0068] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of the present invention can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0069] When the integrated unit / module is implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor or chip can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the on-chip cache, off-chip memory, and storage can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0070] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer electronic device (which may be a personal computer, server, or network electronic device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0071] This application also provides a computer-readable storage medium storing one or more computer programs, which, when executed by a processor, cause the processor to perform the following actions: Figure 3 and Figure 4 The method and its detailed scheme are shown.

[0072] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the methods of any of the above embodiments.

[0073] References to features, advantages, or similar language in this specification do not imply that all features and advantages achievable with this solution should be included or included in any single implementation thereof. Rather, references to features and advantages are understood to mean that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of this solution. Therefore, discussions of features, advantages, and similar language throughout this specification may, but do not necessarily, refer to the same embodiments.

[0074] Furthermore, the features, advantages, and characteristics described herein can be combined in any suitable manner in one or more embodiments. Based on the description herein, those skilled in the art will recognize that this solution can be implemented without one or more specific features or advantages of a particular embodiment. In other instances, additional features and advantages can be appreciated in specific embodiments not presented in all embodiments of this solution.

[0075] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating and labeling data samples, characterized in that, include: Verify the received seed data samples and parse the received natural language description information; The seed data samples are expanded using a pre-defined generative large model to generate a basic dataset, wherein the basic dataset includes multiple sample data. The preset generative large model analyzes the task corresponding to the natural language description information to determine the task type, and recommends the corresponding model training paradigm to the user based on the task type; and Data annotation is performed on each sample data in the base dataset according to the model training paradigm selected by the user, and the labeled dataset is output.

2. The method as described in claim 1, characterized in that, Also includes: Log each step is recorded and a report on the generation and annotation of data samples is generated. The report includes a list of files and the amount of augmented data in the base dataset relative to the seed data samples.

3. The method as described in claim 1 or 2, characterized in that, The step of expanding the seed data samples using a pre-defined generative large model to generate a basic dataset includes: The seed data sample is analyzed to obtain the analysis results of the seed data sample; Based on the seed data samples, multiple different new instructions are generated using the preset generative large model. Perform iterative enhancement operations on the multiple different new instructions respectively to generate multiple instructions with iterative enhancement; Multiple data samples are generated based on the preset generative large model and the iteratively enhanced multiple instructions; and The base dataset is generated based on the multiple data samples.

4. The method as described in claim 3, characterized in that, The generation of the basic dataset based on the multiple data samples includes: The multiple data samples are evaluated according to preset evaluation dimensions to determine the evaluation scores of the multiple data samples; and The data samples whose evaluation scores are greater than a preset threshold are retained from the plurality of data samples to generate the basic dataset.

5. The method as described in claim 3, characterized in that, After generating multiple different new instructions based on the seed data samples using the preset generative large model, the step of expanding the seed data samples using the preset generative large model to generate a basic dataset further includes: Based on the adjustment instructions input by the user, adjustment operations are performed on the plurality of different new instructions, wherein the adjustment operations include one or more of row filtering, editing, and adding.

6. The method as described in claim 1 or 2, characterized in that, The model training paradigms include supervised fine-tuning, direct preference optimization, and group-relative policy optimization.

7. The method as described in claim 6, characterized in that, The step involves performing data annotation on each sample data in the base dataset according to the model training paradigm selected by the user, and outputting a labeled dataset, including: When the model training paradigm is supervised fine-tuning, each sample data in the base dataset is encapsulated into a single line of JSON and then concatenated into a standard JSONL file. When the model training paradigm is direct preference optimization, the generative large model is invoked to generate superior and inferior responses to the instructions in each sample data, and these responses are assembled to form preference pairs; and When the model training paradigm is group relative strategy optimization, the generative large model is invoked to generate an answer group containing a preset number of different candidate answers for each instruction in the sample data, and the preset number of different candidate answers are scored to generate a data structure containing the answer group and the corresponding score.

8. A data sample generation and annotation system, characterized in that, include: The interactive task definition module is used to verify the received seed data samples and parse the received natural language description information. The data augmentation module is used to expand the seed data samples using a preset generative large model to generate a basic dataset, wherein the basic dataset includes multiple sample data. The training method recommendation module is used to analyze the task corresponding to the natural language description information using the preset generative large model to determine the task type, and recommend the corresponding model training paradigm to the user based on the task type; and The annotation generation module is used to perform data annotation on each sample data in the base dataset according to the model training paradigm selected by the user, and output the annotated dataset.

9. The system as described in claim 8, characterized in that, Also includes: A unified system control and interaction module is used to record logs for each step and generate a data sample generation and annotation report. The report includes a file list and the amount of augmented data in the base dataset relative to the seed data sample.

10. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program in the memory, implements the method of any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images