Data generation method based on small sample seed and multiple rounds of reinforcement and electronic device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MOLAR INTELLIGENCE INFORMATION TECHNOLOGY (HANGZHOU) CO LTD
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-26
Smart Images
Figure CN121765062B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence and natural language processing technology, and in particular relates to a data generation method and electronic device based on small sample seeds and multi-round reinforcement. Background Technology
[0002] With the widespread application of large language models in fields such as natural language understanding and generation, their training effectiveness largely depends on the scale and quality of the training data. However, in vertical fields such as healthcare, law, and finance, acquiring high-quality, labeled data is costly and time-consuming, and the number of manually labeled samples is usually limited. Directly training or fine-tuning large language models cannot meet the requirements of refined semantic modeling and domain consistency for practical tasks.
[0003] To address the aforementioned issues, existing technologies typically employ data augmentation to expand the training data scale. This includes methods such as manually adding annotations, template-based rule generation, and retrieval-enhanced generation to construct additional training samples. However, these methods still have certain shortcomings in practical applications.
[0004] On the one hand, the utilization efficiency of small sample seeds is low. Existing data generation methods typically expand superficially based on a small number of samples, making it difficult to automatically extract and summarize the structured information implicit in the task, such as entity type systems and inter-entity relationship patterns, from small samples. This results in limited coverage of the generated data with the core structure of the task. On the other hand, the generated data lacks effective structural constraints. Generation methods that rely solely on large language models are prone to producing samples that do not meet the task format requirements or are inconsistent with the facts, thereby reducing the usability of training data. In addition, data generated directly from internet content often has deviations in domain consistency and task relevance, making it difficult to effectively support model training for vertical domain tasks. At the same time, most existing data generation strategies are static rules or fixed processes, lacking automatic adjustment mechanisms based on model training feedback, making it difficult to continuously optimize the quality and coverage of generated data during iteration.
[0005] Therefore, there is an urgent need for a data generation method that can automatically extract task structure information under small sample conditions and dynamically optimize the data generation strategy in combination with model feedback, so as to continuously improve the training effect of the generated data on downstream tasks while ensuring data quality. Summary of the Invention
[0006] The purpose of this invention is to provide a data generation method and electronic device based on small sample seeds and multi-round reinforcement, which can automatically construct a high-quality, highly diverse training dataset that meets task requirements under the condition of a limited number of manually labeled samples, thereby effectively improving the generalization ability and stability of large language model intelligent question answering.
[0007] According to a first aspect of the embodiments of this application, a data generation method based on small-sample seeds and multi-round reinforcement is provided, including:
[0008] Obtain labeled seed data related to the training task of the intelligent question answering system, and preprocess the labeled seed data to obtain a standardized seed dataset;
[0009] The standardized seed dataset is processed based on the entity extraction model to construct an initial entity set and relation set;
[0010] The entity set and relation set are normalized and weighted, and an entity-relationship graph is constructed.
[0011] First basic question-and-answer data is generated based on the entity relationship graph;
[0012] Define a language logic template, apply consistency constraints to the first basic question and answer data and perform automatic filtering to obtain the second basic question and answer data;
[0013] The second basic question-and-answer data is enhanced by using external information sources to generate enhanced training data that includes external knowledge support;
[0014] Reinforcement learning is used to generate a reinforcement learning decision process by taking the data generation strategy as the decision action and the reinforcement training data as the input.
[0015] According to the reinforcement learning decision-making process, multiple rounds of reinforcement learning are performed to generate preliminary training sample data, and the data generation strategy is dynamically adjusted based on the performance feedback of the intelligent question answering system, thereby automatically generating training sample data that meets the task requirements.
[0016] According to a second aspect of the embodiments of this application, an electronic device is provided, comprising:
[0017] One or more processors;
[0018] Memory, used to store one or more programs;
[0019] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.
[0020] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.
[0021] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:
[0022] Under small sample conditions, a data generation method is designed to automatically extract task structure information using an entity extraction model and dynamically optimize the data generation strategy by combining feedback from a large language model. Consistency constraints are imposed on the data by defining language logic templates to ensure further enhancement of data features. Enhanced training data with external knowledge support is generated using external information sources, enabling the data to be better trained and used by the model. Reinforcement learning is adopted to continuously improve the training effect of the generated data on downstream tasks while ensuring data quality, so that the quality and coverage of the generated data can be continuously optimized during the iteration process. Attached Figure Description
[0023] Figure 1 This is a flowchart illustrating a data generation method based on small sample seeds and multiple rounds of reinforcement, according to an exemplary embodiment.
[0024] Figure 2 This is a modeling diagram of a reinforcement learning Markov decision process according to an exemplary embodiment.
[0025] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0026] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0027] Figure 1 An exemplary embodiment illustrates a flowchart of a data generation method based on small-sample seeds and multi-round reinforcement. (Reference) Figure 1 This invention provides a data generation method based on small-sample seeds and multi-round reinforcement. This embodiment uses data from intelligent question-answering systems (customer service Q&A, automatic replies) in the e-commerce field as training data, but is not limited to this. The method may include the following steps:
[0028] S1: Obtain labeled seed data related to the training task of the intelligent question answering system, and preprocess the labeled seed data to obtain a standardized seed dataset, including the following sub-steps:
[0029] S11: The acquired labeled seed data is processed to standardize the data format, thereby converting it into a preset data structure. The data structure includes at least the input text, labeling results, and sample identification information.
[0030] Specifically, this involves acquiring labeled seed data relevant to the training task of an intelligent question-answering system in the e-commerce field, and converting the input labeled seed data into a unified format. The input text is typically a question or query, the labeled result is the correct answer or correct relation type, and the sample identification information is a unique identifier within the dataset. Assuming the data structure is converted to the following form... Represents the entire data structure set:
[0031]
[0032] in, The question text is a common question in e-commerce, such as "What is the size of this garment?". The annotation result, obtained from the question text, is represented here as "clothes". The sample identifier, obtained by combining it with the question text, is represented here as "size".
[0033] S12: After completing the data format standardization, perform field integrity verification on the seed data, including field missing detection and outlier detection, and remove samples with missing input text, missing annotation results, or annotation results that do not meet the preset format constraints.
[0034] Specifically, a field integrity check algorithm is used to check for missing fields and outliers in the data. For each sample, it is checked whether it contains both input text and annotation results. If the input text or annotation results are missing, or if the annotation results do not conform to the preset format (e.g., do not conform to the standard answer type or relation format), the sample will be discarded. The annotation result format check can be performed using regular expressions or preset validation rules.
[0035] S13: Based on the field integrity verification, perform text cleaning on the retained input text to obtain cleaned text with clear semantics and standardized format;
[0036] Specifically, considering the potential errors in the problem text, text cleaning was performed, including removing redundant spaces, punctuation marks, and stop words, standardizing text formatting, and unifying capitalization. The cleaned text is represented as follows:
[0037]
[0038] in, This includes operations such as text denoising, case normalization, and removal of unnecessary symbols.
[0039] S14: While obtaining the cleaned text, for the various annotation systems that may exist in the labeled seed data, uniformly map the entity type labels, relation type labels or answer category labels to generate a standardized annotation result that is consistent with the preset label set.
[0040] Specifically, for multiple annotation systems, a unified mapping rule is used to map different entity type labels, relation type labels, or answer category labels to a predefined set of standard labels. For example, if there are different entity types "person" and "human", they can be uniformly mapped to "person". The mapping rule can be implemented using a lookup table or mapping function, and the unified label representation is consistent with the mapped data. :
[0041]
[0042] S15: After completing the label unification and mapping, perform sample deduplication and conflict resolution processing on the labeled seed data;
[0043] Specifically, deduplication is performed, finding and removing samples that are duplicates in the labeled results and the input text. Conflict resolution can be achieved by merging samples with conflicting labels or by selecting one label based on priority.
[0044] S16: After deduplication and conflict resolution, the labeled seed data is divided into training subset and validation subset, and statistical modeling is performed on the labeled seed data to calculate entity coverage, relation coverage, sample length distribution and label distribution information, and finally the preprocessed standardized seed dataset is obtained.
[0045] S2: Process the standardized seed dataset based on the entity extraction model to construct an initial entity set and relation set, including the following sub-steps:
[0046] S21: Extract entities, entity types, and relationships between entities from the standardized seed dataset based on the entity extraction model;
[0047] Specifically, construct the initial set of entities and the set of relations:
[0048] Collect preprocessed, standardized seed datasets to obtain a small amount of task-related labeled datasets. :
[0049]
[0050] Where N represents the total number of samples. Represented as a sample, Let i represent the labeling result of the sample, where i represents the i-th sample.
[0051] The entity set is derived from a small number of labeled datasets. Key entities extracted from it (such as names, locations, times, etc.);
[0052] The entity type collection contains the type of each entity (such as "person name", "place name", "organization name", etc.).
[0053] A set of relationships includes descriptions of the relationships between entities (such as "belongs to", "is located in", "associated with", etc.).
[0054] S22: Aggregate the extracted entities, entity types, and relationships between entities to form a standardized entity library;
[0055] Specifically, for each seed sample, the labeled dataset is obtained. Then, the entity set is obtained using the LLM entity extraction algorithm. Entity type collection Relation set The three are then aggregated to form a standardized entity library.
[0056] S23: Based on the entities, entity types, and relationships between entities, an initial entity set and a relationship set are constructed.
[0057] Specifically, the lightweight entity relation set G for the construction task is as follows:
[0058]
[0059] By summarizing the entities in all seed data, an initial entity set is obtained. This set contains all entities and their relationships appearing in the seed samples, forming the knowledge base foundation required for building the intelligent question-answering system. Next, based on this initial set, the data structure can be further optimized through steps such as merging and deduplication to ensure the comprehensiveness and diversity of entities and relationships, providing richer training data.
[0060] The ultimate goal of this stage is to create an initial knowledge base containing entities and their relationships, which will facilitate more efficient data querying and processing in the question-and-answer system.
[0061] S3 normalizes and weights the entity set and relation set, and constructs an entity-relationship graph, including the following sub-steps:
[0062] S31: Merge entities with the same name or similar semantics in the entity set and relation set, and normalize the entity types to obtain the merged and normalized entity set and relation set.
[0063] Specifically, it identifies entities with the same name or similar semantics within entity and relation sets. Semantic similarity for each entity and relation can be calculated using pre-trained language models (such as BERT and GPT) or word embeddings (such as Word2Vec and GloVe).
[0064] When the similarity exceeds a preset threshold, these entities are treated as the same entity and merged. For example, "person" and "human" can be considered synonymous entities and merged into a unified entity name.
[0065] Entity types are normalized by grouping synonymous or similar entity types into a unified standard type. For example, "location" and "place" can be uniformly grouped into the "Location" type. The final result is a merged and normalized set of entities. With relation set , respectively represented as:
[0066]
[0067]
[0068] S32: Calculate the contextual semantic representation of the merged and normalized entity set and relation set, and perform clustering based on semantic similarity;
[0069] Specifically, all entities and relationships are clustered by calculating semantic similarity. Clustering methods can employ similarity-based hierarchical clustering or density-based DBSCAN algorithms. The resulting clustered entity sets can effectively identify semantically similar entities or relationships. For example, if the entities "person" and "human" have high similarity in their contextual representations, they will be clustered together to form the same type of entity.
[0070] S33: Select the string with the highest frequency in the clustered entity set as the standard entity name;
[0071] S34: Based on obtaining the standard entity name, the entity types are hierarchically merged, and entity types with similar semantics or clear subordinate relationships are merged into the same upper-level category to form a hierarchical entity type system.
[0072] S35: Based on the merged and normalized entity set and relation set, and the hierarchical entity type system, construct entity relation triples, and assign corresponding confidence weights to the entity relation triples according to entity co-occurrence frequency, relation stability or external knowledge consistency, to obtain a weighted entity relation set.
[0073] Specifically, the entity set Hierarchical entity type system Relation set The triples are obtained by summing. :
[0074]
[0075] Assign confidence weights to each ternary component. The frequency of occurrence and the output probability are positively correlated with the consistency with external knowledge, ultimately resulting in a weighted entity relation set represented as follows:
[0076]
[0077] S4: Generate the first basic question-and-answer data based on the weighted entity relationship set; including the following sub-steps:
[0078] S41: Based on the entity relationship diagram, extract the entity attribute relationships and the relationship structure between entities, and construct a structured question-and-answer skeleton containing question elements, answer elements and relationship constraints. The structured question-and-answer skeleton is used to limit the question-and-answer logical structure and entity constraint conditions.
[0079] Specifically, extract entity attribute relationships: extract entity attributes (such as "size" and "product category") and the relationships between them from the entity relationship graph, and construct the attribute set of each entity and the relationships between entities.
[0080] Construct the question-and-answer framework: Define the question template (e.g., "What height can wear size L?") and answer type (e.g., "Height 175cm") based on entities and relationships. Add relationship constraints: Define the logic of the question and the entity relationships of the answer through the relationships between entities, ensuring that the question-and-answer questions have clear entity constraints.
[0081] Ultimately, a structured question-and-answer skeleton containing question elements, answer elements, and relational constraints is generated, standardizing the logical structure and entity conditions of the question and answer.
[0082] S42: Instantiate the structured question-and-answer skeleton using a predefined structured question template to generate basic questions that conform to the preset question-and-answer logic and language structure. The structured question template includes at least question forms for entity attributes, entity relationships, or combination relationships.
[0083] Specifically, based on the weighted entity database and relationship graph, controllable structured question-answering data is constructed to expand the basic coverage of the training set. A logically consistent set of basic questions is generated using structured question templates.
[0084]
[0085] in This indicates the problem text. This represents the answer text. The information represents the metadata, M represents the total number of basic sentence patterns, and j represents the current j-th basic question and answer statement.
[0086] S43: After completing the basic question generation, generate multiple question and answer samples for the same structured question and answer skeleton for different entity relationship types and their corresponding weighted relationship information, so as to cover question and answer scenarios of the same entity relationship in different expression methods or under different context conditions;
[0087] Specifically, multiple questions are generated for different entity relationship types:
[0088] For the same entity relationship, different question expressions are generated based on the diversity of the relationship and contextual information. For example, "Which courier can ship immediately?" can be changed to "What does the courier service for immediate shipping include?" or "What does the courier service for immediate shipping include?", ensuring coverage of multiple expression methods.
[0089] Use of weighted relation information: Adjust the way and content of generated questions according to the weight of the relationship between entities. Entities with higher relation weights have a greater impact on the generated questions, and question and answer samples related to these high-weight relationships are generated first.
[0090] Generation under different contextual conditions: Generate adaptive question-answering samples under different contextual conditions.
[0091] These methods generate multiple question-and-answer samples, covering all possible question-and-answer scenarios for the same entity relationship under different expressions and contextual conditions.
[0092] S44: Perform at least one of the following processing on the question-and-answer samples: semantic restatement, reasoning difficulty expansion, or expression style transformation, to finally obtain the first basic question-and-answer data.
[0093] Specifically, semantic paraphrasing, difficulty expansion, and style transformation are performed using a language model to obtain diverse question sentences. For each relation, several templates are defined, such as "Who is the founder of the {head} brand?", and these templates are sent to the LLM model to complete entity filling. Simultaneously, based on these style statements, short reasoning and multiple constraints are added, such as "{who} founded {head}", to obtain multiple primary question-answering data under the same logic. :
[0094]
[0095] in, The text represents the question, and M represents the total number of filler questions based on the basic sentence structure.
[0096] S5: Define a language logic template, apply consistency constraints to the first basic question-and-answer data and perform automatic filtering to obtain the second basic question-and-answer data, including the following sub-steps:
[0097] S51: Perform answer recoverability verification on the generated first basic question and answer data, determine whether the answers in the question and answer samples can be reverse located based on the corresponding entity relationship graph or entity attribute information, and retain only the question and answer samples whose answers can be uniquely or deterministically recovered from the relationships between entities;
[0098] Having already obtained the basic coverage capability for expanding the training set, we will use structured question templates to generate logically consistent basic questions:
[0099]
[0100] At this point, the first set of basic question-and-answer data is obtained based on this. It can then be expressed as:
[0101]
[0102] Where Q represents the first basic question and answer data, and A represents the answer text of the first basic question and answer data.
[0103] S52: Based on the answer recoverability verification, perform consistency verification; the consistency verification refers to using the current task model or large language model to perform self-questioning and self-answering reasoning on the question-and-answer sample, and judging whether the model can stably generate consistent or equivalent answers in multiple reasoning processes, so as to filter question-and-answer samples with unstable reasoning or large semantic ambiguity.
[0104] Specifically, answer recoverability represents the set of first-level basic question-and-answer data. The answer text A must match the text answer in the entity database. A difference threshold of less than 10% is used to filter question-and-answer samples with unstable reasoning or significant semantic ambiguity, as shown below:
[0105]
[0106] If the conditions are not met, unstable or semantically ambiguous question-and-answer samples will be filtered out.
[0107] S53: After completing the model consistency verification, generate multiple sets of candidate question and answer samples for the same entity relationship or the same question and answer logic using different prompting methods or different language expressions;
[0108] Specifically, model self-consistency means taking the current version of the task model F and asking and answering questions on its own. For the same entity relationship or the same question-and-answer logic, multiple sets of candidate question-and-answer samples are generated using different prompting methods or different language expressions, and all of them must ultimately satisfy the correct answer.
[0109] S54: Use a large language model to comprehensively score the multiple sets of candidate question and answer samples, and retain only the question and answer data with scores higher than a preset threshold as the second basic question and answer data under constraint filtering.
[0110] Specifically, the multi-perspective generative representation generates multiple question-answer pairs for the same triple using different prompts. Using an LLM (Local Level Model) as the scoring arbiter, only the second set of basic question-answer data with scores above a threshold is retained, ultimately yielding the second set of basic question-answer data. It can be represented as:
[0111]
[0112] in This indicates that the second set of basic question-and-answer data with scores above a threshold has been retained. This indicates the text of the answer.
[0113] S6: Enhance the second basic question-and-answer data using external information sources to generate enhanced training data supported by external knowledge, including the following sub-steps:
[0114] S61: Based on the entities and entity relationships in the second basic question and answer data, construct a retrieval query for external retrieval;
[0115] Specifically, regarding the high-quality basic data obtained Access external data sources through browser search tools to construct search queries that include entities, relationships, or questions;
[0116] S62: Utilize external information sources to obtain the document content or webpage content corresponding to the search query;
[0117] Specifically, the content of documents or web pages corresponding to the query is obtained through external information sources. In this case, Python can be used to directly crawl open-source e-commerce Q&A web pages to obtain query records, thus obtaining a set of external basic Q&A data. :
[0118]
[0119] in This refers to the question text obtained from an external information source. This represents the corresponding answer text. The expression represents metadata, where N represents the total number of basic sentence patterns, and i represents the current i-th basic question-and-answer statement.
[0120] S63: Perform a relevance analysis on the content of the document or webpage to screen out retrieval evidence that is highly relevant to the entity or entity relationship;
[0121] Specifically, the open-source e-commerce Q&A webpage data crawled by Python is filtered to obtain retrieval evidence related to the entity or relationship.
[0122] S64: Combine the retrieval enhancement generation mechanism to fuse the retrieval evidence with the original question-and-answer data to generate enhanced training data that includes external knowledge support.
[0123] Specifically, based on the retrieved evidence, the generated entity data is enhanced using a retrieval enhancement generation mechanism; the retrieved evidence is then introduced into the generation process via a browser access method to generate enhanced training data containing external knowledge. :
[0124]
[0125] By generating augmented training data that incorporates external knowledge, the knowledge density and timeliness of the training data can be improved.
[0126] S7 utilizes reinforcement learning to take the data generation strategy as a decision action, and uses the reinforcement training data as input to generate a reinforcement learning decision process, including the following sub-steps:
[0127] S71: Model the data generation process as a Markov decision process to describe the data generation state, generation action and its state transition relationship;
[0128] Specifically, reinforcement learning problem modeling is based on augmented training data that incorporates external knowledge. The large model data generation strategy is modeled as a Markov decision process (MDP), which includes a state space, an action space, and state transition relationships.
[0129]
[0130] in Represents the state space (data + model state). Represents the action space (data generation strategy). Represents the state transition probability. Represents the reward function, ∈(0,1] represents the discount factor. Each round of data generation, lightweight training, and validation is considered as one time step. .
[0131] S72: Define the current training data distribution features, entity or relation coverage, and model performance metrics on the validation set as the state space of reinforcement learning, and generate state vectors;
[0132] Specifically, the state space is defined as the current data generation state, which is the state of reinforcement learning. The state includes at least one or more of the following information: the distribution characteristics of the current training data, the coverage of entities or relationships, the quality statistics of the generated data, and the performance indicators of the model on the validation set.
[0133] The implementable definitions of each substate are as follows:
[0134] Augmented training data containing external knowledge In time step t, it is represented as Its data distribution characteristics are as follows:
[0135]
[0136] Avglen represents the average sample length, TR represents the proportion of different data types (BasicQA / WebQA / HardQA), and HR represents the proportion of difficult samples.
[0137] Entity / relationship coverage is represented as:
[0138]
[0139] in This represents the set of entities that appear in the current dataset. Represents the set of relations that appear.
[0140] Data quality statistics are expressed as follows:
[0141]
[0142] Where CR represents the consistency constraint pass rate, and NR represents the proportion of filtered samples.
[0143] The model performance index is expressed as :
[0144]
[0145] Where Acc represents the accuracy rate. represents the score, and Loss represents the loss.
[0146] Therefore, in time step t, the state vector is defined as:
[0147]
[0148] in Represented as augmented training data incorporating external knowledge Data distribution characteristics, This is represented as entity / relationship coverage. This is represented as data quality statistics. This is represented as a model performance metric.
[0149] S73: Define the data generation strategy as a decision action in reinforcement learning, wherein the decision action includes at least the adjustment of the data type, generation difficulty, or generation ratio.
[0150] Specifically, the action space is defined as follows: the data generation strategy used to control the data generation process is defined as the decision action of reinforcement learning. The decision action includes at least one or more of the following: selection of the data type to be generated, setting of the difficulty of the generated data, setting of the number of generated samples, adjustment of the generation parameters, and the calling method of the generation module.
[0151] First, generate data types. Represented as:
[0152]
[0153] in These represent basic question-and-answer, webpage question-and-answer, and difficult question-and-answer data types, respectively.
[0154] Generation difficulty Represented as:
[0155]
[0156] These are respectively represented as easy, medium, and difficult.
[0157] Sample size Represented as:
[0158]
[0159] Ultimately, the action principle is defined as the strategy for choosing "how to generate data," and the decision action can be represented as... :
[0160]
[0161] The decision-making action is dynamically adjusted by the generation of data types, generation difficulty, and the generation ratio of sample size.
[0162] S8: Based on the reinforcement learning decision-making process, perform multiple rounds of reinforcement learning to generate difficult sample data, and dynamically adjust the data generation strategy according to the performance feedback of the intelligent question answering system, thereby automatically generating training data that meets the task requirements.
[0163] Figure 2 This is a modeling diagram of a reinforcement learning Markov decision process according to an exemplary embodiment, with reference to... Figure 2 The present invention provides a reinforcement learning Markov decision process, which includes the following iterative steps:
[0164] S81: Select the corresponding data generation method based on the reinforcement learning decision-making process generated from the current data;
[0165] Specifically, based on the current state of reinforcement learning, an appropriate generation strategy is selected. This strategy may include different question-answering generation methods, such as rule-based generation, sample-based generation, or adversarial generation. Depending on the model's current performance and task requirements, a decision is made on whether to generate simple samples (for basic training) or difficult samples (to improve the model's robustness).
[0166] S82: Generate candidate training data according to the data generation method, and perform quality screening on them to obtain new training data;
[0167] Specifically, candidate training data is generated based on the selected data generation method. For example, rule-based generation methods may generate questions with fixed formats, while adversarial generation methods may generate challenging questions. These candidate data are then subjected to quality screening, eliminating low-quality samples that do not meet preset criteria. Quality screening criteria may include semantic accuracy, logical consistency, and relevance to task requirements.
[0168] S83: Update the task model using the newly added training data;
[0169] Specifically, new training data is input into the task model to update and optimize it. This process can be carried out using standard supervised learning methods, where the model's loss function is adjusted based on the new training data, with the goal of improving the model's performance in real-world tasks.
[0170] S84: Generate reward signals based on the updated evaluation results;
[0171] Specifically, the reward signal is constructed based on the changes in model performance after the data generation strategy is executed. The reward signal is used to characterize the contribution of the current data generation strategy to the model training effect.
[0172] The performance gain bonus is represented as:
[0173]
[0174] in Let t represent the score at stage t, and Mertic be the gain reward function.
[0175] Data coverage rewards are represented as follows:
[0176]
[0177] in This represents the set of entities that appear in the current dataset. Represents the set of relations that appear. Set it to 0.5.
[0178] Quality penalty items are represented as follows:
[0179]
[0180] For high-quality datasets In time step t, it is represented as , Set to 0.5, NosieRate is the noise penalty function.
[0181] The comprehensive reward function is expressed as:
[0182]
[0183] in The table lists hyperparameters, which you can set according to your dataset.
[0184] S85: Update the data generation strategy according to the reward signal, and repeat S81-S84 until the strategy converges or the preset termination condition is met.
[0185] Specifically, the strategy is iteratively updated by using reinforcement learning algorithms to select and update the data generation strategy based on the current state and the reward signal.
[0186] Strategy defined as Define, maximize the objective function Represented as:
[0187]
[0188] Where T represents the total time steps.
[0189] Combining the above decision-making actions :
[0190]
[0191] Gradient update express:
[0192]
[0193] The dominance function Represented as:
[0194]
[0195] in It is represented as a state-value function, which can be any state-value function under reinforcement learning.
[0196] Ultimately, the optimal or near-optimal data generation strategy for the current state is obtained through training. .
[0197] S86: Automatically generates training data that meets the task requirements after the preset conditions are met;
[0198] Specifically, the generative strategy obtained through reinforcement learning can be... The output model can be directly used to generate training data that meets the requirements of the task.
[0199] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the data generation method based on small-sample seeds and multi-round reinforcement as described above. Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, provided by an embodiment of the present invention, for a data generation method based on small sample seeds and multi-round reinforcement. Except for... Figure 3 In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0200] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the data generation method based on small-sample seeds and multi-round reinforcement as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of a wind turbine, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.
[0201] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0202] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A data generation method based on small-sample seeds and multi-round reinforcement, characterized in that, include: Obtain labeled seed data related to the training task of the intelligent question answering system, and preprocess the labeled seed data to obtain a standardized seed dataset; The standardized seed dataset is processed based on the entity extraction model to construct an initial entity set and relation set; The entity set and relation set are normalized and weighted, and an entity-relationship graph is constructed. First basic question-and-answer data is generated based on the entity relationship graph; Define a language logic template, apply consistency constraints to the first basic question and answer data and perform automatic filtering to obtain the second basic question and answer data; The second basic question-and-answer data is enhanced by using external information sources to generate enhanced training data that includes external knowledge support; Specifically, this includes: constructing a retrieval query for external retrieval based on the entities and entity relationships in the second basic question-and-answer data; acquiring document content or webpage content corresponding to the retrieval query using external information sources; performing relevance analysis on the document content or webpage content to filter out retrieval evidence highly related to the entities or entity relationships; and combining the retrieval evidence with the original question-and-answer data using a retrieval enhancement generation mechanism to generate enhanced training data containing external knowledge support. Using reinforcement learning, a data generation strategy is used as a decision action, and the reinforced training data is used as input to generate a reinforcement learning decision process. Specifically, this includes: modeling the data generation process as a Markov decision process to describe the data generation state, generation action, and their state transition relationships; defining the current training data distribution characteristics, entity or relation coverage, and model performance metrics on the validation set as the state space of reinforcement learning, and generating state vectors; defining the data generation strategy as the decision action of reinforcement learning, wherein the decision action includes at least adjustments to the data type, generation difficulty, or generation ratio. According to the reinforcement learning decision-making process, multiple rounds of reinforcement learning are performed to generate preliminary training sample data, and the data generation strategy is dynamically adjusted based on the performance feedback of the intelligent question answering system, thereby automatically generating training sample data that meets the task requirements.
2. The data generation method based on small sample seeds and multi-round reinforcement as described in claim 1, characterized in that, Obtain labeled seed data related to the training task of the intelligent question answering system, and preprocess the labeled seed data to obtain a standardized seed dataset, including: The acquired labeled seed data is processed to standardize the data format, thereby converting it into a preset data structure. The data structure includes at least the input text, labeling results, and sample identification information. After the data format is standardized, the seed data is subjected to field integrity verification, including field missing detection and outlier detection, and samples with missing input text, missing annotation results or annotation results that do not meet the preset format constraints are removed. Based on the field integrity verification, text cleaning is performed on the retained input text to obtain cleaned text with clear semantics and standardized format. While obtaining the cleaned text, for the various annotation systems that may exist in the labeled seed data, the entity type labels, relation type labels or answer category labels are uniformly mapped to generate standardized annotation results that are consistent with the preset label set. After completing the label unification and mapping, sample deduplication and conflict resolution are performed on the labeled seed data; After deduplication and conflict resolution, the labeled seed data is divided into training subset and validation subset. Statistical modeling is then performed on the labeled seed data to calculate entity coverage, relation coverage, sample length distribution, and label distribution information, ultimately obtaining the preprocessed standardized seed dataset.
3. The data generation method based on small sample seeds and multi-round reinforcement as described in claim 1, characterized in that, The standardized seed dataset is processed based on an entity extraction model to construct an initial entity set and relation set, including: Based on the entity extraction model, entities, entity types, and relationships between entities are extracted from the standardized seed dataset. The extracted entities, entity types, and relationships between entities are aggregated to form a standardized entity library; An initial set of entities and a set of relationships are constructed based on the entities, entity types, and relationships between entities.
4. The data generation method based on small sample seeds and multi-round reinforcement as described in claim 1, characterized in that, The entity set and relation set are normalized and weighted, and an entity-relationship graph is constructed, including: Entities with the same name or similar semantics in the entity set and relation set are merged, and the entity types are normalized to obtain the merged and normalized entity set and relation set. Calculate the contextual semantic representation of the merged and normalized entity set and relation set, and perform clustering based on semantic similarity; Select the most frequently occurring string from the clustered entity set as the standard entity name; Based on obtaining standard entity names, entity types are hierarchically merged, merging entity types with similar semantics or clear subordinate relationships into the same upper-level category to form a hierarchical entity type system. Based on the merged and normalized entity set and relation set, and the hierarchical entity type system, entity relation triples are constructed. Then, according to the entity co-occurrence frequency, relation stability, or external knowledge consistency, corresponding confidence weights are assigned to the entity relation triples to obtain the entity relation graph.
5. The data generation method based on small sample seeds and multi-round reinforcement as described in claim 1, characterized in that, The generation of the first basic question-and-answer data based on the entity relationship graph includes the following sub-steps: Based on the entity relationship graph, entity attribute relationships and inter-entity relationship structure are extracted to construct a structured question-and-answer skeleton containing question elements, answer elements, and relationship constraints. The structured question-and-answer skeleton is used to limit the question-and-answer logical structure and entity constraint conditions. The structured question-and-answer skeleton is instantiated using a predefined structured question template to generate basic questions that conform to the preset question-and-answer logic and language structure. The structured question template at least includes question forms targeting entity attributes, entity relationships, or combination relationships. After generating the basic question, multiple question-and-answer samples are generated for the same structured question-and-answer skeleton for different entity relationship types and their corresponding weighted relationship information, so as to cover question-and-answer scenarios of the same entity relationship in different expressions or under different context conditions. Perform at least one of the following processing steps on the question-and-answer samples: semantic restatement, reasoning difficulty expansion, or expression style transformation, to finally obtain the first basic question-and-answer data.
6. The data generation method based on small sample seeds and multi-round reinforcement as described in claim 1, characterized in that, The first basic question-and-answer data is subjected to consistency constraints and automatic filtering to obtain the second basic question-and-answer data, including: Perform answer recoverability verification on the generated first basic question and answer data to determine whether the answers in the question and answer samples can be reverse located based on the corresponding entity relationship graph or entity attribute information, and only retain the question and answer samples whose answers can be uniquely or deterministically recovered from the relationships between entities; Based on the answer recoverability verification, a consistency verification is performed; the consistency verification refers to using the current task model or large language model to perform self-questioning and self-answering reasoning on the question-and-answer sample, and judging whether the model can stably generate consistent or equivalent answers in multiple reasoning processes, so as to filter out question-and-answer samples with unstable reasoning or large semantic ambiguity. After completing the model consistency verification, multiple sets of candidate question-and-answer samples are generated for the same entity relationship or the same question-and-answer logic, using different prompting methods or different language expressions. The large language model is used to comprehensively score the multiple sets of candidate question-and-answer samples, and only the question-and-answer data with scores higher than a preset threshold are retained as the second basic question-and-answer data under the constraint filtering.
7. The data generation method based on small sample seeds and multi-round reinforcement as described in claim 1, characterized in that, According to the reinforcement learning decision-making process, multiple rounds of reinforcement learning are executed to generate initial training sample data, and the data generation strategy is dynamically adjusted based on the performance feedback of the intelligent question answering system, thereby automatically generating training sample data that meets the task requirements, including: S81: Select the corresponding data generation method based on the reinforcement learning decision-making process generated from the current data; S82: Generate candidate training data according to the data generation method, and perform quality screening on them to obtain new training data; S83: Update the task model using the newly added training data; S84: Generate reward signals based on the updated evaluation results; S85: Update the data generation strategy according to the reward signal, and repeat S81-S84 until the strategy converges or the preset termination condition is met. S86: Once the preset conditions are met, training data that meets the task requirements will be automatically generated.
8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.
9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.