Data set construction method for collaborative iteration of antagonism enhancement and semantic consistency verification

Abstract

The invention relates to the technical field of artificial intelligence model training, in particular to an adversarial enhancement and semantic consistency verification collaborative iteration data set construction method, which comprises the following steps: collecting original data, and performing data enhancement on data to be enhanced according to an enhancement rule to obtain enhanced data; and adding the enhanced data which completely accords with the semantics to a data set, abandoning the enhanced data which completely does not accord with the semantics, adding a feedback label to part of the enhanced data which accords with the semantics, and then re-enhancing. In order to solve the problem of low sample utilization rate in the data enhancement process in the prior art, a feedback link from self-consistency verification to data enhancement is introduced in the scheme, the consistency result is verified, part of enhanced data which is not completely accordant is returned after being labeled, and the enhanced data is enhanced again, so that the number of finally output samples is increased, and the data enhancement efficiency is improved. And the utilization rate of the sample is further improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model training technology, specifically to a dataset construction method that coordinates adversarial enhancement and semantic consistency verification in an iterative manner. Background Technology

[0002] In existing technologies, methods for constructing large language model supervised fine-tuning (SFT) datasets mainly fall into two categories: manual annotation and automated augmentation. Automated augmentation utilizes data augmentation algorithms or pre-trained models to rewrite, expand, or perturb the original samples, thereby generating more candidate training samples. Its structure includes an augmentation module, a verification module, and a selection module. The connection relationship between these components is typically as follows: the original sample is input into the augmentation module to generate augmented samples → the augmented samples enter the verification module for semantic or format checks → the selection module decides whether to include the sample based on rules or thresholds. The process is usually executed sequentially, lacking an interactive feedback mechanism between the augmentation and verification modules; the verification result is only used to determine whether a sample passes the test, without adjusting the augmentation strategy.

[0003] Existing technologies already include technical solutions for data enhancement and consistency verification.

[0004] For example, patent application CN202510555392.6 discloses a data augmentation method and system based on Chinese characteristics and semantic consistency. The method includes: acquiring raw data; performing semantically consistent synonym replacement and random insertion on the raw data; and noise augmentation based on Chinese characteristics, including word-level noise augmentation and lexical-level noise augmentation; generating and outputting the augmented data. This method not only deeply considers the inherent characteristics of the Chinese language but also emphasizes ensuring the semantic consistency of the data, aiming to promote efficient learning of global semantic coherence by the model during the training phase. These algorithms are used to process data from the real world across vocabulary and sentence dimensions, more effectively augmenting and balancing existing datasets, enhancing the practicality of the data in the model training process, and thus further promoting the optimization and improvement of model performance.

[0005] However, in actual implementation, the inventors found that since this type of technical solution usually performs a linear processing flow for enhancing data and semantic consistency, and then discards data that does not meet the requirements, this linear processing flow is prone to discarding a large number of samples, thereby reducing the sample utilization rate and affecting the data set expansion process. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, a dataset construction method that combines adversarial enhancement and semantic consistency verification through collaborative iteration is provided.

[0007] The specific technical solution is as follows:

[0008] A dataset construction method that coordinates adversarial enhancement and semantic consistency verification in an iterative manner includes:

[0009] Step S1: Collect raw data and use the raw data as the data to be enhanced;

[0010] Step S2: Perform data augmentation on the data to be augmented according to the augmentation rules to obtain augmented data;

[0011] Step S3: Perform data validation on the augmented data and label the validation results. Add the augmented data that fully conforms to the semantics to the dataset, and discard the augmented data that does not conform to the semantics at all.

[0012] Step S4: Determine if the maximum number of iterations has been reached;

[0013] If so, proceed to step S6;

[0014] If not, proceed to step S5;

[0015] Step S5: Add feedback annotations to the partially semantically correct augmented data, and then return it to step S2 as new augmented data;

[0016] Step S6: Output the dataset.

[0017] On the other hand, step S1 includes:

[0018] Step S11: Obtain the raw data from multiple sources;

[0019] Step S12: Perform deduplication processing on the original data to obtain deduplicated data;

[0020] Step S13: Format the deduplicated data to obtain formatted data;

[0021] Step S14: Clean the formatted data to remove invalid data and obtain the data to be enhanced.

[0022] On the other hand, step S2 includes:

[0023] The data to be enhanced is obtained by extracting several enhancement strategies and performing enhancement.

[0024] data;

[0025] The enhancement strategies include synonym replacement, sentence structure perturbation, misspelling perturbation, and long...

[0026] Contextual perturbation.

[0027] On the other hand, in step S2, the enhancement called on the enhanced data annotation

[0028] Strategy;

[0029] Step S5 includes:

[0030] Step S51: Identify the partially semantically consistent augmented data to extract semantic features, and match the semantic features with the augmentation rules to determine the non-matching data.

[0031] Enhance the rule as a failure mode;

[0032] Step S52: Determine the enhancement strategy label as the feedback label according to the failure mode;

[0033] When step S2 is performed, the selected enhancement strategy is adjusted according to the feedback annotation.

[0035] On the other hand, in step S3, data verification is performed by calculating a consistency score;

[0036] The process of generating the consistency score includes:

[0037] Sscore=α·s_emb+β·s_nli+γ·s_rule+δ·s_llm-λ·p;

[0038] In the formula, Sscore is the consistency score;

[0039] s_emb is the original data and the corresponding augmented data in the word embedding space.

[0040] Cosine similarity; s_nli is the semantic consistency between the original data and the corresponding augmented data, evaluated by a natural language inference model.

[0041] s_rule is the rule validation score for the enhanced data;

[0042] s_llm is the arbitration score based on a large language model;

[0043] p is the penalty term corresponding to the erroneous sample;

[0044] α, β, γ, δ, and λ are weight coefficients.

[0045] On the other hand, step S3 includes:

[0046] Step S31: Obtain the original data corresponding to the augmented data, and calculate the cosine similarity using the BERT embedding model;

[0047] Step S32: Use the NLI model to determine whether the original data and the corresponding augmented data imply a relationship.

[0048] Step S33: Verify whether the entities, table names, and aggregate functions in the enhanced data and the original data are consistent to obtain the rule verification score;

[0049] Step S34: Arbitrate the discrepancies between the augmented data and the original data using a large language model to obtain the arbitration score;

[0050] Step S35: Calculate the consistency score, and then classify the augmented data using a score threshold.

[0051] On the other hand, in step S3, when the consistency score is greater than or equal to 0.8, it indicates that the enhanced data fully conforms to the semantics;

[0052] When the consistency score is greater than or equal to 0.5 and less than 0.8, it indicates that the enhanced data part conforms to semantics;

[0053] When the consistency score is less than 0.5, it indicates that the augmented data does not conform to semantics at all.

[0054] On the other hand, the feedback annotations added in step S5 include: adjusting the disturbance intensity, changing the enhancement strategy, and performing multiple enhancements.

[0055] On the other hand, after performing step S6, the following is also included:

[0056] Step S7: Train the artificial intelligence model using the dataset, determine the sample difficulty based on the training process data, and adjust the strategy for generating the augmented data next time based on the sample difficulty.

[0057] The above technical solution has the following advantages or beneficial effects:

[0058] To address the issue of low sample utilization in existing data augmentation processes, this solution introduces a feedback loop from consistency verification to data augmentation. By verifying the consistency results and adding annotations to partially inconsistent augmented data before returning it for further augmentation, the number of samples in the final output is increased, thereby improving the utilization rate of the samples. Attached Figure Description

[0059] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0060] Figure 1 This is an overall schematic diagram of an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of step S1 in an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of step S3 in an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of step S5 in an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of step S7 in an embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0067] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0068] This invention includes:

[0069] A dataset construction method that coordinates adversarial enhancement and semantic consistency verification in an iterative manner, such as... Figure 1 As shown, it includes:

[0070] Step S1: Collect raw data and use it as the data to be enhanced;

[0071] Step S2: Perform data augmentation on the data to be augmented according to the augmentation rules to obtain augmented data;

[0072] Step S3: Perform data validation on the augmented data and label the validation results. Add augmented data that fully conforms to the semantics to the dataset, and discard augmented data that does not conform to the semantics at all.

[0073] Step S4: Determine if the maximum number of iterations has been reached;

[0074] If so, proceed to step S6;

[0075] If not, proceed to step S5;

[0076] Step S5: Add feedback annotations to some of the semantically correct augmented data, and then return it to step S2 as new data to be augmented;

[0077] Step S6: Output the dataset.

[0078] Specifically, to address the issue of low sample utilization in the data augmentation process of existing technologies, this solution introduces a feedback loop from consistency verification to data augmentation. By verifying the consistency results and adding annotations to some partially inconsistent augmented data before returning them for further augmentation, the number of samples in the final output is increased, thereby improving the utilization rate of the samples.

[0079] In one embodiment, such as Figure 2 As shown, step S1 includes:

[0080] Step S11: Obtain raw data from multiple sources;

[0081] Step S12: Perform deduplication on the original data to obtain deduplicated data;

[0082] Step S13: Format the deduplicated data to obtain formatted data;

[0083] Step S14: Clean the formatted data to remove invalid data and obtain the data to be enhanced.

[0084] Specifically, to achieve a better process for collecting raw data, this embodiment includes the aforementioned preprocessing steps. The preprocessing steps mainly include deduplication, formatting, and data cleaning.

[0085] The deduplication process involves matching rows in the collected raw data to extract and remove duplicate rows, thereby reducing redundant data. Next, the deduplicated data is formatted, including adjusting the data format and assigning zeros to blank values. Finally, the formatted data is matched using a dictionary to identify and remove or correct erroneous data, ensuring that the input data meets the basic requirements for subsequent processing.

[0086] In implementation, the above process typically includes: obtaining an initial sample set from publicly available database question-and-answer corpora (such as the Spider dataset) and project-built business question-and-answer pairs. The initial samples contain SQL queries and corresponding natural language questions; during sample collection, it is ensured that common SQL operations (such as SELECT, JOIN, GROUP BY) and industry-specific long-tail questions are covered. ETL scripts are used to clean the raw data: removing non-standard SQL statements, completing missing fields, standardizing text encoding formats, and ensuring data validity.

[0087] In one embodiment, step S2 includes:

[0088] Several augmentation strategies are used to augment the data to be augmented, resulting in augmented data.

[0089] The enhancement strategies include synonym substitution, sentence structure perturbation, misspelling perturbation, and long context perturbation.

[0090] Specifically, to achieve better enhancement results, this embodiment configures multiple enhancement strategies for the data to be enhanced, including:

[0091] Synonym replacement: Search for synonyms for words such as "search" and "query" in the dictionary based on their semantic features and replace them to maintain semantic consistency;

[0092] Structural perturbation: Adjusting the order of conditions or JOIN operations in SQL statements to generate equivalent statements;

[0093] Typo perturbation: Introducing common spelling errors based on dictionaries in natural language problems;

[0094] Long contextual perturbation: Add distracting descriptions before the question, such as "someone raised this at a business analysis meeting".

[0095] The aforementioned enhancement strategies are extracted and applied to the data to be enhanced according to pre-configured enhancement rules.

[0096] In one embodiment, in step S3, data verification is performed by calculating a consistency score.

[0097] The process of generating the consistency score includes:

[0098] Sscore=α·s_emb+β·s_nli+γ·s_rule+δ·s_llm-λ·p;

[0099] In the formula, Sscore is the consistency score;

[0100] s_emb represents the cosine similarity between the original data and the corresponding augmented data in the word embedding space;

[0101] s_nli represents the semantic consistency between the original data and the corresponding augmented data, as evaluated using a natural language inference model.

[0102] s_rule is the rule validation score for augmented data;

[0103] s_llm is the arbitration score based on a large language model;

[0104] p is the penalty term corresponding to the erroneous sample;

[0105] α, β, γ, δ, and λ are weight coefficients.

[0106] Specifically, in order to achieve a better verification effect on the augmented data, in this embodiment, the above four sets of models are used for verification respectively to obtain the corresponding verification scores and perform weighted calculations, thereby achieving a better evaluation effect on the augmented data. At the same time, by adding penalty items to deduct the corresponding scores, the generated erroneous samples are directly eliminated.

[0107] In one embodiment, such as Figure 3 As shown, step S3 includes:

[0108] Step S31: Obtain the original data corresponding to the augmented data, and calculate the cosine similarity using the BERT embedding model;

[0109] Step S32: Use the NLI model to determine whether the original data and the corresponding augmented data imply a relationship.

[0110] Step S33: Verify the consistency of entities, table names, and aggregate functions in the augmented data and the original data to obtain a rule verification score;

[0111] Step S34: Arbitrate the discrepancies between the augmented data and the original data using a large language model to obtain an arbitration score;

[0112] Step S35: Calculate the consistency score, and then classify the augmented data using the score threshold.

[0113] Specifically, the evaluation process described above is based on a pre-trained artificial intelligence model.

[0114] For cosine similarity, the enhanced data and the corresponding original data are first segmented into words. Then, the segmentation results are converted into word embedding space to form word vectors. The angle between the word vectors of the two is calculated to obtain the cosine similarity.

[0115] To address semantic consistency, a pre-trained natural language inference model is introduced here. This model extracts semantics from the augmented data and the corresponding original data, and then compares them to determine the degree of semantic consistency between the original samples and the augmented samples.

[0116] Furthermore, for parameters, entity content, and called functions that should not be modified during the enhancement process, the specific format content in the original data is first extracted to obtain the corresponding instance content, including entities, table names, and aggregate functions in SQL. Then, the enhanced data is searched to determine whether there is consistent content, thereby obtaining the rule verification score.

[0117] Finally, to determine the reasonableness of the samples, a large-scale language model is used to analyze the semantics of the augmented data and the corresponding original data to obtain an arbitration score.

[0118] The scores from the multiple sources mentioned above are fused by configuring specific weight values. In a typical embodiment, the weight values ​​are:

[0119] α=0.25, β=0.25, γ=0.25, δ=0.25, λ=1, p∈{0,0.1,0.2,0.5}.

[0120] In step S3, when the consistency score is greater than or equal to 0.8, it indicates that the augmented data fully conforms to the semantics.

[0121] When the consistency score is greater than or equal to 0.5 and less than 0.8, it indicates that the augmented data part conforms to semantics;

[0122] When the consistency score is less than 0.5, it indicates that the augmented data does not conform to the semantics at all and should be discarded.

[0123] In one embodiment, step S3 involves invoking an enhancement strategy for the enhanced data annotation.

[0124] like Figure 4 As shown, step S5 includes:

[0125] Step S51: Identify the partially semantically consistent augmented data to extract semantic features, and match the semantic features with the augmentation rules to determine the non-matching augmentation rules as failure modes;

[0126] Step S52: Determine the enhancement strategy annotations as feedback annotations based on the failure mode;

[0127] When step S2 is executed, the selected enhancement strategy is adjusted according to the feedback annotations.

[0128] Specifically, to achieve a better correction effect on data that does not fully conform to semantics, this embodiment first identifies the partially semantically consistent augmented data selected through consistency scoring to extract semantic features. Then, the semantic features are matched with several pre-matched augmentation rules. These augmentation rules mainly match the constraints added to each artificial intelligence model during the consistency generation process, such as numerical consistency, semantic similarity, and logical accuracy. This can also be achieved by viewing the sub-scores output by each model and labeling the corresponding failure modes. Based on this, the required augmentation strategy is adjusted according to the failure modes, such as removing some augmentation strategies or re-executing augmentation strategies. The augmentation strategy labels are determined as feedback labels, and then the augmented data is regenerated. During the generation process, the selected augmentation strategy is adjusted according to the feedback labels, and additional processing steps are added.

[0129] In one embodiment, the added feedback annotations in step S5 include: adjusting the perturbation intensity, changing the enhancement strategy, and performing multiple enhancements.

[0130] Specifically, in order to correct data that does not fully conform to semantics, the above feedback annotations are also added in this embodiment.

[0131] Adjusting the perturbation intensity includes changing the number of structural perturbations, synonym substitutions, and added contexts involved in the enhancement strategy, reducing the number of changes to lessen the impact on semantics.

[0132] Changing the enhancement strategy includes removing synonym replacements or other strategies to avoid problems such as tampering with key values.

[0133] Furthermore, multiple enhancements can increase the variability between samples, which is useful when generating large datasets.

[0134] In one embodiment, such as Figure 5 As shown, after executing step S6, the following is also included:

[0135] Step S7: Train the artificial intelligence model using the dataset, determine the sample difficulty based on the training process data, and adjust the strategy for generating augmented data for the next time according to the sample difficulty.

[0136] Specifically, to control the difficulty of samples in the dataset, this embodiment also establishes a feedback loop from the self-training process to the dataset training process. Specifically, once the dataset is completed, it is written to a MongoDB repository and tagged with metadata (enhancement method, validation score, task type, version number).

[0137] Subsequently, the dataset is imported into a training platform (such as a training framework based on PyTorch Lightning) to fine-tune the model. During the training process, the number of epochs required for the model to converge is measured and compared with the typical number of training epochs to determine the difficulty of the dataset.

[0138] Subsequently, the difficulty of the dataset was changed by adding corresponding annotation parameters during the data augmentation process, such as increasing the perturbation intensity, using multiple augmentations, and enabling multiple augmentation strategies.

[0139] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A dataset construction method that coordinates adversarial enhancement and semantic consistency verification in an iterative manner, characterized in that, include: Step S1: Collect raw data and use the raw data as the data to be enhanced; Step S2: Perform data augmentation on the data to be augmented according to the augmentation rules to obtain augmented data; Step S3: Perform data verification on the augmented data and label the verification results. Add the augmented data that fully conforms to the semantics to the dataset, and discard the augmented data that does not conform to the semantics at all. Step S4: Determine if the maximum number of iterations has been reached; If so, proceed to step S6; If not, proceed to step S5; Step S5: Add feedback annotations to the partially semantically correct augmented data, and then return it to step S2 as new augmented data; Step S6: Output the dataset.

2. The dataset construction method according to claim 1, characterized in that, Step S1 includes: Step S11: Obtain the raw data from multiple sources; Step S12: Perform deduplication processing on the original data to obtain deduplicated data; Step S13: Format the deduplicated data to obtain formatted data; Step S14: Clean the formatted data to remove invalid data and obtain the data to be enhanced.

3. The dataset construction method according to claim 1, characterized in that, Step S2 includes: The enhanced data is obtained by extracting several of the enhancement strategies from the data to be enhanced. The enhancement strategies include synonym replacement, sentence structure perturbation, misspelling perturbation, and long context perturbation.

4. The dataset construction method according to claim 3, characterized in that, In step S2, the enhancement strategy is invoked for the enhanced data annotation; Step S5 includes: Step S51: Identify the partially semantically consistent augmented data to extract semantic features, and match the semantic features with the augmentation rules to determine the non-matching augmentation rules as failure modes; Step S52: Determine the enhancement strategy label as the feedback label according to the failure mode; When step S2 is performed, the selected enhancement strategy is adjusted according to the feedback annotation.

5. The dataset construction method according to claim 1, characterized in that, In step S3, data verification is performed by calculating a consistency score. The process of generating the consistency score includes: Sscore=α·s_emb+β·s_nli+γ·s_rule+δ·s_llm-λ·p; In the formula, Sscore is the consistency score; s_emb is the cosine similarity between the original data and the corresponding augmented data in the word embedding space; s_nli represents the semantic consistency between the original data and the corresponding augmented data, as evaluated by a natural language inference model. s_rule is the rule validation score for the enhanced data; s_llm is the arbitration score based on a large language model; p is the penalty term corresponding to the erroneous sample; α, β, γ, δ, and λ are weight coefficients.

6. The dataset construction method according to claim 5, characterized in that, Step S3 includes: Step S31: Obtain the original data corresponding to the augmented data, and calculate the cosine similarity using the BERT embedding model; Step S32: Use the NLI model to determine whether the original data and the corresponding augmented data imply a relationship. Step S33: Verify whether the entities, table names, and aggregate functions in the enhanced data and the original data are consistent to obtain the rule verification score; Step S34: Arbitrate the discrepancies between the augmented data and the original data using a large language model to obtain the arbitration score; Step S35: Calculate the consistency score, and then classify the augmented data using a score threshold.

7. The dataset construction method according to claim 5, characterized in that, In step S3, when the consistency score is greater than or equal to 0.8, it indicates that the enhanced data fully conforms to the semantics. When the consistency score is greater than or equal to 0.5 and less than 0.8, it indicates that the enhanced data part conforms to semantics; When the consistency score is less than 0.5, it indicates that the augmented data does not conform to semantics at all.

8. The dataset construction method according to claim 3, characterized in that, In step S5, the added feedback annotations include: adjusting the disturbance intensity, changing the enhancement strategy, and performing multiple enhancements.

9. The dataset construction method according to claim 1, characterized in that, After performing step S6, the method further includes: Step S7: Train the artificial intelligence model using the dataset, determine the sample difficulty based on the training process data, and adjust the strategy for generating the augmented data next time based on the sample difficulty.

Images