Multi-round cross validation and weak supervision noise cleaning method based on large language model

Abstract

The invention discloses a multi-round cross validation and weak supervision noise cleaning method based on a large language model. According to the method, weak supervised learning and large language model reasoning capability are fused, and multi-round cleaning and label optimization are performed on a low-confidence sample by introducing a small amount of high-quality label seeds and combining automatic rule construction and cross validation processes. According to the method, a verification and feedback system with a large language model as a core is constructed, efficient purification and enhancement of weak tags in large-scale text data are achieved, and high-quality training data and intelligent tag optimization support are provided for natural language processing tasks such as relation extraction, text classification and entity recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and natural language processing, and particularly relates to a multi-round cross-validation and weakly supervised noise cleaning method based on a large language model. BACKGROUND

[0002] In the field of machine learning and natural language processing, the labeling quality of training data has a decisive influence on model performance. At present, supervised learning, unsupervised learning and weakly supervised learning are common data labeling strategies, but in practical applications, the above-mentioned have certain limitations.

[0003] Traditional supervised learning relies on manual labeling to obtain high-quality labels, and its labeling accuracy can usually reach more than 95%. However, this method has high labor costs, long labeling period and poor scalability, and is difficult to meet the needs of large-scale or real-time tasks. The unsupervised learning method completely eliminates manual labeling and relies on data distribution to discover patterns. Although it has advantages in initial data acquisition, it often cannot generate explicit labels due to the lack of domain knowledge guidance. The accuracy of the labeling result is usually less than 60%, especially when facing semantic ambiguity, complex structure or polysemy scenes. In addition, the output label often needs to be explained and classified by humans, which increases the burden of post-processing.

[0004] Weakly supervised learning, as a compromise between the above two, automatically generates weak labels by introducing seed labels, rule systems or external knowledge bases, thereby reducing labeling costs and improving efficiency. Although this strategy can achieve labeling coverage of more than 80% in some scenarios, it has obvious shortcomings in noise control. Weak labels have multiple sources of noise, including rule conflicts, label ambiguity, sample distribution deviation, etc., and traditional statistical filtering, voting fusion and other methods are difficult to accurately identify incorrect labels from the semantic level. When the rules are not updated to adapt to the new data distribution, the overall labeling accuracy may even drop from 85% to less than 60%. In addition, the weak supervision system generally lacks confidence evaluation and label verification mechanism, which easily leads to interference of the downstream model by incorrect labels during training, affecting the generalization ability.

[0005] In recent years, large language models (LLM) such as GPT, LLaMA, etc. have shown strong capabilities in language understanding, reasoning and generation. However, most existing researches use LLM as a direct labeler, ignoring its potential in label verification, noise judgment and rule optimization. The lack of interactive feedback and rule adaptation capability makes it difficult to build a "verification-adjustment-convergence" closed loop process for weak supervision optimization.

[0006] Therefore, there is an urgent need for an innovative mechanism that combines the semantic reasoning capabilities of large language models with the advantages of weakly supervised rule-driven mechanisms, enabling high-quality label generation and multi-round dynamic noise cleaning for large-scale data while maintaining low labeling costs, and improving the accuracy and reliability of the overall training data. SUMMARY

[0007] The present application aims to provide a large language model-based multi-round cross-validation and weakly supervised noise cleaning method to solve the problems of high noise, instability and difficulty in optimizing rules in current weakly supervised methods. The method introduces LLM assistance in the weak label generation and cleaning stage through the "label seed guidance + rule generation + confidence evaluation + large model collaborative verification + rule update iteration" process, and constructs a closed-loop self-evolution weak supervision framework.

[0008] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0009] The large language model-based multi-round cross-validation and weakly supervised noise cleaning method comprises the following steps:

[0010] S1, original data acquisition and construction: collect multi-source heterogeneous unstructured data, and construct an original data set containing pure text, semi-structured text and text-image mixed description;

[0011] S2, data preprocessing and initial weak label generation: perform deduplication, format standardization and word segmentation preprocessing operations on the original data, generate initial weak labels in combination with a "triple rule library" composed of regular rules, domain dictionaries and heuristic strategies, record rule IDs and matching features in the weak labels, and form a traceable large-scale weakly supervised labeling data set;

[0012] S3, large language model verification and confidence evaluation: input the large-scale weakly supervised labeling data into a large language model verification module, combine semantic consistency verification, rule matching rationality evaluation and context consistency analysis, construct a confidence scoring system, and divide it into high confidence data and low confidence data;

[0013] S4, high confidence data into high quality knowledge base: associate the high confidence data with the verified entity and relationship information, generate structured triples and store them in the high quality knowledge base;

[0014] S5, low confidence data into active learning loop: review the low confidence data, and after labeling is completed, extract the labeling logic and convert it into executable rule instructions to generate high quality labeling data;

[0015] S6, check the model feedback fine-tuning and mechanism closed loop: using high-quality labeled data, in the setting time period or low confidence sample ratio exceeds the threshold, automatically trigger the attention mechanism of the large language model checking module and the confidence algorithm fine-tuning, continuously optimize its label judgment ability, form the closed loop iteration mechanism of "weak label generation-large model checking-feedback-rule update-model fine-tuning".

[0016] Further, in S1, the unstructured data includes webpage text, report document, question and answer dialogue and mixed content of text and image.

[0017] Further, S2 is specifically: standardizing the original data, including content deduplication, format unification, word segmentation and syntax cleaning; then, calling the "triple rule base" composed of regular rules, domain dictionary and heuristic strategy based on frequency statistics and distribution characteristics, the data is initially weakly supervised labeled to generate initial weak labels, the weak labels record the adaptive rule ID and its trigger features at the same time, forming a traceable large-scale weakly supervised labeled data set;

[0018] Further, S3 is specifically: inputting the large-scale weakly supervised labeled data into the large language model checking module, quantitatively evaluating each data by constructing a three-dimensional confidence scoring system including semantic matching degree, rule matching rationality and context consistency, and the full score is 100 points, and 80 points is taken as the classification threshold: the data with confidence higher than or equal to 80 points is classified as high-confidence data, and the data with confidence lower than 80 points or with ambiguous labeling is classified as low-confidence data.

[0019] Further, S5 is specifically: submitting the low-confidence data to experts with domain knowledge for review and manual labeling, structurally extracting the expert labeling results, summarizing the labeling logic, and converting it into weakly supervised rule update instructions with executable ability, and updating and synchronizing it to the "triple rule base" in S2 in real time, realizing the explicitization of expert knowledge and automatic enhancement of rules.

[0020] Further, S6 is specifically: using high-quality labeled data as fine-tuning data source, periodically or when the low-confidence sample ratio exceeds the preset threshold, automatically triggering the attention mechanism and confidence evaluation algorithm fine-tuning of the large language model checking module, continuously improving the checking ability and adaptability of the model, and constructing the closed loop iteration optimization path of "weak label generation-large model checking-expert review-rule update-model fine-tuning".

[0021] Compared with the prior art, the present application has the following advantages:

[0022] (1) The label quality is significantly improved: the large language model is introduced to make semantic supplementary judgment on the low-confidence samples, which effectively makes up for the boundary ambiguity and context inconsistency of rule matching.

[0023] (2) Noise rule dynamic elimination: Multi-round CV combined with consistency detection mechanism to realize automatic optimization and repair of rule base, avoiding artificial dependence and rule overfitting.

[0024] (3) Strong convergence and self-iterative optimization: The system has a closed-loop structure, with continuous iteration and evaluation of convergence ability, and finally converges to a stable label output space.

[0025] (4) Strong model scalability: The LLM part is based on modular prompt design, suitable for various text classification, entity recognition, relation extraction and other tasks.

[0026] (5) Reduce labeling cost: This method only needs a small amount of label seeds, and can get a large amount of high-quality label set through multi-round optimization of rules and large models. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The overall flowchart of the multi-round cross-validation and weakly supervised noise cleaning method based on large language model.

[0028] Figure 2 The principle diagram of the large language model automatic verification and confidence test module. DETAILED DESCRIPTION

[0029] The present application will be further described in detail below in conjunction with specific embodiments.

[0030] Example 1

[0031] To verify the effectiveness of the multi-round cross-validation and weakly supervised noise cleaning method based on large language model, the following gives a complete implementation example combined with the generation formula AI customization and rapid screening scene of ionic liquid lubricant under extreme working conditions, the specific process is as follows:

[0032] (1) Original data collection and construction:

[0033] The sources of collection include English SCI journal papers, experimental reports, test data records and English and Chinese technical patents, covering the performance of lubricants under extreme working conditions such as high temperature, high pressure, high radiation and vacuum, and the data types include: PDF full text, chart description, structural formula picture and corresponding text description, summary sentences of lubricating performance in the text. After collection, a data set containing pure text (such as abstract), semi-structured data (such as image-text annotation pair), and graph-text embedded content is constructed, and is uniformly converted into JSONL format.

[0034] (2) Data preprocessing and initial weak label generation: Perform standardization cleaning, including unit unification (such as converting °C to K), chemical name standardization (such as converting EMIM to 1-ethyl-3-methylimidazole), and chemical structure formula conversion, etc. Then, combined with the "triple rule base", weak supervision labeling is performed. Example rules: Regular rules: match "at __ temperature __, pressure __, the lubrication coefficient is" to extract structure-working condition-performance triplets; dictionary matching rules: identify the combination mode of [ionic liquid name]+[performance index]; heuristic strategy: identify improvement factors based on verb phrases "improve" and "optimize". Record the matched rule ID and trigger position in the labeled data to form a large-scale weakly supervised labeling dataset.

[0035] (3) Large language model verification and confidence evaluation: input the large-scale weakly supervised labeling data into the fine-tuned large language model (such as the version based on LLama-3 or GPT-4), and perform cross-validation, including: judging whether the semantics and the original text are consistent (such as "lubrication effect improvement" is indeed derived from "friction coefficient from 0.12 to 0.05"); check the logical consistency and syntactic correctness of the triplets; analyze whether the context working condition has interference factors (such as additive intervention). Scoring uses three-dimensional scoring, and after weighting, the confidence score is generated. Mark high-confidence data when the confidence score is ≥80.

[0036] (4) High-confidence data into high-quality knowledge base: the following triplets are marked as high-confidence data: <emim-tfsi>– <300K, 100MPa> – <friction coefficient 0.06>, it is associated into a high-quality knowledge base, linking structure information (SMILES expression), experimental parameters and result indicators, forming a graph node oriented to lubrication performance optimization.

[0037] (5) Low confidence data enters the active learning loop: Some data, such as lubricants using fluorinated imidazole cations, exhibit excellent thermal stability at high shear rates, and are marked as low confidence due to abstract expression, missing values, etc. The system submits this sentence segment to a materials chemistry expert for review, and the expert confirms it as: <fluorinated imidazole cation IL> - <high shear rate (>10 4 s -1 )> - <thermal stability improvement>. The system automatically extracts the logical rules used for reasoning in the expert's annotations, such as "high shear rate → investigate thermal stability indicators", and converts them into new weak supervision rules, updating them to the "triple rule base".

[0038] (6) Verify model feedback fine-tuning and mechanism closed loop: When the cumulative number of review samples exceeds 15% of the total amount of low confidence data, trigger model fine-tuning. Use the expert's review and annotation samples as fine-tuning data, and fine-tune the LLM verification module through low learning rate incremental learning. The fine-tuned model can more accurately judge the potential meaning of "ambiguous working condition description" in the next round of weak label review, significantly improving the model's semantic judgment ability and rule generalization ability.

[0039] Effect verification: In the automatic processing of 500 papers related to ionic liquid lubricants, the weak label accuracy of the method can be improved from 63.5% of the original rule base annotation to 89.2%, and a materials knowledge graph containing 13,000 structure-condition-performance triples can be constructed within 72 hours, significantly improving the quality of available data and model training efficiency in generative materials design tasks.

Claims

1. A method for multi-round cross-validation and weakly supervised noise cleaning based on a large language model, characterized in that, Comprise the following steps: S1, original data acquisition and construction: collect multi-source heterogeneous unstructured data, construct original data set containing pure text, semi-structured text and text-image mixed description; S2, data preprocessing and initial weak label generation: the original data is de-duplicated, format standardized, word segmentation preprocessing operation, combined with the "triplet rule base" composed of regular rules, domain dictionary and heuristic strategy to generate initial weak label, weak label records rule ID and matching feature, form traceable large-scale weakly supervised labeling data set; S3, large language model verification and confidence evaluation: input large-scale weakly supervised labeling data into large language model verification module, combined with semantic consistency test, rule matching rationality evaluation and context consistency analysis, construct confidence scoring system, divided into high confidence data and low confidence data; S4, high confidence data into high quality knowledge base: the high confidence data is associated with the verified entity and relationship information, and the structured triplets are stored in the high quality knowledge base; S5, low confidence data into active learning loop: review the low confidence data, and after labeling, extract the labeling logic and convert it into executable rule instructions to generate high quality labeling data; S6, feedback fine-tuning of verification model and mechanism closed loop: using high quality labeling data, when the low confidence sample ratio exceeds the threshold value in the set time period or low confidence sample ratio exceeds the threshold value, the attention mechanism and confidence algorithm of the large language model verification module are automatically triggered for fine-tuning, continuously optimizing the label judgment ability of the model, forming a closed loop iteration mechanism of "weak label generation-large model verification-feedback-rule update-model fine-tuning".

2. The method of claim 1, wherein, In S1, the unstructured data includes web text, report documents, question and answer dialogues and mixed text and image content.

3. The method of claim 1, wherein, S2 is: standardizing the original data, including content de-duplication, format unification, word segmentation and syntax cleaning; then, calling the "triplet rule base" composed of regular rules, domain dictionary and heuristic strategy based on frequency statistics and distribution characteristics, the data is initially weakly supervised labeled, and the initial weak label is generated. Weak label records the rule ID and its trigger feature, forming a traceable large-scale weakly supervised labeling data set.

4. The method of claim 1, wherein, S3 is: input large-scale weakly supervised labeling data into large language model verification module, through building a three-dimensional confidence scoring system including semantic matching degree, rule matching rationality and context consistency, each data is quantitatively evaluated, and the full score is 100 points. Take 80 points as the classification threshold: data with confidence higher than or equal to 80 points is classified as high confidence data, and data with confidence lower than 80 points or ambiguous labeling is classified as low confidence data.

5. The method of claim 1, wherein, S5 is: low confidence data is submitted to experts with domain knowledge for review and manual labeling, the expert labeling results are structured extracted, the labeling logic is summarized, and the weak supervision rule update instructions with executable ability are converted, which are updated in real time and synchronized to the "triplet rule base" in S2. Realize the explicitization of expert knowledge and automatic enhancement of rules.

6. The method of claim 1, wherein, S6 is specifically: taking high-quality labeled data as fine-tuning data source, periodically or when the proportion of low-confidence samples exceeds the preset threshold, automatically triggering the attention mechanism and confidence evaluation algorithm fine-tuning of the large language model verification module, continuously improving the verification ability and adaptability of the model, and constructing a closed-loop iterative optimization path of "weak label generation-large model verification-specialist review-rule update-model fine-tuning".

Images