A domain corpus data review and automatic correction method based on generative model

Abstract

The present invention belongs to the field of artificial intelligence technology, and provides a method for domain corpus data review and automatic correction based on a generative model, aiming to solve the problems of domain corpus data review relying on experts, low efficiency and subjective influence. The technical solution includes: pre-processing the original corpus, standardizing, classifying, correcting and reorganizing it through LLM; using multiple LLMs to generate opinions, perform cluster analysis, calculate semantic entropy to evaluate factual correctness, and score classification; manually review the medium-scoring corpus, and automatically correct the unqualified corpus; cyclically review until it is qualified, and collect feedback to update the LLM. This method reduces manual dependence, enhances the model's self-review and update capabilities, realizes automatic correction, and improves the accuracy and efficiency of multi-domain corpus review. The method of the present invention can be applied to different fields such as education, medical care, finance, and law to improve the accuracy and efficiency of domain corpus data review.

Claims

1. A method for reviewing and automatically correcting domain corpus data based on a generative model, characterized in that: The following steps are involved: Step 1: Preprocess the original corpus to obtain standardized corpus. The corpus types include single-turn dialogue, multi-turn dialogue and statement. Step 2: Use the Large Language Model (LLM) to review the standardized corpus and obtain the scoring results; Step 2.1: M LLMs generate opinions on the current corpus based on the prompt project. Each LLM generates K opinions, resulting in a total of K × M opinions. Step 2.2: Perform cluster analysis on the opinions obtained in step 2.1 and calculate the generation probability of each semantic equivalence class; Step 2.3: Calculate the semantic entropy based on the generation probability obtained in step 2.2 to determine the factual correctness of the corpus. Step 2.4: Based on the preset pass standard k1 and fail standard k2, the corpus is divided into three categories: high score, medium score, and low score, and the scoring results are output; Step 2.4.

1. Set the initial pass criterion k1 and fail criterion k2. These criteria are adjusted based on domain knowledge and experience. Step 2.4.2: Compare the semantic entropy of each corpus with the pass standard k1 and the fail standard k2; Step 2.4.3: Classify the corpus with semantic entropy values ​​lower than k1 as high-scoring corpus; Step 2.4.4: Classify the corpus with a semantic entropy value higher than k2 as low-scoring corpus, and the low-scoring corpus as unqualified corpus; Step 2.4.5: Classify the corpus with semantic entropy values ​​between k1 and k2 as medium-scoring corpus; Step 2.4.6: Organize the classification results and generate a JSON object for each corpus containing the original corpus paragraph, semantic entropy value, and scoring category; Step 2.4.7: Output the scoring results of all corpora; Step 3: Manually review the corpus with a medium LLM review result and obtain the final score feedback; Step 4: Automatically correct the unqualified corpus to obtain the corrected corpus; Step 4.1: Distribute unqualified corpus to manual rewriting and LLM rewriting modules according to the preset ratio; Unqualified corpora enter the manual rewriting module and LLM rewriting module respectively according to the ratio of p and 1-p, where p obeys the following formula: Among them, m is the cumulative number of corpus items reviewed, and the parameter It is used to adjust the rate of change of p according to the total number of corpora. According to the characteristics of the sigmoid function, as m gradually increases, p will follow a smooth decreasing trend and remain basically unchanged when m is greater than a certain threshold. That is, as the corpus accumulates, the manual workload in the manual rewriting module will gradually decrease. At the end of the process, the manual rewriting module and the LLM rewriting module jointly output the modified corpus. Step 4.2: Multiple people simultaneously rewrite the corpus that has entered the manual rewriting module to form modified corpus pairs, which are then entered into the modified corpus pair warehouse. Step 4.3: When the modified corpus reaches the preset number in the warehouse, it is input into the LLM update module to fine-tune the LLM; Step 4.4: Automatically rewrite the corpus that enters the LLM rewriting module and output the revised corpus; Step 5: Review the revised corpus again until it passes the review and becomes qualified; Step 6: Collect scoring feedback and update the LLM review model; Step 6.1: Collect a preset number of rating feedbacks and calculate the relationship between semantic entropy and final rating; Step 6.2: Recalculate the pass standard k1 and the fail standard k2 based on the statistical results; Step 6.3: Modify the specified context when the target corpus review task is changed or the results of several updates are not ideal; Step 6.4: Use the updated k1, k2, and specified context to update the LLM audit model to improve the accuracy of subsequent audits.

2. A method for reviewing and automatically correcting domain corpus data based on a generative model according to claim 1, characterized in that: Step 1 includes the following steps: Step 1.1: Standardize the format of the original corpus from the Internet, manual collection, or generated by large language models and convert it into JSON format objects; Step 1.2: Classify and count the JSON objects obtained in step 1.1, and classify them according to the corpus type, including single-turn dialogue, multi-turn dialogue, and statement; Step 1.3: Using LLM-based fuzzy recognition technology, correct the word order and punctuation of the classified corpus obtained in step 1.2; Step 1.4: Perform syntactic analysis, part-of-speech tagging, semantic analysis, and sentence reorganization on the revised corpus obtained in step 1.3 to obtain a reorganized text; Step 1.5: Map the reorganized text into JSON format and output it as standardized corpus.

3. A method for reviewing and automatically correcting domain corpus data based on a generative model according to claim 2, characterized in that: Step 2.1 includes the following steps: Step 2.1.1, select multiple LLMs that have been fine-tuned by instructions; Step 2.1.2: Segment the standardized corpus into paragraphs or sentences suitable for LLMs processing; Step 2.1.3: Design a prompt project for each LLM to guide the model to generate opinions about the correctness of the corpus facts; Step 2.1.4: Input the segmented corpus into each LLM to generate multiple different viewpoints , represents the opinion of the j-th output of model i, i=1,...,K, j=1,...,M; Step 2.1.5: Repeat step 2.1.4, and generate K different views for each LLM to increase the diversity of the results; Step 2.1.6: Collect all the opinions generated by LLM, totaling K×M.

4. A method for reviewing and automatically correcting domain corpus data based on a generative model according to claim 3, characterized in that: Step 2.2 includes the following steps: Step 2.2.

1. Use the pre-trained BERT model to transform each opinion Encoded as a semantic vector of fixed dimension; Step 2.2.2: Calculate the cosine similarity between all semantic vectors as the similarity measure for cluster analysis. Step 2.2.3: Apply the K-means clustering algorithm to divide the semantic vectors into multiple clusters based on cosine similarity, i.e., semantic equivalence classes; Step 2.2.4: Count the number of opinions in each cluster and calculate the generation probability of each semantic equivalence class : in is a semantic equivalence class, For this semantic equivalence class The viewpoint in It is an artificially set reminder engineering standard. Manually set by the reviewer based on the review content, it is the standard for reviewing the corpus or the keywords that the corpus must meet; Step 2.3 includes the following steps: Step 2.3.1, according to the generation probability , the calculation formula of semantic entropy is applied ; Step 2.3.2: Calculate the semantic entropy value for each semantic equivalence class. Step 2.3.3: Summarize the semantic entropy values ​​of all semantically equivalent classes to obtain the semantic entropy measure of the entire corpus; Step 2.3.4: Analyze the semantic entropy value to determine the factual correctness of the corpus. The lower the semantic entropy value, the higher the factual correctness. Step 2.3.5: Output the semantic entropy value of each corpus.

5. The method for reviewing and automatically correcting domain corpus data based on a generative model according to claim 1, characterized in that: Step 3 includes the following steps: Step 3.1: Input the intermediate-scoring corpus output by the LLM review into the public review module and have multiple reviewers perform scoring. Step 3.2: Statistically analyze the reviewers' scores, remove significant outliers, and calculate the mean. Step 3.3: Identify disputed items and input them into the expert review module for re-examination; Step 3.4: The expert review module reviews the disputed items and determines whether they are qualified or not; Step 3.5: Integrate the original corpus and the final score, and output the score feedback as a JSON object.

6. A method for reviewing and automatically correcting domain corpus data based on a generative model according to claim 1, characterized in that: Step 5 includes the following steps: Step 5.1: Re-enter the revised corpus into the LLM review module for a new round of review; Step 5.2: Judge the audit result obtained in step 5.

1. If it is qualified, output the qualified corpus. If it is unqualified, return to step 4 for further correction. Step 5.3: Repeat steps 5.1 and 5.2 until all corpora pass the review and are qualified.

Images