A subject corpus annotation method and system based on multiple rounds of classifier iteration

Abstract

The application discloses a subject corpus annotation method and system based on classifier multi-round iteration, and belongs to the field of natural language processing and machine learning. The method comprises the following steps: constructing a multi-level subject classification system, based on a Fasttext classifier and a seed set, performing multi-round iterative reasoning on a target subject, dynamically generating positive and negative samples in each round of iteration, and gradually improving the classification accuracy; using a large model to score the positive samples in multiple dimensions to screen high-quality positive samples, and extracting keywords to optimize the classification boundary; generating subject questions through a WebQA method and retrieving supplementary low-recall rate subject corpus; training an information filter to identify and filter out low-quality content such as advertisements and garbage; and finally realizing efficient and accurate subject annotation through the series process of the classifier and the filter. The application solves the problems of low efficiency, large resource consumption and subject understanding deviation of traditional annotation, and is suitable for efficient and accurate subject annotation of large-scale text data.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and machine learning, and in particular to a method and system for annotating subject-specific corpora based on multi-round iteration of classifiers. Background Technology

[0002] In large language model pre-training, the quality of high-quality corpora and subject-specific classification corpora directly affects the model's domain adaptability and reasoning ability. High-quality corpora must meet characteristics such as reliable data sources, high content accuracy, low noise content, and strong semantic integrity. Their quality directly determines the effectiveness and accuracy of basic knowledge acquisition during model pre-training. If the proportion of high-quality corpora in the pre-training corpus is insufficient, or if there are quality defects such as content bias or noise interference, it will lead to problems such as basic semantic understanding deviations and incomplete general knowledge reserves in subsequent applications, thereby weakening the model's adaptability in cross-domain scenarios. Existing technologies relying on large models for subject labeling have the following problems:

[0003] Inefficient: Large model annotation is slow, consumes a lot of resources, and is difficult to process massive amounts of data;

[0004] Subject-specific comprehension bias: The large model's definition of specific subjects is unclear, leading to insufficient labeling accuracy;

[0005] Data imbalance: The scarcity of corpora for subjects with low recall rates affects the model training effect;

[0006] Noise interference: Unlabeled data contains advertisements and spam, which reduces data quality.

[0007] Therefore, there is an urgent need for an efficient, accurate, and scalable subject annotation method to support the construction of high-quality corpora and model optimization. Summary of the Invention

[0008] To address the problems of low annotation efficiency, high resource consumption, subject comprehension bias, and lack of low-recall subject corpora in existing technologies, the present invention aims to provide a subject corpus annotation method and system based on multi-round iteration of classifiers.

[0009] The objective of this invention is achieved through the following technical solution: a subject-specific corpus annotation method based on multi-round iteration of a classifier, comprising the following steps:

[0010] Based on preset classification criteria, a multi-level subject classification system is constructed;

[0011] Based on the text classifier, multiple rounds of iterative reasoning are performed on the unlabeled text in the unlabeled dataset. In each round of iterative reasoning, a training sample set is dynamically constructed based on the output of the previous round of the text classifier and the unlabeled dataset.

[0012] Among them, a large model is introduced in the iterative reasoning to score the relevance, scientificity, popular science and content safety of positive samples, and positive samples with a comprehensive score higher than the threshold are selected as high-quality positive samples under the premise of content safety.

[0013] For subjects with low recall rates, a question database is generated and deduplicated using a vector model, and then web search is used to retrieve and supplement the data.

[0014] Furthermore, the construction of a multi-level subject classification system based on preset classification criteria includes: the multi-level subject classification system includes first-level subjects, second-level subjects, and third-level subjects, and assigns a unique code to each subject at each level.

[0015] Furthermore, the preset classification criteria are subject classification principles based on national standards.

[0016] Furthermore, the multi-round iterative reasoning includes four rounds, specifically:

[0017] First round: Train the initial text classifier using gold standard positive samples and mixed negative samples;

[0018] Second round: Based on the output of the text classifier after the first round of training, samples with different confidence intervals are extracted according to the proportion of normal distribution, and negative samples of the dataset to be labeled are used as negative samples for this round of training.

[0019] The third round: Based on the output of the text classifier after the second round of training, the positive samples are scored for relevance, scientificity, popular science and content safety by a large model to select high-quality positive samples and supplement them with the same number of negative samples.

[0020] Fourth round: Extract keyword features from the gold standard positive samples, generate prompt words, and optimize the text classifier's understanding of subject boundaries, repeating the third round of iteration.

[0021] Furthermore, the text classifier employs the Fasttext classifier.

[0022] Furthermore, the content security judgment specifically involves: using a large model to filter the content of positive samples; if the content is unhealthy, it is marked as spam; and training a Fasttext classifier based on the spam information as an information filter to remove low-quality content.

[0023] Furthermore, the specific steps for generating a question database for subjects with low recall rates, deduplicated using a vector model, and then using web search to recall and supplement the data are as follows:

[0024] Based on the text in the gold standard dataset of the target discipline, a large language model is used to generate multiple questions related to the discipline.

[0025] The generated questions are vector-encoded, and the cosine similarity between question pairs is calculated. Deduplication is performed based on a preset similarity threshold to obtain a deduplicated question library.

[0026] Use a search engine to perform a web search on the deduplicated question database to obtain relevant Uniform Resource Locators and webpage snapshot information;

[0027] Based on the Uniform Resource Locator (URL), the resource type is determined. If it points to a resolvable static resource, the text content is extracted using an optical character recognition (OCR) algorithm. If it points to a Hypertext Markup Language (HMR) resource, data cleaning is performed to extract the text content.

[0028] The extracted text content is added to the corpus of the target subject to improve the coverage and balance of the corpus in that subject.

[0029] This invention also provides a subject-specific corpus annotation system based on multi-round classifier iteration, comprising:

[0030] Classification system construction module: used to construct a multi-level subject classification system based on preset classification criteria;

[0031] Multi-round iterative reasoning module: used for multi-round iterative reasoning on unlabeled text;

[0032] Large model collaboration module: used for scoring samples, extracting keywords, and updating hit and scoring prompts;

[0033] WebQA Supplement Module: Used to generate a question bank for subjects with low recall rates and use web searches to recall supplementary data;

[0034] Information filtering module: Used to train and execute information filters to remove low-quality content.

[0035] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method.

[0036] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0037] The beneficial effects of this invention are:

[0038] 1) Significantly improves annotation efficiency and reduces direct calls to large models;

[0039] 2) By extracting keywords and optimizing prompts, the difference between the large model and the seed set in terms of subject definition is reduced;

[0040] 3) WebQA methods effectively supplement the corpus of subjects with low recall and optimize data distribution;

[0041] 4) The information filter and the large model scoring mechanism work together to ensure high-quality training data. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a subject corpus annotation method based on multi-round classifier iteration, provided as an embodiment of the present invention.

[0044] Figure 2 A flowchart illustrating a three-level subject classification method provided in an embodiment of the present invention;

[0045] Figure 3 A flowchart illustrating a multi-round iterative method for a subject classifier provided in an embodiment of the present invention;

[0046] Figure 4 A flowchart illustrating the alignment of large model and gold standard data subject features in an R4 subject classifier, as provided in an embodiment of the present invention;

[0047] Figure 5 This is a flowchart illustrating an information filter training and iteration method according to an embodiment of the present invention.

[0048] Figure 6 This is a flowchart illustrating a WebQA-based method for supplementing subject-specific text, as provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0051] like Figure 1As shown, this embodiment of the invention provides a subject corpus annotation method based on multi-round classifier iteration, including the following steps:

[0052] Step 1: Construct a multi-level subject classification system based on preset classification criteria;

[0053] Step 2: Perform multiple rounds of iterative inference on the unlabeled text in the unlabeled dataset based on the text classifier to complete the iterative training of the text classifier. In each round of iterative inference, a training sample set is dynamically constructed based on the output of the text classifier in the previous round and the unlabeled dataset.

[0054] Step 3: In the iterative reasoning process, a large model is introduced to score the relevance, scientific validity, popular science value, and content safety of positive samples, and high-quality positive samples are selected based on the comprehensive score under the premise of content safety.

[0055] Step 4: For subjects with low recall rates, generate a question database, deduplicate the questions using a vector model, and then use web search to retrieve and supplement the data.

[0056] Step 5: Label the subject-specific corpus using the trained text classifier.

[0057] As a preferred embodiment, such as Figure 2 As shown, the multi-level subject classification system described in step one includes three levels, specifically:

[0058] First-level disciplines, such as "Applied Sciences" and "Arts and Humanities", are identified by uppercase letters such as "A, B, C, etc." in this example.

[0059] Second-level disciplines, such as "Engineering" and "Information and Communication Technology", are identified by lowercase letters such as "a", "b", and "c" in this example.

[0060] Third-level disciplines, such as "Computer Theory Mathematics" and "Medical Informatics," are identified by Arabic numerals such as "1, 2, 3" in this example.

[0061] Any tertiary discipline can be named with a combination of uppercase letters, lowercase letters, and numbers. If the primary discipline "Applied Science" is named A, the secondary discipline "Information and Communication Technology" can be named "Ab", and the tertiary discipline "Medical Informatics" can be named "Ab3".

[0062] As a preferred embodiment, such as Figure 3 As shown, step two specifically includes the following sub-steps:

[0063] The text classifier (subject classifier) ​​undergoes four rounds of iterative training. The following will use subject "Ab3" as an example to illustrate the process:

[0064] R1 Training: Data is drawn from the "Ab3" subject of the gold standard dataset (labeled α) as positive samples, and the total number of positive samples is set as S. Then, a total of M negative samples are randomly drawn from other subjects under the "Ab" secondary subject except "Ab3" according to the distribution ratio. And N negative samples are randomly drawn from other secondary subjects except "Ab" according to the ratio, ensuring that S≈M+N (the positive and negative samples must be of the same order of magnitude, which will not be elaborated below). The first round of training of the Ab3 Fasttext classifier is completed and labeled as Ab3-R1.

[0065] R2 Training: Based on the inference results of the R1 classifier on the α and β sets, a reasonable threshold is set according to the histogram. For example, the threshold can be set to T so that R1 infers the same number of positive and negative examples on the α and β sets. Positive samples are drawn from the samples that R1 infers positively on the α set (the most discriminative texts in the subject); negative samples consist of two parts: one part is the texts that are inferred negatively on the α set (introducing reasonable noise), and the other part is the texts that are inferred negatively on the unlabeled dataset (introducing new negative sample features), drawn proportionally from different confidence intervals. A second round of training, Ab3 classifier, is then conducted based on R1, and the completed training is labeled Ab3-R2.

[0066] R3 Training: Based on the inference results of the R2 classifier on the α and β sets, reasonable thresholds are set according to the histogram to infer pseudo-positive and negative samples. Reasonable prompts are set, and the pseudo-evidence samples in the β set are scored in four aspects using the large model collaboration module: scientific validity (0-10), popular science (0-10), relevance (0-10), and spam (0 or 1). Spam includes harmful content such as pornography, advertising, and violence. After scoring, samples belonging to spam are extracted and will be used as positive samples for the "information filter". The remaining samples with an average score >7 are used as positive samples for this round of training (introducing new positive samples). The composition of negative samples is the same as that of R2, and they are sampled proportionally from the negative samples inferred by the R2 classifier in the α and β sets. The training iteration of the R3 classifier is completed.

[0067] R4 Training: Slightly different from R3, we used positive example feature summarization and hit detection to align the large model's definition of the discipline with the gold standard set's definition of the discipline, such as... Figure 4 As shown, samples that the R3 classifier infers as positive in the α set are sampled. Keyword extraction is performed using the large model co-operation module, extracting several keywords from each sample to form a large set of keywords related to the subject. Then, samples that the R3 classifier infers as positive in the β set are sampled again, and keywords are extracted. The frequency of the keywords in the above set is counted, and the top 100 high-frequency keywords are used as the definition reference for the subject, updating the cue words for labeling in the large model. The process of generating positive and negative samples using R3 is then repeated iteratively with the R4 classifier.

[0068] As a preferred embodiment, such as Figure 5 As shown, step three specifically involves:

[0069] Both R3 and R4 training processes used large model scoring and labeling, with the labeling content including the filtering of spam information. The definition of spam information includes various types of unhealthy content.

[0070] In R3 sample generation, texts judged as spam are used as positive examples, and texts randomly selected from the gold standard set are used as negative examples. A Fasttext classifier can be trained as an information filter, and the same process is repeated in R4 sample generation.

[0071] In a preferred embodiment, step four specifically involves:

[0072] For subjects that generate few positive example texts during R4 inference, WebQA methods can be used to supplement the corpus for that subject to ensure the balance of subsequent subject labels. Figure 6 As shown. WebQA is a method that uses search engines to find academic texts related to a specific subject through questions, searches, and web crawling.

[0073] First, extract the text from the α set of the subject and generate subject-specific questions using the large model collaborative module. For the generated question set, use a vector model for encoding and compare cosine similarity (semantic similarity) pairwise. If the similarity is greater than a threshold (e.g., 0.9), randomly discard one question. For the remaining questions, use a search engine API to perform a web search, obtaining the returned Uniform Resource Locator (URL), snapshot, and other information. If the URL points to a parsable static resource (such as a PDF or image), optical character recognition (OCR) algorithms can be used for text extraction. If the URL points to a Hypertext Markup Language (HTML) resource, the text content is cleaned using regular expression matching, including but not limited to the following:

[0074] Image content cleaning: Remove image links contained in HTML, such as... <figure>or <src img>wait.

[0075] Email, phone number, and other content cleanup: Remove private information such as email addresses and phone numbers.

[0076] HTML formatting cleansing: Remove meaningless HTML formatting tags, such as font formatting, paragraph formatting, background formatting, etc., and generally includes CSS syntax.

[0077] Noise cleaning: Removes navigation bar, advertisement, header and footer tags and other content.

[0078] This invention also provides a subject-specific corpus annotation system based on multi-round classifier iteration, comprising:

[0079] Classification system construction module: Used to construct a multi-level subject classification system based on preset classification criteria.

[0080] Multi-round iterative reasoning module: Used for multi-round iterative reasoning on unlabeled text.

[0081] Large model collaboration module: used for scoring samples, extracting keywords, and updating hit and scoring prompts.

[0082] WebQA Supplement Module: Used to generate a question bank for subjects with low recall rates and use web searches to recall supplementary data.

[0083] Information filtering module: Used to train and execute information filters to remove low-quality content.

[0084] For the system implementation, since it basically corresponds to the method implementation, the relevant parts can be referred to in the description of the method implementation.

[0085] This invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method.

[0086] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0087] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0088] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.< / src> < / figure>

Claims

1. A subject corpus annotation method based on classifier multi-round iteration, characterized in that, The method comprises the following steps: constructing a multi-level subject classification system based on preset classification criteria; based on the text classifier, performing multiple rounds of iterative reasoning on the unannotated texts in the to-be-annotated data set, and dynamically constructing a training sample set according to the output result of the text classifier of the previous round and the to-be-annotated data set; the multiple rounds of iterative reasoning include four rounds, specifically: the first round: using the gold standard positive samples and mixed negative samples to train an initial text classifier; the second round: based on the output result of the text classifier after the first round of training, samples in different confidence intervals are extracted according to a normal distribution ratio, and the negative samples in the to-be-annotated data set are combined as the negative samples for this round of training; the third round: based on the output result of the text classifier after the second round of training, the high-quality positive samples are selected by using a large model to score the relevance, scientificity, popularization and content safety of the positive samples, and the same number of negative samples are supplemented; the fourth round: extracting key features from the gold standard positive samples, generating prompt words and optimizing the understanding of the subject boundary of the text classifier, and repeating the third round of iterative process; wherein, in the iterative reasoning process, the large model is used to score the relevance, scientificity, popularization and content safety of the positive samples, and the positive samples with a comprehensive score higher than a threshold value are selected as high-quality positive samples under the premise of content safety; for low-recall-rate subjects, a question library is generated, and after de-duplication by a vector model, web search is used to recall supplementary data.

2. The method of claim 1, wherein, The multi-level subject classification system comprises a first-level subject, a second-level subject and a third-level subject, and each subject at each level is assigned a unique code.

3. The method of claim 1, wherein, The preset classification criteria are based on the subject classification principles of national standards.

4. The method of claim 1, wherein, The text classifier uses a Fasttext classifier.

5. The method of claim 1, wherein, The content safety judgment specifically includes: using a large model to screen the content of the positive samples, if the content is unhealthy, it is marked as spam information, and the Fasttext classifier is trained based on the spam information as an information filter to remove low-quality content.

6. The method of claim 1, wherein, For low-recall-rate subjects, a question library is generated, and after de-duplication by a vector model, web search is used to recall supplementary data, specifically: based on the texts in the gold standard data set of the target subject, a large language model is used to generate multiple questions related to the subject; vector encoding is performed on the generated questions, and the cosine similarity between question pairs is calculated, and de-duplication is performed based on a preset similarity threshold to obtain a de-duplicated question library; using a search engine to perform web search on the de-duplicated question library to obtain related uniform resource locators and web snapshot information; based on the uniform resource locators, the resource types are determined, if they point to analyzable static resources, an optical character recognition algorithm is used to extract the text content, if they point to hypertext markup language resources, data cleaning is performed to extract the text content; the extracted text content is supplemented to the corpus of the target subject to improve the coverage and balance of the subject corpus.

7. A system for subject corpus annotation based on multiple rounds of iterations of classifiers, characterized in that, It comprises: a classification system construction module for constructing a multi-level subject classification system based on preset classification criteria; A multi-round iterative reasoning module is configured to perform multi-round iterative reasoning on the unlabeled text, and the multi-round iterative reasoning includes four rounds, specifically: In the first round, an initial text classifier is trained using gold standard positive samples and mixed negative samples; In the second round, based on the output result of the text classifier after the first round of training, samples in different confidence intervals are extracted according to a normal distribution ratio, and negative samples in a to-be-labeled data set are combined as negative samples for the second round of training; In the third round, based on the output result of the text classifier after the second round of training, high-quality positive samples are selected by performing relevance, scientificity, popularization and content safety judgment on the positive samples by a large model, and the same number of negative samples are supplemented; In the fourth round, key features are extracted from the gold standard positive samples, prompt words are generated, and the understanding of the text classifier for the discipline boundary is optimized, and the third round of iteration is repeated; A large model coordination module is configured to score the samples, extract keywords, hit the scoring prompt words and update the scoring prompt words; A WebQA supplement module is configured to generate a question library for a low-recall discipline and use web search to recall supplementary data; An information filtering module is configured to train and execute an information filter to eliminate low-quality content.

8. An electronic device comprising a memory and a processor, characterized in that The memory is coupled to the processor, and the memory is configured to store program data, and the processor is configured to execute the program data to implement the method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-6.

Images