A cold start scenario discipline classification model iteration optimization method and system

Abstract

The application discloses a cold start scene discipline classification model iteration optimization method and system, wherein the iteration optimization method adopts double iteration optimization, is based on constructing an initial sample pool, and starts model training; a three-stage filtering and dynamic priority evaluation mechanism is designed to screen high-value samples from massive unannotated data, so that precise expansion of training data is realized; a large language model is used to analyze the root cause of model misjudgment, automatically identify misjudgment types, and supplement weak data in a targeted manner; finally, through sample pool version management, continuous updating and backtracking of data are realized. The application solves the problem of lack of annotated data faced by the discipline classification model in the cold start scene, significantly reduces the dependence of model training and iteration on intensive manual annotation and field data analysis through systematic and efficient automatic data construction and optimization closed loop, and realizes the coordinated evolution of model classification performance and training data in the automatic closed loop.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing technology, and specifically relates to an iterative optimization method and system for a subject classification model in a cold start scenario. Background Technology

[0002] With the rapid development of digitized academic resources, automated and high-precision subject classification technology has become a core requirement for filtering and organizing professional knowledge from massive amounts of text. This type of technology aims to provide fundamental support for applications such as educational content recommendation, academic resource navigation, and subject corpus construction. However, in the actual process of building subject classification models based on massive amounts of text (such as massive amounts of web page data), the following prominent technical challenges are faced:

[0003] 1. Cold Start Problem: In the initial stages of subject classification tasks, there is often a lack of large-scale, high-quality labeled data. Existing methods typically rely on experts manually labeling a small amount of seed data or using coarse rules based on keywords for initial data collection. However, manual labeling is costly and inefficient, while rule-based methods have low recall and high noise. Both methods struggle to obtain sufficient and balanced initial training data, resulting in weak initial model performance that cannot meet the needs of subsequent precise selection of subject datasets, thus falling into a vicious cycle of "no high-quality data → poor model → inability to select high-quality data".

[0004] 2. Data Bias and Incomplete Coverage: Even with initial training data, its distribution often exhibits significant bias. For example, the data may be overly concentrated in certain popular disciplines or popular subfields within those disciplines (such as "machine learning" in computer science), while lacking coverage of less popular disciplines and subfields. Furthermore, the data quality distribution is extremely uneven, predominantly consisting of well-structured and standardized text, while lacking coverage of common colloquial or mixed-quality text. This data bias causes the trained model to perform reasonably well in a "greenhouse environment," but once applied to real, diverse, and massive amounts of web page text data, its ability to distinguish less popular disciplines and general-quality text drops sharply, severely impacting its robustness in practical applications and making true "precise selection" difficult.

[0005] 3. Inefficient Iterative Optimization: To improve model performance, current technologies typically employ iterative optimization strategies. This involves using the current model to predict unlabeled data, then manually reviewing the predictions to select new samples for the training set. This process heavily relies on manual analysis by domain experts, which is not only time-consuming and labor-intensive, but also subjective, and makes it difficult to systematically quantify the distribution of model weaknesses. Experts struggle to quickly and accurately identify model weaknesses from thousands of predicted samples, resulting in unclear data augmentation directions, slow optimization processes, and limited effectiveness. This inefficient iterative approach severely restricts the feasibility of rapidly building high-performance subject classification models in massive data environments.

[0006] To address these challenges, existing research has attempted to introduce methods such as active learning and semi-supervised learning. However, these methods typically focus only on a single optimization objective (such as model uncertainty) and fail to incorporate disciplinary knowledge structures (such as curriculum systems) and data quality dimensions into a unified screening framework. Furthermore, these methods lack mechanisms for automated and structured diagnosis of model defects, thus failing to achieve a fundamental shift from "human experience-driven" to "algorithm-data-driven."

[0007] Therefore, for application scenarios that build subject classification models based on massive text data for accurate data screening, there is an urgent need for an innovative method that can automatically de-cold start, systematically eliminate data bias, and intelligently guide iterative optimization. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides an iterative optimization method and system for subject classification models in cold-start scenarios. This method addresses key technical problems in the early stages of subject classification tasks, such as difficulties in model training due to a lack of labeled data, weak model generalization ability due to uneven data distribution, and high optimization costs due to low efficiency of manual iteration. It automatically and efficiently constructs a broad-coverage and balanced training dataset from massive amounts of text, significantly reducing the reliance on intensive manual annotation and domain data analysis, and achieving the co-evolution of model classification performance and training data in an automated closed loop.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides an iterative optimization method for a subject classification model in a cold start scenario, comprising the following steps:

[0011] S1. For the target subject, extract initial positive samples from the acquired original dataset based on preset rules, and extract initial negative samples from data of other subjects. Merge the positive and negative initial samples to form an initial training sample pool.

[0012] S2. Based on the initial training sample pool, an initial subject classification model is obtained by training a text classification algorithm;

[0013] S3. Use the initial subject classification model to obtain high-value samples from the original dataset using a three-level funnel filtering and dynamic priority evaluation mechanism. Use a large language model or manual methods to perform fine annotation. Merge the newly labeled samples with the historical sample pool, remove duplicates, and update the training sample pool.

[0014] S4. Retrain the subject classification model based on the updated sample pool, and use the large language model to analyze the misclassified samples of the trained subject classification model through structured prompt words, and then supplement the positive and negative sample data with targeted augmentation to generate a targeted augmentation sample pool.

[0015] S5. Retrain the subject classification model based on the targeted augmentation sample pool. When the performance of the subject classification model reaches the preset target, the iteration terminates and the final subject classification model is output.

[0016] Furthermore, in step S1, the preset rules include one or more of the following: URL domain name filtering, keyword matching, metadata filtering, classification system mapping, and large model-assisted filtering.

[0017] Furthermore, in step S2, the text classification algorithm includes at least one of FastText, TextCNN, BiLSTM, BERT, and their lightweight variants.

[0018] Furthermore, in step S3, the three-stage funnel filtering and dynamic priority evaluation mechanism includes:

[0019] The first level is a rapid sorting based on the confidence of the initial subject classification model, which divides the original dataset into high-confidence positive candidate samples, high-confidence negative candidate samples, and fuzzy samples according to a preset threshold.

[0020] The second level is quality filtering and classification. Based on the preset quality filtering rules, samples that do not meet the quality standards are removed to obtain a candidate sample pool. Then, the text quality level classifier is used to classify the samples in the candidate sample pool into high, medium and low quality, and calculate the distribution of high, medium and low quality ratios in the candidate sample pool.

[0021] The third level is a multi-dimensional intelligent priority assessment, which calculates the comprehensive priority score of the samples based on diversity factors, uncertainty factors, and quality distribution adjustment factors, and sorts the samples in descending order according to the comprehensive priority score.

[0022] Furthermore, the third level, which is a multi-dimensional intelligent prioritization assessment, includes the following sub-steps:

[0023] (1) Label each sample in the candidate sample pool with course tags, obtain the coarse-grained course tag set in the candidate sample pool, and calculate the coverage and relative coverage of each course tag in the candidate sample pool;

[0024] (2) The relative coverage of each course label corresponding to each sample is weighted and summed to obtain the diversity factor of each sample in the candidate sample pool;

[0025] (3) Calculate the uncertainty factor of the sample based on the predicted score of each sample according to the subject classification model;

[0026] (4) Calculate the sample quality distribution adjustment factor based on the distribution of high, medium and low quality ratios in the candidate sample pool and the quality level of each sample;

[0027] (5) The diversity factor, uncertainty factor and quality distribution adjustment factor of each sample are weighted and summed to obtain the comprehensive priority score;

[0028] (6) Sort the samples in descending order based on the comprehensive priority score.

[0029] Further, step S4 includes the following sub-steps:

[0030] S4.1 Collect positive and negative samples predicted by the subject classification model in the evaluation dataset, select false positive sample sets and false negative sample sets according to the true subject labels of the samples, and input them into the large language model respectively. Use structured prompt words to perform root cause analysis and output the misclassification type and confused subject for each sample. The evaluation dataset is constructed from data sources other than the original dataset source and is labeled by manual or large model to ensure that it covers text samples from multiple subjects.

[0031] S4.2 Count the number of misclassified samples of each type in the false positive sample set and the false negative sample set respectively, and use Laplace smoothing to calculate the misclassification ratio of each misclassification type in the predicted positive and negative samples respectively;

[0032] S4.3 Based on a preset ratio threshold, determine whether the misjudgment ratio of each misjudgment type reaches the condition that targeted enhancement is required;

[0033] S4.4 Filter out the corresponding types of samples that need to be enhanced from the original dataset, and further filter out the positive samples and negative samples for targeted enhancement, and merge them to generate a targeted enhancement sample pool.

[0034] Furthermore, in step S4.1, the structured prompt word template is as follows:

[0035] The first template is used to analyze texts that are misclassified as positive samples by the model, and to identify the misclassification type as CONF-confusing subject or NOISE-low-quality text interference. If the type is CONF, the name of the confusing subject is identified.

[0036] The second template is used to analyze texts that are misclassified as negative samples by the model. It identifies the reasons for misclassification as CONF-confusing subjects or NOISE-low-quality text interference. If the type is CONF, it identifies the name of the confusing subject.

[0037] Secondly, the present invention provides an iterative optimization system for a subject classification model in a cold start scenario, comprising:

[0038] The initial sample pool building block is used to quickly obtain an initial dataset that can be used to start training.

[0039] The sample pool management and update module is used to manage, update, and control the capacity of the text in the sample pool.

[0040] The model training module uses an efficient text classification algorithm to train a subject classification model based on the current version's training sample pool.

[0041] The training sample iterative multi-dimensional evaluation and diversified screening module intelligently selects new training samples that can maximize the model's discrimination ability and knowledge coverage;

[0042] The automatic misjudgment analysis and targeted data augmentation module, based on the root cause analysis capability of the large language model, automatically diagnoses the misjudgment type and cause of the classification model, and generates targeted augmented positive and negative samples for misjudged labels whose proportion exceeds a preset threshold, updates the training sample pool and retrains the model, and repeats the process multiple times until the proportion of all misjudged labels is lower than the preset threshold.

[0043] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned iterative optimization method for the subject classification model in a cold start scenario.

[0044] Fourthly, the present invention provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the iterative optimization method for the subject classification model in the cold start scenario.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] 1. Dual optimization granularity: "Major version iteration" rapidly expands the scale and diversity of data, while "minor version iteration" accurately fixes model defects, with efficiency significantly higher than the single iteration mode.

[0047] 2. Achieve intelligent and diversified subject data construction: Innovatively introduce the curriculum system as a screening guide, and through a dynamic priority scoring algorithm, optimize the knowledge coverage, decision boundary clarity and quality distribution balance of the dataset, so that the data construction process changes from "passive screening" to "active design".

[0048] 3. Improve overall model performance: The final trained model not only performs well on mainstream samples, but also shows stronger discrimination ability on niche subject subfields and texts of different quality, significantly improving the model's practicality and robustness.

[0049] 4. Forming a fully automated optimization closed loop: The entire system has self-diagnosis and self-optimization capabilities, which greatly reduces the workload of manually analyzing model defects and designing data augmentation strategies, and achieves efficient automated iteration. Attached Figure Description

[0050] Figure 1 This is a flowchart of the iterative optimization method described in this invention.

[0051] Figure 2 This is a flowchart of the iterative multidimensional evaluation and diversified screening of training samples in this invention.

[0052] Figure 3 This is a flowchart of the automatic misjudgment analysis and targeted data augmentation of the present invention.

[0053] Figure 4 The flowchart shows an iterative optimization system for a subject classification model in a cold start scenario provided by this invention. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] The symbols used in the embodiments of this invention are defined as follows:

[0056] 1. Subject Set: Suppose there are I subjects in total, the subject set is represented as: ;

[0057] 2. Define the version number; the major version number expression is: The minor version number expression is ;

[0058] 3. Define different versions of the training sample pool: The positive training sample pool for subject i in the m-th major version and the n-th minor version is... The negative sample pool is ;

[0059] 4. Define the massive text dataset as DATA.

[0060] 5. Define the evaluation text dataset as follows: The corresponding subject labels are recorded as follows: .

[0061] In one embodiment of this invention, I=4, representing four disciplines: chemistry, computer science, biology, and architecture, numbered 1, 2, 3, and 4 respectively. The text dataset DATA contains 1 million webpage text samples, and the evaluation text dataset contains 40,000 samples. The iterative optimization method of this invention is used to construct classification models for these four disciplines.

[0062] This invention provides an iterative optimization method for a subject classification model in a cold start scenario, the process of which is as follows: Figure 1 As shown, it includes the following steps:

[0063] Step 1: Initial sample pool construction and initial model training

[0064] Quickly obtain an initial dataset suitable for training and build an initial model. For each subject... :

[0065] (1.1) Extracting subject matter from the text dataset DATA based on pre-defined simple rules (such as URL domain filtering and keyword matching). Related Articles of text constitute a discipline The initial positive sample pool is expressed as follows:

[0066]

[0067] in, This indicates the capacity (number of samples) of the set.

[0068] In one embodiment of the present invention, subject-specific keywords are used to match text datasets to extract subject-related web page text, thereby constructing initial positive samples for the subject. ).

[0069] (1.2) From similar data in the other I-1 disciplines, approximately [number] samples are randomly selected from each. Texts, combined to form disciplines The initial negative sample pool is expressed as follows:

[0070]

[0071] in, For the discipline (Non-discipline within the subject set) The initial positive sample pool (other disciplines) represents the pool of samples selected according to rules that are relevant to the discipline. Related text collections.

[0072] Step 2: Model Training and Evaluation

[0073] (2.1) Data format conversion: Convert the positive sample pool of subject i in step one at the m-th major version and the n-th minor version. and negative sample pool The dataset is merged to construct a complete labeled training set. The merged data is then converted to the input format required by the FastText algorithm, assigning a corresponding category prefix to each sample (positive samples are formatted as "__label__1 text content", and negative samples as "__label__0 text content") to clearly define the category of each sample. The complete dataset after format conversion is denoted as […]. Each sample carries a clear binary classification label (1 indicates belonging to a subject). 0 indicates that it does not belong to any subject. ).

[0074] (2.2) Model training: The training set constructed based on the above steps The FastText text classification algorithm was used for model training. This model learns the bag-of-words features and n-gram features of the text through a neural network, using cross-entropy loss as the optimization objective. By continuously adjusting the internal model parameters, the average loss value across all training samples is minimized. This optimization process is completed iteratively using backpropagation and gradient descent algorithms. The final trained model is a binary classification model. The training process can be formally represented as:

[0075]

[0076] in, The training process for the FastText model is essentially about finding the model parameters that minimize the cross-entropy loss function.

[0077] After training, the resulting model That is, the discipline used at the current stage This is a classification model. The model can accept any input text, determine and output whether the text belongs to a specific subject. The binary classification label is generated, and a score between [0,1] is output, indicating that the text belongs to the subject. The confidence level.

[0078] (2.3) Model evaluation: Based on the current model For the evaluation set (including text) and subject labels Inference and performance evaluation are performed.

[0079] The specific steps are as follows: (Targeting the subject) ,Will Each item in the list is converted into a binary label. , where 1 indicates that the sample belongs to a subject 0 indicates that it does not belong. (Using the...) Inference is performed to obtain the predicted score for each sample. Based on predicted scores With real labels The area under the receiver operating characteristic curve (AUROC) is calculated. A higher AUROC value indicates that the model has better classification performance.

[0080] The goal of model iteration is to ensure that AUROC is greater than or equal to a preset threshold. If the current model's performance meets the preset target, then output the current model directly as the final model; otherwise, proceed to step three for a major version iteration.

[0081] Step 3: Major Version Iteration Cycle (Large-Scale Data Expansion Phase)

[0082] In major version iteration cycles, the current model is used. The data is used for inference, multi-dimensional filtering, and annotation to massively expand the training samples. Its core process is as follows: Figure 2 The diagram shows the "three-stage funnel filtration and dynamic priority evaluation" scheme.

[0083] The specific steps are as follows:

[0084] (3.1) Acquisition of Course Tagging System

[0085] The list of course tags for a subject is obtained based on sources such as the teaching syllabus. The expression is:

[0086]

[0087] in, Let i be the k-th course label (e.g., "data structure") corresponding to subject i. This represents the total number of course tags for subject i.

[0088] At the same time, an importance weight is set for each course tag, and the sum of the weights of all course tags corresponding to subject i should be 1. If they are set to the same weight, the weights of all course tags corresponding to subject i will be... That is, assigning the same weight to each course label under subject i.

[0089] In one embodiment of the present invention, K1=83, K2=73, K3=78, and K4=95. For subject 1, the weights of all course tags are... For Subject 2, the weight of all course tags is: For subject 3, the weight of all course tags is: For subject 4, the weight of all course tags is: .

[0090] (3.2) Three-stage funnel-shaped progressive screening

[0091] (3.2.1) Level 1: Fast splitting based on model confidence

[0092] (3.2.1.1) Use the current classification model Perform batch inference on the samples in DATA to obtain the score of each sample X belonging to subject i. .

[0093] (3.2.1.2) Set the number of samples in the core candidate pool C1 to be... .

[0094] (3.2.1.3) Divert traffic according to a preset threshold: If If it is a "high-confidence negative candidate sample", it is classified as a "high-confidence negative candidate sample" and added to the Neg_Candidate pool; if If the sample is positive, it is classified as a "high-confidence positive candidate sample" and a sample is drawn from it. Each sample is added to candidate pool C1 for the next stage of processing. The value is between [0,1]; if If the sample is categorized as a "fuzzy sample", then sampling will be performed from it. Each sample is added to the core candidate pool C1 for the next stage of processing.

[0095] Typically, the proportion of "fuzzy samples" in candidate pool C1 is greater than the proportion of "high-confidence positive candidate samples," that is... This ensures that more samples with ambiguous decision boundaries in the current model are processed to clarify their subject attributes and added to the training positive or negative samples, which helps to improve the classification judgment ability of the model after iteration.

[0096] The core candidate pool C1 includes A number of "high-confidence positive candidate samples" and A "fuzzy sample".

[0097] In one embodiment of the present invention, a threshold is set. , , During the first major version iteration During the second major version iteration .

[0098] (3.2.2) Second stage: quality filtration and classification

[0099] (3.2.2.1) For the core candidate pool C1, unqualified samples are eliminated according to the screening conditions composed of a set of quality filtering rules, and the samples that pass the screening constitute the core candidate pool C2.

[0100] In one embodiment of the present invention, four filtering rules are set, specifically: filtering based on the length of English characters, filtering based on the proportion of repeated lines, filtering based on the proportion of repeated characters in a line, and filtering based on the proportion of short lines.

[0101] The filtering thresholds are set as follows: English character length < 900, line repetition ratio > 0.4, line repetition character ratio > 0.4, and short line ratio > 0.9.

[0102] (3.2.2.2) The quality level of each sample x in the core candidate pool C2 is calculated and evaluated, denoted as:

[0103]

[0104] in, This is a text quality level classifier. It takes text as input and calculates the output quality level label g(x), which takes three quality levels: High, Medium, and Low.

[0105] In one embodiment of the present invention, the text quality classifier used is DeBERTa-V3 Scorer, which is a text quality assessment tool built based on the DeBERTa-V3 pre-trained model.

[0106] (3.2.3) Level 3: Intelligent Priority Assessment Based on Multi-Dimensionality

[0107] For each sample x in the core candidate pool C2, the following calculation is performed:

[0108] (3.2.3.1) Quick course tagging.

[0109] Methods such as keyword matching, fine-tuning inference with a multi-label course classifier, or approximate matching based on semantic embedding are used to assign q coarse-grained course labels to the samples. ,Right now .

[0110] In one embodiment of the present invention, an approximate matching method based on semantic embedding is used for course tagging.

[0111] First, prepare the conceptual description text for the course tag l. For example, the text describing the concept of Newton's second law is: "Newton's second law is the core cornerstone of classical mechanics. It reveals the causal relationship of changes in the state of motion of an object."

[0112] Then, use the embedding model to generate course knowledge vectors:

[0113]

[0114] One embodiment of the present invention specifically employs a BGE embedding model. The same embedding model is used to calculate the text vector of sample x. Calculate the knowledge vector and vector for each course. The cosine similarity is used to select the q vectors with the highest cosine similarity; the corresponding course tags are the coarse-grained course tags for sample x. In this embodiment, q=5.

[0115] (3.2.3.2) Calculate the course coverage status in the core candidate pool.

[0116] Statistics on the coverage of each course tag in the current core candidate pool C2:

[0117] in, For a course tag, This indicates that the coarse-grained course tags in the core candidate pool C2 include... The number of samples, This represents the total number of samples in the core candidate pool C2.

[0118] Based on coverage The relative coverage rate is calculated as follows:

[0119]

[0120] in, This is a smoothing factor to prevent division by zero; The smaller the value, the scarcer it is. In this embodiment, Set it to 0.01.

[0121] (3.2.3.3) Calculate the diversity factor.

[0122] Based on relative coverage, the diversity factor of sample x is calculated by weighted summation of the coverage of all coarse-grained course tags, as shown in the following formula:

[0123]

[0124] in, Tag the course The preset importance weights.

[0125] If the course labels for sample x are mostly those with low coverage, then this sample will be selected with higher priority in subsequent steps. This strategy can improve the diversity of the overall sample pool and make up for knowledge gaps.

[0126] (3.2.3.4) Calculate the uncertainty factor.

[0127] Based on the predicted scores of the samples using the current subject classification model, the uncertainty factor of sample x is calculated using the following formula:

[0128]

[0129] in, Represents absolute value. This represents the predicted score of the subject classification model for sample x at the current stage.

[0130] From the formula, we can obtain that The function is in The maximum value is 1.

[0131] like A value around 0.5 indicates that the current subject classification model's decision boundary for sample x is unclear. A higher uncertainty factor can be set, giving that sample a higher priority for selection as a training sample in subsequent model iterations. This strategy can improve the ability to judge difficult samples and optimize the decision boundary.

[0132] (3.2.3.5) Calculate the mass distribution adjustment factor.

[0133] The proportions of each quality level in the current candidate pool are calculated, with the proportions of high quality, medium quality, and low quality denoted as follows: The target ratio of high-quality, medium-quality, and low-quality products is pre-set as follows: ,satisfy .

[0134] For sample X, calculate the quality distribution adjustment factor based on its quality level g(x). Its function is to dynamically adjust the selection bias of samples of different quality levels in the priority scoring, so as to actively approximate the preset target quality distribution. The quality distribution adjustment factor is substituted as a negative term into the subsequent comprehensive priority scoring. A positive value indicates a penalty (lower priority), a negative value indicates a reward (higher priority), and a value of zero indicates no effect. The formula is as follows:

[0135]

[0136] In the formula, g(x) represents the quality level of sample x, which includes three levels: High, Medium, and Low. This is an indicator function; it takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. M 1 and M 2 is a preset positive constant, representing the penalty for medium-quality and low-quality samples.

[0137] The calculation logic for the quality distribution adjustment factor is as follows: if the proportion of high-quality samples in the current training set... The target has been exceeded. (Right now If a new high-quality sample is found to be in violation, a penalty is imposed, with the penalty increasing as the violation rate increases; the actual proportion of high-quality samples in this category... The target ratio has been reached or exceeded. Only then is a fixed penalty value applied to new medium-quality samples. M 1; The calculation method for low-quality samples is similar to that for medium-quality samples.

[0138] The above design helps to suppress the over-collection of high-quality samples and avoid data distribution imbalance. At the same time, it strictly controls the proportion of medium-quality and low-quality samples within the target threshold to prevent them from interfering too much with model training. Before the proportion reaches the target, medium-quality and low-quality samples are not penalized.

[0139] Samples with higher quality distribution adjustment factors have a higher selection priority in subsequent steps. In this way, the text quality composition of training samples is actively managed, comprehensively covering high, medium, and low quality texts.

[0140] In one embodiment of the present invention, a configuration is provided. , , , , .

[0141] (3.2.3.6) Calculate the overall priority score using the following formula:

[0142]

[0143] Wherein, α, β, and γ are adjustable balance coefficients, with values ​​ranging from [0,1], satisfying α+β+γ=1.

[0144] In one embodiment of the present invention, α=0.25, β=0.65, and γ=0.1.

[0145] (3.3) Selection and labeling of high-value samples

[0146] 1. Apply the calculated comprehensive priority score to all samples in the core candidate pool C2. Sort in descending order.

[0147] 2. Select the top rankings The samples constitute a high-value candidate set. .

[0148] In one embodiment of the present invention, during the first major version iteration During the second major version iteration .

[0149] 3. Use large language models or manual annotation to assist in the process. The samples were finely labeled to determine three types of information for each sample:

[0150] a) Real subject labels (positive / negative samples) are used to determine whether a sample belongs to the target subject i;

[0151] b) Precise course labels ;

[0152] c) Quality grade confirmation.

[0153] (3.4) Results output and data update preparation

[0154] 1. Output the labeled positive samples as the new positive sample set. .

[0155] 2. Denote the annotated negative sample set as `annot_neg`. If the number of samples in the annotated negative sample set is greater than or equal to the number of samples in the annotated positive sample set, then output the annotated negative sample set `annot_neg` as the newly added negative sample set. If the number of samples in the negative sample set is less than the number of samples in the positive sample set, then from the Neg_Candidate pool obtained from the first-level split, remove text samples that are completely duplicates of the annot_neg negative sample set, and then randomly select... After each sample, the new negative sample set is combined with the labeled negative sample set (annot_neg) to form the output negative sample set. In the above expression, for The number of samples, for The number of samples.

[0156] (3.5) Sample pool update

[0157] The newly added samples are merged with historical samples to generate a new sample pool. The positive and negative sample pools are updated using the following formula:

[0158]

[0159] in, The operations include precise text deduplication and sorting by blending ratio. Retain some old samples and ensure the total capacity does not exceed the maximum capacity. ,Proportion The value range is [0,1].

[0160] For the positive sample pool, specifically: First, from Remove from and Completely duplicated text samples yield a sample set. Next, according to the mixing ratio Retain some old samples, i.e., from the old sample pool Randomly selected from One sample, and a new sample Combine the samples; if the number of samples after combination does not exceed the maximum capacity. Then output all combined samples. If the number of combined samples exceeds the maximum capacity... Then, randomly select from the combined sample pool. For the negative sample pool, follow the same steps.

[0161] In one embodiment of the present invention, the mixing ratio , .

[0162] (3.6) Trigger a new round of training

[0163] Using the method described in step two, the updated sample pools Pos(i, m+1, 1) and Neg(i, m+1, 1) are used to train the model model(i, m+1, 1), completing one "major version iteration". If the current model performance meets the preset target, the current model is directly output as the final model; otherwise, proceed to step four for a "minor version iteration".

[0164] Step 4: Minor version iteration cycle (fine-grained performance optimization phase)

[0165] For the current version model After filtering based on the evaluation set, false positive and false negative sample sets are constructed. A large language model is then used for misjudgment analysis to identify the main misjudgment types, guiding small-scale targeted augmentation and fine-tuning of the training samples, thereby optimizing the model's performance. The specific process is as follows: Figure 3 As shown. Specifically, it includes the following sub-steps:

[0166] (4.1) Sample Collection and Analysis: To ensure the objectivity and generalizability of the evaluation results, the evaluation set was constructed from data sources other than the training dataset, and its coverage of text samples across multiple disciplines was ensured through manual or large-model annotation. The evaluation set (including text samples) was constructed from data sources other than the training dataset. and subject labels Randomly selected from ) Each subject tag is a subject samples and The subject tag is not a subject. The samples constitute the discipline Evaluation set. Using the current model model(i, m, n), inference and prediction are performed on the evaluation set samples for subject i. Samples with positive predictions are collected to form the sample set. Collect samples that are predicted to be negative to form a sample set. In the above expression, Represents the sample set The k-th text sample, Represents the sample set The k-th text sample.

[0167] Based on subject labels Judgment, from the sample set The selection process identifies those that do not actually belong to any discipline. The samples constitute the false positive sample set From the sample set Selecting those that actually belong to the discipline The samples constitute the false negative sample set .

[0168] In one embodiment of the present invention, The training data comes from web page texts from Common Crawl (CC) in 2023 and earlier, while the evaluation data is randomly selected from web page texts from CC 2024 / 2025. A large language model is used to annotate the data with subject attributes one by one until 10,000 samples for each subject are collected. No quality level is limited during the selection of evaluation data to reflect the diversity of text quality in real-world application scenarios. Through this method, the evaluation dataset is constructed. ,in For text content, Subject-specific labels were used. The evaluation dataset consists of 40,000 texts covering four subjects, with a balanced distribution of samples across subjects and diverse text quality.

[0169] (4.2) Automatic analysis of large model: Using structured prompt words, the above false positive sample set and false negative sample set are respectively input into the large language model (LLM) for root cause analysis.

[0170] In one embodiment of the present invention, the large language model used is QWen3-32B.

[0171] (4.2.1) For the subject Define two prompt word template functions for false positive analysis:

[0172] To address the false positive misclassification, the corresponding model misclassifies "samples that do not belong to subject i" as samples that do belong to subject i. The analysis prompt word template used is as follows:

[0173] "You are a machine learning expert. Please analyze the following..." The subject classification model determines that it belongs to { Text samples that do not actually belong to a specific subject category are misclassified by the model. It is necessary to determine the primary reason for the misclassification:

[0174] a. Main types of misjudgment (choose one): - CONF: Easily confused disciplines—The text belongs to other disciplines, but is confused due to differences in discipline { The topics overlapped, and the model misjudged them as belonging to different disciplines. NOISE: Low-Quality Textual Interference – Text and Subject Matter The whole is unrelated, but contains scattered disciplines. a) Terminology or surface features leading to model misjudgment. b) If the type is CONF, please select the specific confused subject name from the subject list. The complete list of subject names is {NAME_list}. c) Brief explanation: [Explain the reason in one sentence] Please output in the following JSON format: { "error_type": "CONF", "confused_subject": "Computer Science", / / "explanation": "..." is only needed when error_type is CONF.

[0175] Text content to be analyzed: {sample content}

[0176] In the above prompt word template, Let NAME_i be the subject name corresponding to subject i, and let NAME_list be a list of all subject names.

[0177] To address the false negative case misclassification, the corresponding model incorrectly classifies "samples belonging to subject i" as not belonging to subject i. The analysis prompt word template used is as follows:

[0178] "You are a machine learning expert. Please analyze the following..." The subject classification model determines that it does not belong to { The text sample in question was misclassified by the model, even though it belonged to a different subject area. It's necessary to determine the primary reason for this misclassification:

[0179] a. Main types of misjudgment (select one): - CONF: Easily confused disciplines—the discipline to which the text belongs { However, due to overlap with other subject topics, it was misjudged by the model as not belonging to the subject. NOISE: Low-Quality Textual Interference – Text and Subject Matter The content was substantially relevant, but the model failed to recognize it because the expression was non-academic (such as popular science, event promotion, product introduction), the language quality was low, or it contained non-academic content.

[0180] b. If the type is CONF, please select the specific subject name to be obfuscated from the subject list. The complete subject list is {NAME_list}.

[0181] c. Brief explanation: [Explain the reason in one sentence] Please output in the following JSON format: { "error_type": "CONF", "confused_subject": "Architecture", / / "explanation": "..." is only required when error_type is CONF.

[0182] Text content to be analyzed: {sample content}

[0183] (4.2.2) Processing of analysis results

[0184] After analysis using a large language model, the JSON output of each sample is parsed to obtain structured misclassification type labels.

[0185] For false positive samples Resulting in the misjudgment type .like This leads to specific confusing disciplines. .

[0186] For false negative samples To obtain the misjudgment type .like This leads to specific confusing disciplines. .

[0187] (4.3) Statistical analysis of misjudgments

[0188] Based on the analysis results of the large model, various misjudgments are statistically analyzed to provide a quantitative basis for subsequent targeted augmentation.

[0189] (4.3.1) Calculation of the false positive rate

[0190] For each confusing subject Computation is a discipline However, it was mistakenly identified as a subject. The proportion of false positives out of all models that predict as positive:

[0191]

[0192] in, This indicates that it belongs to the discipline Mistakenly identified as a subject The number of false positive samples. For models in disciplines The number of samples predicted as positive on the evaluation set, i.e. . is the Laplace smoothing factor, which takes a positive constant value.

[0193] In one embodiment of the present invention, .

[0194] Calculate the proportion of false positives caused by low-quality text interference out of all positive samples predicted by the model:

[0195]

[0196] in, This represents the number of false positive samples caused by low-quality text interference.

[0197] (4.3.2) Calculation of the false negative example misjudgment ratio

[0198] For each confusing subject Computation is a discipline However, due to its relation to the discipline The proportion of samples misclassified as negative out of all samples predicted as negative by the model:

[0199]

[0200] in, This belongs to the discipline Because of the discipline The number of samples misclassified as negative is confused. Let be the total number of samples that the model predicts as negative on the evaluation set for subject i, i.e. .

[0201] Calculate the proportion of false negatives caused by low-quality text interference out of all negative samples predicted by the model:

[0202]

[0203] in, This represents the number of false negative samples caused by low-quality text interference.

[0204] (4.3.3) Proportional threshold judgment and enhanced triggering

[0205] Set a threshold for the proportion of false positives The value ranges from [0,1]. For each type of misclassification, determine whether targeted augmentation is needed.

[0206] To address false positives, the following enhanced trigger signals are derived based on the proportion of each false positive:

[0207]

[0208] in This is an indicator function; a value of 1 indicates that "subject" needs to be selected. Mistaken for a subject This type is enhanced; a value of 0 indicates that no enhancement is needed. This is an indicator function; a value of 1 indicates that enhancement is needed for false positives caused by low-quality text interference.

[0209] To address false negatives, the following enhanced trigger signals are derived based on the proportion of each false negative:

[0210]

[0211] in This is an indicator function; a value of 1 indicates that "subject" needs to be selected. Because of the discipline Enhance the type of "confusion that is misjudged as negative". This is an indicator function; a value of 1 indicates that enhancement is needed for false negatives caused by low-quality text interference.

[0212] If the trigger function for all false positive types is 0, that is:

[0213]

[0214] This indicates that the proportion of all misjudgment types has fallen below the threshold. The model performance has reached expectations, so the minor version iteration process is terminated, and the current discipline is output. The classification model.

[0215] In one embodiment of the present invention, .

[0216] (4.4) Targeted sample screening:

[0217] For each type of misjudgment that is triggered, a specific number of samples are selected from the corresponding data source and added to the training positive sample pool or negative sample pool to achieve sample augmentation.

[0218] (4.4.1) False positives enhance sample screening

[0219] For each triggered confusing subject (Right now ), then from the discipline dataset In the middle, random selection Samples, including subject The negative sample pool, denoted as . .

[0220] If a false positive is triggered by low-quality text ( Then from the garbage text library Random selection Samples, including subject The negative sample pool, denoted as . .in, This is a pre-built junk text library containing garbled text, advertisements, and completely irrelevant content that has been filtered out in the second-level quality filter.

[0221] In one embodiment of the present invention, , .

[0222] (4.4.2) False negatives enhance sample screening

[0223] For each triggered confusing subject (Right now ), then from the discipline dataset In the middle, the method of large language model annotation is used to filter subjects that are involved simultaneously. Elements, but the core belongs to the discipline samples Article, add discipline The positive sample pool, denoted as . .

[0224] If a false negative instance of low-quality text is triggered ( Then, from low-quality related text libraries Random selection Samples, including subject The positive sample pool, denoted as . .in, This is a low-quality related text library stored by subject, containing texts identified as "low-quality" in the quality grading and confirmed by the large language model annotation, and related to the subject. Related text.

[0225] In one embodiment of the present invention, , .

[0226] (4.4.3) Output targeted augmentation sample set

[0227] Output the targeted augmentation positive sample set for the next round of sample pool update. and targeted augmentation of negative sample sets .

[0228]

[0229] (4.5) Sample pool fine-tuning and update:

[0230] The selected targeted augmentation sample set is merged with the existing sample pool to form the training data for the next version. The updated positive and negative sample pools are as follows:

[0231]

[0232] The sample pool is fine-tuned and updated using the method described in step (3.5).

[0233] In one embodiment of the present invention, the mixing ratio , .

[0234] (4.6) Triggering fine-tuning training:

[0235] Using the methods described in sub-steps (2.1) and (2.2) of step two, the optimized model model(i, m, n+1) is trained using the finely tuned sample pool, completing a "small version iteration".

[0236] (4.7) Multiple rounds of fine-tuning:

[0237] Repeat steps (4.1)-(4.6) again, updating the positive and negative sample pools to Pos(i,m,n+2) and Neg(i,m,n+2), respectively. Repeat this process multiple times until the proportion of all misclassified types is below the threshold. End minor version iteration, output positive and negative sample pools. , It also outputs a classification model trained based on the updated positive and negative sample pools. ,in For the first The final minor version number corresponding to each major version.

[0238] Step 5: Iteration Termination

[0239] Repeat steps three and four to continuously update the positive and negative sample pools, and train the classification model based on the updated samples. After multiple iterations, the iteration terminates when the model's performance on the evaluation set reaches the preset target, and the final positive sample pool is output. Negative sample pool Classification model Where M is the final major version number. This is the final minor version number corresponding to the Mth major version.

[0240] In one embodiment of the present invention, the initial model performance evaluation results are as follows: the AUROC values ​​of the classification models for chemistry, computer science, biology, and architecture are 0.43, 0.42, 0.68, and 0.50, respectively. Target value Set to 0.75, that is This is the termination condition for the classification model iteration. Since the AUROC classification models for all subjects failed to meet the preset target, step three was executed for major version optimization.

[0241] After the first major version optimization, the AUROC values ​​of the classification models for Chemistry, Computer Science, Biology, and Architecture are 0.80, 0.51, 0.78, and 0.55, respectively. The AUROC values ​​of the Chemistry and Biology classification models have reached the preset targets, and the models are directly output; the other two disciplines undergo minor version optimization in step four.

[0242] During the first minor version optimization, no significant misclassification types were found in the Computer Science discipline, while the Architecture discipline found that the Biology discipline was a significantly easily confused discipline. After manual review and summary, it was concluded that this was likely because keywords such as "structure" in the Biology discipline were easily confused with related expressions of "structure" in the Architecture discipline. Therefore, following the method of this invention, Biology samples were selected and added to the training negative samples for the Architecture discipline. The optimized Architecture classification model achieved an AUROC value of 0.82, reaching the preset target, and the model was directly output. For the Computer Science discipline, step three was repeated for a major version iteration.

[0243] After the second major version optimization, the AUROC value of the computer science classification model is 0.89, reaching the preset target, and the model is directly output.

[0244] The cold-start scenario subject classification model iterative optimization method provided by this invention adopts a "major version-minor version" dual-loop iterative architecture: clearly distinguishing between large-scale data augmentation and small-scale precise enhancement, forming an efficient hierarchical optimization strategy. The method integrates multi-dimensional evaluation and screening algorithms of the curriculum system: in the "major version iteration," a three-level funnel filtering and dynamic priority scoring are used to intelligently screen samples from massive amounts of data that can simultaneously promote knowledge coverage, boundary exploration, and quality balance. A large model-driven root cause analysis and quantitative enhancement mechanism is provided: in the "minor version iteration," a large language model is used to automatically analyze model errors, statistically analyze the proportion of each type of misjudgment in the model's predicted samples, and trigger targeted data augmentation for serious defect types whose proportions exceed a preset threshold, achieving precise and automated data augmentation.

[0245] This invention also provides a "dual-loop iterative" subject classification model optimization system. This system alternates between "large version iteration" (coarse expansion based on multidimensional evaluation and diversified screening) and "small version iteration" (fine-tuning based on misjudgment analysis), ultimately achieving co-evolution of the model and training data. Specifically, as follows... Figure 4 As shown.

[0246] The model optimization system specifically includes the following modules:

[0247] The initial sample pool building block is used to quickly obtain an initial dataset that can be used to start training.

[0248] The sample pool management and update module is used to manage, update, and control the capacity of the text in the sample pool.

[0249] The model training module uses an efficient text classification algorithm to train a subject classification model based on the current version's training sample pool.

[0250] The training sample iterative multi-dimensional evaluation and diversified screening module intelligently selects new training samples that can maximize the model's discrimination ability and knowledge coverage, thus completing large-scale data augmentation.

[0251] The automatic misclassification analysis and targeted data augmentation module leverages the root cause analysis capabilities of a large language model to automatically diagnose the types and causes of misclassifications in the classification model. This module analyzes false positives and false negatives using structured prompt word templates, calculates the proportion of each type of misclassification, and generates targeted augmented positive and negative samples for misclassified labels whose proportions exceed a preset threshold. It then updates the training sample pool and retrains the model, repeating this process multiple times until the proportion of all misclassified labels is below the preset threshold, thus achieving intelligent augmentation of weak links in the training sample pool.

[0252] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the iterative optimization method for the subject classification model in the cold start scenario described in the present invention.

[0253] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the cold start scenario subject classification model iterative optimization method of the present invention.

[0254] The exemplary embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the exemplary embodiments of this application should be included within the protection scope of this application.

Claims

1. An iterative optimization method for a subject classification model in a cold start scenario, characterized in that, Includes the following steps: S1. For the target subject, extract initial positive samples from the acquired original dataset based on preset rules, and extract initial negative samples from data of other subjects, and merge them to form an initial training sample pool; S2. Based on the initial training sample pool, an initial subject classification model is obtained by training a text classification algorithm; S3. Using the initial subject classification model, a three-level funnel filtering and dynamic priority evaluation mechanism is employed to obtain high-value samples from the original dataset. Fine-grained annotation is then performed using a large language model or manual methods. The newly annotated samples are merged with the historical training sample pool, deduplicated, and the training sample pool is updated. The mechanism includes: The first level divides the original dataset into high-confidence positive candidate samples, high-confidence negative candidate samples, and fuzzy samples according to a preset threshold, and samples are drawn from the high-confidence positive candidate samples and fuzzy samples to form a sample pool C1; The second stage involves removing substandard samples from sample pool C1 based on preset quality filtering rules to obtain sample pool C2. Then, a text quality level classifier is used to classify the samples in the pool as high, medium, and low quality, and the distribution of high, medium, and low quality ratios in the pool is calculated. The third level calculates the comprehensive priority score of each sample in sample pool C2 based on the diversity factor, uncertainty factor and quality distribution adjustment factor, sorts the samples in descending order according to the comprehensive priority score, and selects a preset number of samples with the highest ranking as high-value samples. S4. Retrain the subject classification model based on the updated training sample pool, collect the positive and negative samples predicted by the subject classification model in the evaluation dataset, select false positive and false negative sample sets according to the real subject labels of the evaluation samples, and input them into the large language model respectively. Perform root cause analysis using structured prompt words, and output the misclassification type and confused subject for each sample. The evaluation dataset is constructed from data sources other than the original dataset source, and is labeled manually or by the large model to ensure that it covers text samples from multiple subjects. The structured prompt word template is as follows: The first template is used to analyze texts that are misclassified as positive samples by the model, and to identify the misclassification type as CONF-confusing subject or NOISE-low-quality text interference. If the type is CONF, the name of the confusing subject is identified. The second template is used to analyze texts that are misclassified as negative samples by the model. It identifies the reasons for misclassification as CONF-confusing subjects or NOISE-low-quality text interference. If the type is CONF, it identifies the name of the confusing subject. The number of misclassified samples of each type was counted in the false positive sample set and the false negative sample set respectively. Laplace smoothing was used to calculate the misclassification ratio of each misclassification type in the predicted positive and negative samples respectively. Based on a preset ratio threshold, determine whether the misjudgment ratio of each misjudgment type reaches the condition that targeted enhancement is required; Select the corresponding types of samples that need to be enhanced from the original dataset, and select positive and negative samples for targeted enhancement, and merge them to generate a targeted enhancement sample pool; S5. Retrain the subject classification model based on the targeted augmentation sample pool. When the performance of the subject classification model reaches the preset target, the iteration terminates and the final subject classification model is output.

2. The iterative optimization method for a subject classification model in a cold start scenario according to claim 1, characterized in that, In step S1, the preset rules include one or more of the following: URL domain name filtering, keyword matching, metadata filtering, classification system mapping, and large model-assisted filtering.

3. The iterative optimization method for a subject classification model in a cold start scenario according to claim 1, characterized in that, In step S2, the text classification algorithm includes at least one of FastText, TextCNN, BiLSTM, BERT, and their lightweight variants.

4. The iterative optimization method for a subject classification model in a cold start scenario according to claim 1, characterized in that, The third level, which is a multi-dimensional intelligent prioritization assessment, includes the following sub-steps: (1) Label each sample in the sample pool C2 with course tags, obtain the coarse-grained course tag set in the sample pool C2, and calculate the coverage and relative coverage of each course tag in the sample pool C2. (2) The relative coverage of each course label corresponding to each sample is weighted and summed to obtain the diversity factor of each sample in sample pool C2; (3) Calculate the uncertainty factor of the sample based on the predicted score of each sample according to the subject classification model; (4) Calculate the sample quality distribution adjustment factor based on the distribution of high, medium and low quality ratios in sample pool C2 and the quality level of each sample; (5) The diversity factor, uncertainty factor and quality distribution adjustment factor of each sample are weighted and summed to obtain the comprehensive priority score; (6) Sort the samples in descending order based on the comprehensive priority score.

5. An iterative optimization system for a cold-start scenario subject classification model implementing the method of any one of claims 1-4, characterized in that, include: The initial training sample pool building module is used to quickly obtain an initial dataset that can be used to start training; The training sample pool management and update module is used to manage, update, and control the capacity of the text in the training sample pool. The model training module uses an efficient text classification algorithm to train a subject classification model based on the current version's training sample pool. The training sample iterative multi-dimensional evaluation and diversified screening module intelligently selects new training samples that can maximize the discrimination ability and knowledge coverage of the subject classification model. The automatic misjudgment analysis and targeted data augmentation module, based on the root cause analysis capability of the large language model, automatically diagnoses the misjudgment type and cause of the classification model, and generates targeted augmented positive and negative samples for misjudged labels whose proportion exceeds a preset threshold, updates the training sample pool and retrains the subject classification model, and repeats the process multiple times until the proportion of all misjudged labels is lower than the preset threshold.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the iterative optimization method for the subject classification model in the cold start scenario as described in any one of claims 1-4.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the cold start scenario subject classification model iterative optimization method as described in any one of claims 1-4.

Images