A hierarchical multi-label attribution method and system that integrates atomic rule-driven trusted features and knowledge distillation

Abstract

This invention discloses a hierarchical multi-label attribution method and system that integrates atomic rule-driven credible features and knowledge distillation, belonging to the field of natural language processing technology. First, this invention constructs an atomic rule base for weakly supervised text annotation. Then, it uses a large language model as a teacher model to correct and supplement the weak annotation results, extracting the probability distribution of soft labels and intermediate layer feature representations on each level of labels. Next, it evaluates the credibility of the teacher model's output, selecting a subset of credible soft labels and credible feature dimensions. Then, it constructs a student model with a hierarchical output structure, designs a joint loss function, and distills the student model for training. Finally, it deploys only the student model for inference, outputting hierarchical multi-label attribution results and key evidence fragments. This invention, through the combination of atomic rules and credible knowledge distillation, significantly reduces inference costs while improving the accuracy, stability, and interpretability of hierarchical multi-label attribution.

Description

Technical Field

[0001] This invention relates to the field of natural language processing and generation technology in artificial intelligence, and particularly to an attribution method for hierarchical multi-label classification of text. Specifically, it is a hierarchical multi-label attribution method that integrates atomic rule weakly supervised annotation, reliable feature screening and knowledge distillation techniques. Background Technology

[0002] In many real-world business scenarios, tagging systems exhibit a clear hierarchical structure: the upper level consists of coarse-grained parent category tags, while the lower level comprises more granular subcategory tags and even multi-level descendant tags. While hierarchical tagging systems can better reflect the business knowledge structure, they also present the following problems: a large and uneven number of tags, with very few long-tail tags; complex inclusion, parallel, and overlapping relationships exist between tags at different levels, easily leading to confusion; and business stakeholders typically require interpretable and accountable classification results, which a purely black-box model cannot adequately meet.

[0003] Early text classification methods were mostly based on sparse features such as TF-IDF and n-grams, combined with classifiers such as SVM, LR, and Naive Bayes. For hierarchical labels, a parent classifier is typically trained first to perform coarse classification of the text; then, several sub-class classifiers are trained under each parent class, adopting a top-down hierarchical decision-making process. These methods can achieve certain results in small-scale label systems and scenarios with relatively stable features, but their feature representation capabilities are limited, making it difficult to capture fine-grained differences at the semantic level. Furthermore, parent class errors can be amplified cascaded, leading to irreversible errors in downstream sub-class classification. Additionally, because multiple classifiers are trained independently, there is a lack of a unified optimization objective across different levels, limiting overall performance.

[0004] With the advent of word vectors and shallow neural networks, some works have used CNNs, RNNs, attention networks, etc. to encode text, and then connect multi-label or hierarchical softmax classification heads at the top layer. However, in scenarios with a large number of labels, deep layers, and extremely uneven class distribution, these methods still face problems such as overfitting, poor generalization, and difficulty in interpretation.

[0005] In recent years, large-scale pre-trained language models (such as BERT and RoBERTa) have made significant progress in text classification tasks. There are two typical approaches: one is to input text into a pre-trained model, use vectors as text representations, and then connect fully connected layers to complete the classification; the other is to use a "prompt + pre-trained model" approach, injecting label names or descriptions into the input sequence, allowing the model to output classification results in a "fill-in-the-blank" or "natural language understanding" manner. While these methods outperform traditional machine learning methods in terms of accuracy, they still have some problems in complex business environments. First, they are sensitive to long-tail labels and data noise. Large models tend to memorize noisy samples in the training data, and some seemingly useful but unstable features are amplified, leading to unstable classification results in new data or abnormal scenarios. More importantly, directly performing multi-level, multi-label fine-tuning on a large model requires significant computing and storage resources, which is not conducive to deployment in resource-constrained business systems.

[0006] To reduce the inference cost of large-scale pre-trained language models, knowledge distillation is widely used in tasks such as text classification, question answering, and sequence labeling. A typical approach involves using a teacher model with a large number of parameters to generate soft labels and intermediate layer representations on the training set or unlabeled data. Then, by minimizing Kullback-Leibler divergence and feature matching loss, the knowledge from the teacher model is transferred to a smaller, more efficient student model. In text classification scenarios, existing large-model distillation methods mainly focus on improving the classification accuracy of the student model to approach or approximate that of the teacher model, and compressing the model size while maintaining accuracy to improve inference efficiency. However, in hierarchical multi-label text classification tasks, this approach lacks distillation design for hierarchical label results, leading to insufficient understanding of the hierarchical structure by the student model, and a lack of selection and constraint of reliable features. In real-world applications, training data may contain noise factors such as inconsistent labeling, corpus distribution shifts, and short-term hotspots. Some features learned by the teacher model on this data are unstable. Traditional distillation typically "accepts all" the teacher's output, and the student model will also inherit these unstable or even misleading patterns, affecting the reliability and interpretability of the model in real-world applications.

[0007] In recent years, some technical solutions have applied the Retrieval-Augmented Generation (RAG) framework to text processing tasks. This involves first retrieving relevant information from a knowledge base using vector retrieval, and then feeding the retrieved results along with the original input into a large language model for answering or reasoning. However, existing RAG applications mostly focus on question answering and knowledge retrieval scenarios, lacking specialized designs for multi-level tag attribution. Targeted solutions are still lacking for issues such as hierarchical routing of tag systems, fine-grained ranking of candidate tags, and adjudication within a controlled candidate set.

[0008] In summary, the following are the core problems existing in current hierarchical multi-label text classification technologies:

[0009] First, the lack of a reliable feature filtering mechanism makes the model prone to learning noise and unstable patterns in the data;

[0010] Second, the hierarchical structure information of the tag system was not fully utilized, and the dependency relationship between the parent class and the child class was not effectively modeled;

[0011] Third, the training and inference costs are too high during the fine-tuning and inference process of large models, making it difficult to implement in resource-constrained business systems. Summary of the Invention

[0012] This invention addresses the problems of existing technologies, such as lack of credible features, insufficient utilization of the hierarchical structure information of the labeling system, and high training and inference costs. It proposes a hierarchical multi-label attribution scheme that integrates atomic rule weakly supervised annotation, credible feature screening, and knowledge distillation. By integrating atomic rule weakly supervised annotation, credible features, and knowledge distillation, and introducing a distillation framework of "teacher large model - student small model", the inference cost is significantly reduced while ensuring the accuracy of hierarchical multi-label attribution. This achieves efficient, stable, and interpretable attribution capabilities under complex labeling systems.

[0013] To achieve the above objectives, this invention provides a hierarchical multi-label attribution method that integrates atomic rule-driven credible features and knowledge distillation, comprising the following steps:

[0014] Step 1, Data Preparation: Construct a hierarchical tagging system containing multi-level parent-child tags; construct an atomic rule library consisting of several atomic rules, each atomic rule containing the rule triggering condition and its associated target tag; use the atomic rule library to automatically match the text, generate candidate tags according to the matched rules, and extract the text fragments that trigger the rules as evidence fragments of the corresponding tags to form a weakly labeled sample set.

[0015] Step 2, Teacher Model Inference: Using a large language model as the teacher model, the text and the weakly labeled samples are constructed as task instruction prompts and input into the teacher model. The soft label probability distribution of the teacher model output on each level of labels, as well as the intermediate layer feature representation generated during the inference process, are obtained.

[0016] Step 3, Trustworthy Feature Filtering: Evaluate the consistency, stability and business relevance of the soft label probability distribution and intermediate layer feature representation output by the teacher model, and filter out trustworthy soft labels and trustworthy feature dimension subsets that meet the preset trustworthiness thresholds respectively.

[0017] Step 4, Constructing a student model: Construct a student model using a lightweight pre-trained model. The top layer of the student model has a hierarchical output structure that matches the hierarchical labeling system, and its intermediate layers reserve a feature space that is aligned with the subset of the credible feature dimensions.

[0018] Step 5, Model Distillation Training: Construct a joint training objective function, and use labeled data, trusted soft labels, and trusted features to perform end-to-end training on the student model through backpropagation to minimize the joint training objective function;

[0019] Step 6, Model Inference: Load and deploy only the trained student model, output the probability distribution of the text to be processed on each level of label, select the final attribution label according to the preset threshold, and combine the learning of the relevant features of the evidence fragments during training to output one or more fragments with the highest correlation with each label as key evidence sentences.

[0020] Furthermore, the specific method for constructing the atomic rule base in step 1 is as follows: domain expert experience, classification rules, typical expression pattern rules, tag definition rules, or keyword table rules are decomposed into atomic rules, and each atomic rule is formalized as a triggering condition, an associated set of target tags, and a rule weight or confidence level; the associated set of target tags is a subset of tags at one or more levels in the hierarchical tag system.

[0021] Furthermore, the specific method for generating weakly labeled samples using the atomic rule base in step 1 is as follows: each text segment in the text is traversed and matched against the atomic rule base. If a text segment meets the triggering condition of a certain atomic rule, it is considered a hit. For each label, the weights of all rules that hit the label are aggregated and converted into weak label scores. A candidate label set is generated according to a preset threshold, and the text segments that hit the rules and their corresponding rules and labels are retained as a set of evidence fragments. At the same time, some samples are manually labeled to form manually labeled labels as hard labels for subsequent training.

[0022] Furthermore, the credible feature selection in step 3 includes: First, combining the weak annotations generated by atomic rules, performing consistency analysis on the teacher model prediction results, and selecting stable label predictions as credible soft labels; Second, evaluating the feature importance of the teacher intermediate representations, and selecting a subset of features with stable importance and high correlation with the correct labels as credible feature dimensions; Then, combining the consistency analysis results, the degree of consistency with the annotations, and the feature importance stability index, selecting each level, each label, and the corresponding feature dimension, and retaining only the soft label distribution and features that reach the preset credibility threshold as credible soft labels and credible features, thus forming a credible feature library.

[0023] Furthermore, in step 4, the hierarchical output structure of the student model maps the hierarchical labels to a unified vector space and explicitly encodes the parent-child relationship through the parent-child constraint structure, enabling the model to output multi-level, multi-label probability distributions simultaneously.

[0024] Furthermore, the joint training objective function in step 5 includes hard label loss, soft label loss, and feature distillation loss. The hard label loss is calculated based on the student model's prediction results after manual verification of weak labels, ensuring the student model can directly fit high-quality annotations. The soft label loss uses KL divergence to calculate the loss between the probability distribution output by the student model and the soft labels, guiding the student model to learn the fine-grained probability structure of the teacher model within the trustworthy region. The feature distillation loss establishes an alignment relationship between the intermediate layer of the student model and the trustworthy feature dimensions of the teacher model, minimizing the distance between them in the corresponding dimensions to transfer stable and reliable discriminative features from the teacher model to the student model. The hard label loss, soft label loss, and feature distillation loss are combined in a weighted manner to form the joint training objective function.

[0025] To achieve the above objectives, the present invention also provides a hierarchical multi-label attribution system that integrates atomic rule-driven trusted features and knowledge distillation, comprising:

[0026] The data preparation module is configured to construct a hierarchical tag system containing multi-level parent and child tags; construct an atomic rule library consisting of several atomic rules, each atomic rule containing the rule triggering condition and its associated target tag; use the atomic rule library to automatically match the text, generate candidate tags according to the matched rules, and extract the text fragments that trigger the rules as evidence fragments of the corresponding tags to form a weakly labeled sample set.

[0027] The teacher model inference module is configured to use a large language model as the teacher model. The text and the weakly labeled samples are constructed as task instruction prompts and input into the teacher model. The module obtains the soft label probability distribution of the teacher model output on each level of labels, as well as the intermediate layer feature representation generated during the inference process.

[0028] The credible feature filtering module is configured to evaluate the consistency, stability, and business relevance of the soft label probability distribution and intermediate layer feature representation output by the teacher model, and respectively filter out credible soft labels and credible feature dimension subsets that meet the preset credible threshold.

[0029] The student model building module is configured to build a student model using a lightweight pre-trained model. The top layer of the student model has a hierarchical output structure that matches the hierarchical labeling system, and its intermediate layers reserve a feature space that is aligned with the subset of the credible feature dimensions.

[0030] The model distillation training module is configured to construct a joint training objective function and, using labeled data, trusted soft labels, and trusted features, perform end-to-end training on the student model through backpropagation to minimize the joint training objective function.

[0031] The model inference module is configured to provide the trained student model with the probability distribution of the text to be processed on each level of labels, select the final attribution label according to a preset threshold, and combine the learning of the relevant features of the evidence fragments during training to output one or more fragments with the highest correlation with each label as key evidence sentences.

[0032] Furthermore, the teacher model inference module includes: an input construction unit for constructing task instruction prompts from text and weakly labeled samples; a soft label extraction unit for obtaining the soft label probability distribution output by the teacher model at each level of labels; and an intermediate feature extraction unit for obtaining intermediate layer feature representations generated by the teacher model during the inference process, including deep semantic vectors and label embedding representations of text segments.

[0033] Furthermore, the credible feature screening module includes: a soft label credibility evaluation unit, used to calculate credibility scores and screen credible soft labels by integrating teacher model prediction probabilities, rule weak label scores and manual hard label consistency information; and a feature dimension screening unit, used to statistically analyze teacher intermediate layer feature dimensions that are highly correlated with each label to form a subset of credible feature dimensions.

[0034] Furthermore, the model distillation training module includes: a hard label loss calculation unit, used to calculate the binary cross-entropy loss between student model predictions and manual annotations; a soft label distillation loss calculation unit, used to calculate the KL divergence loss between student model predictions and credible soft labels, using the credibility score as the weight; a feature distillation loss calculation unit, used to calculate the mean squared error loss between student-mapped features and teacher-original features on a subset of credible feature dimensions; and a joint optimization unit, used to weighted summation of the three losses to form a total loss and perform backpropagation optimization on the model accordingly.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] (1) By combining atomic rules with a large model, the annotation cost is significantly reduced and the quality of training samples is improved. Business rules are transformed into an atomic rule library, which automatically matches and weakly annotates the text, generating candidate labels and corresponding evidence fragments. The large model then corrects and supplements the weak annotation results. This approach reduces the workload of manual annotation while improving the overall quality of training samples in terms of coverage, accuracy, and label hierarchy consistency.

[0037] (2) Improve distillation quality and robustness through reliable feature screening. The distribution of soft labels and intermediate representations at each level and for each label are evaluated and screened for reliability by combining the weak label results after manual verification. Only label predictions that are stable under different samples and perturbations and highly consistent with manual annotations are retained as reliable features. Compared with traditional distillation methods, this effectively suppresses the transfer of noisy knowledge to the student model.

[0038] (3) Integrating hierarchical structure into the modeling process enhances multi-label attribution capability. By explicitly constructing a hierarchical output structure consistent with the label system and adding parent-child label consistency constraints during training, the student model can simultaneously learn the hierarchical dependencies and co-occurrence relationships between parent-child classes and multi-level labels. Compared with traditional multi-label classification methods, this approach has higher structural consistency and attribution accuracy in multi-level, multi-label scenarios.

[0039] (4) Achieve efficient inheritance of large model capabilities by multi-source joint distillation. By comprehensively utilizing manually labeled hard labels, reliable soft labels and intermediate feature distillation, a joint loss is constructed on the student model, which takes into account both the accuracy of hard labels and the fine-grained discrimination capability of the large model.

[0040] (5) Inference overhead is significantly reduced and interpretability is enhanced. Only a lightweight student model is deployed during the inference stage, which greatly reduces computational resources and response time. At the same time, by aligning credible features with label decisions during training, the student model can output mainly based on text fragments, which significantly improves the interpretability of attribution results. Attached Figure Description

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0042] Figure 1 This is a schematic diagram of the hierarchical multi-label attribution method that integrates atomic rule-driven credible features and knowledge distillation in this invention.

[0043] Figure 2 This is a schematic diagram of the hierarchical multi-label attribution system that integrates atomic rule-driven trusted features and knowledge distillation in this invention.

[0044] Figure 3 This is a schematic diagram illustrating the implementation process of hierarchical multi-label attribution in an example of the present invention. Detailed Implementation

[0045] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0046] This invention presents a hierarchical multi-label attribution method that integrates atomic rule-driven credible features and knowledge distillation. It combines weakly supervised atomic rule annotation, credible feature selection, and knowledge distillation. By constructing a distillation framework of "teacher large model – student small model," the large model is treated as the teacher, from which credible hierarchical discriminative features are extracted and selected. The extracted credible features are then used to perform constrained distillation on the small model, while explicitly modeling hierarchical structures such as parent-child classes. This achieves efficient and interpretable hierarchical text classification reasoning without relying on online retrieval or large model reasoning.

[0047] Specifically, this hierarchical multi-label attribution method firstly designs an atomic rule base to automatically match evidence fragments in the text, thereby constructing a large-scale weakly labeled sample set at low cost.

[0048] Then, within the knowledge distillation framework of using a large model as the teacher and a small model as the student, the strong representation and reasoning capabilities of the large model are utilized to correct and supplement the weak labels generated by atomic rules. A feature extraction and hierarchical representation mechanism oriented towards hierarchical labeling systems is designed to collect parent class features, child class features, and cross-level relationship features from the large model. At the same time, a credible feature screening and constraint strategy is introduced to evaluate the stability, consistency, and business relevance of features and soft labels generated by the large model. Only features that meet the credibility criteria (including soft label distribution, hierarchical structure information, and intermediate representations corresponding to evidence fragments) are used to guide the student model's learning. Based on this, a distillation loss function adapted to hierarchical multi-label structures is constructed, enabling the student model to accurately learn hierarchical relationships such as parent-child relationships while compressing parameter scale and inference overhead, thereby improving classification performance and hierarchical consistency under large-scale, multi-level labeling systems.

[0049] Finally, by distilling and aligning the relevant features of the evidence fragments, the resulting small model can not only output hierarchical multi-label prediction results, but also provide evidence sentences that play a key role in each label. Thus, while maintaining or approaching the attribution ability of the large model, it significantly reduces inference costs and improves the stability and interpretability of hierarchical multi-label attribution results.

[0050] See Figure 1 The hierarchical multi-label attribution method that integrates atomic rule-driven credible features and knowledge distillation, as presented in this invention, specifically includes the following steps:

[0051] Step 1, Data Preparation.

[0052] This step specifically involves constructing a hierarchical tagging system containing multi-level parent-child tags, an atomic rule base, and annotation data.

[0053] When constructing a hierarchical labeling system, firstly, a multi-level label structure is defined, including parent labels and their subordinate sub-labels, forming a tree or directed acyclic graph structure. Each label node is then configured with a label name, label description, and hierarchical / superior relationship. Next, the dataset is collected, and preprocessing operations such as cleaning and normalization are performed on the data.

[0054] Based on this, an atomic rule library is constructed, consisting of several atomic rules. Each atomic rule contains a triggering condition and its associated target labels. When constructing the atomic rule library, domain expert experience, classification rules, typical expression pattern rules, label definition rules, or keyword list rules are broken down into atomic rules. Each atomic rule is formalized as a triggering condition, an associated set of target labels, and a rule weight or confidence level. The triggering conditions include keyword matching, regular expressions, phrase templates, or simple syntactic patterns; the associated set of target labels is a subset of labels from one or more levels in the constructed hierarchical labeling system.

[0055] Furthermore, the preprocessed text is automatically matched using the constructed atomic rules, candidate labels are generated based on the matched rules, and the text fragments that trigger the rules are extracted as evidence fragments for the corresponding labels, forming a weakly labeled sample set.

[0056] As further explanation, the process of generating weakly labeled samples using the atomic rule base includes: traversing the atomic rule base to match each text segment in the text; if a text segment meets the triggering condition of a certain atomic rule, it is considered a hit; for each label, aggregating the weights of all rules that hit that label, and obtaining the weak label score after transformation by the Sigmoid function; generating a candidate label set according to a preset threshold, and retaining the text segments that hit the rules and their corresponding rules and labels as a set of evidence fragments; and simultaneously performing manual fine labeling on some samples to form manually labeled labels as hard labels for subsequent training.

[0057] Step 2, Teacher Model Inference: Select the large teacher model and obtain hierarchical multi-label predictions and intermediate layer features.

[0058] This step uses a large language model as the teacher model. For example, deepseek-r1 (671b) can be used as the teacher model.

[0059] Based on this, according to the candidate labels and evidence fragments provided by the labeling system and atomic rules in step 1, appropriate task instruction prompts are constructed and input into the teacher model to guide the teacher model to make corrections and additions based on the given text and candidate information.

[0060] The teacher model can output soft labels in the form of attribution results and probability distributions at each level of labels based on task instructions and samples, and perform consistency judgment and correction on weak labels generated by atomic rules. At the same time, it outputs intermediate layer feature representations of the input text during the inference process, which are used for subsequent reliable feature selection and feature distillation.

[0061] The intermediate layer feature representations here include the overall text vector representation, the deep semantic vector of each text segment, and the label embedding representation of each label.

[0062] As a further explanation, the teacher model also calculates the relevance score between the text segment and the label using the inner product scoring function, which is used to analyze the relevance between the label and the text segment.

[0063] Step 3: Screening of trustworthy features and trustworthy soft tags.

[0064] This step is used to evaluate the consistency, stability, and business relevance of the soft label probability distribution and intermediate layer feature representation output by the teacher model, and to select a subset of credible soft labels and credible feature dimensions that meet the preset credibility threshold.

[0065] Furthermore, this step, in its specific implementation, includes the following sub-steps:

[0066] First, regarding the consistency of the teacher model's prediction results in sample analysis, we combine the weak labels generated by atomic rules to statistically analyze the recall, precision, and confidence of the teacher model, and select label predictions that are stable and consistent with human annotations, and mark them as reliable soft labels.

[0067] Secondly, feature importance assessment is performed on the teacher intermediate layer feature representation, focusing on feature dimensions related to the corresponding positions of rule trigger fragments and evidence fragments. Feature subsets that are stable in importance across multiple rounds of reasoning and multiple sample subsets and are highly correlated with the correct label are selected as credible feature dimensions.

[0068] Then, based on comprehensive consistency analysis, consistency with annotations, feature importance and stability, each level, each label and corresponding feature dimension are scored and filtered. Only the distribution and features of soft labels that reach the preset confidence threshold are retained as credible soft labels and credible features, forming a credible feature library that can be used for distillation.

[0069] As a supplementary explanation, this step, when performing consistency analysis, specifically includes: integrating the teacher model prediction probability, the weak label score of the rule, and the consistency information with the manual hard label, calculating the credibility score of each sample-label pair through a weighted function, and including the label prediction into the set of credible soft labels when the credibility score reaches a preset threshold.

[0070] This step, when assessing feature importance, specifically includes: statistically analyzing the intermediate-level feature dimensions of teachers that are highly correlated with each label on a large number of samples; obtaining the feature dimension subsets corresponding to each label through variance contribution or importance scoring methods; and taking the union of the feature dimension subsets of all labels as the credible feature dimension subset.

[0071] Step 4: Construct a student model with a hierarchical output structure.

[0072] This step uses a lightweight pre-trained model to build the student model. For example, Chinese-RoBERTa-base (125M) can be selected as the student model.

[0073] This step further constructs a hierarchical output structure at the top level of the student model that matches the hierarchical labeling system, and reserves a feature space in the middle layer of the model that is aligned with the teacher's credible feature dimension, so as to receive feature distillation related to evidence fragments during training.

[0074] The hierarchical output structure here can map hierarchical labels to a unified vector space and explicitly encode parent-child relationships through parent-child constraint structures, enabling the model to output multi-level, multi-label probability distributions simultaneously.

[0075] Furthermore, this hierarchical output structure can be implemented through the following scheme:

[0076] A learnable label embedding representation is constructed for all labels; the representations of text segments are aggregated through an attention mechanism to obtain a sample-level global representation; for each label, its label embedding is associated with the sample global representation and then processed by an activation function to obtain the predicted probability of the label; and the hierarchical output structure includes a parent-child constraint structure for encoding the constraint relationship between parent and child labels.

[0077] As further explanation, the parent-child constraint structure here is a structural constraint matrix. This structural constraint matrix explicitly encodes the inclusion relationship between parent and child labels and is used to enforce the parent-child label consistency constraint during training or inference. That is, when any child label is predicted as a positive example, all its ancestor labels must also be predicted as positive examples.

[0078] The feature space alignment in this step maps the features of the student model to the feature space dimension of the teacher model through a linear mapping layer.

[0079] Step 5, Model Training.

[0080] This step first constructs a joint training objective function that includes hard label loss, soft label distillation loss, and feature distillation loss. Here, the joint training objective function is the joint loss function. Then, using labeled data, reliable soft labels, and reliable features, the student model is trained end-to-end through backpropagation to minimize the joint loss function.

[0081] The hard-label loss in the joint training objective function constructed in this step is based on the label results after manual verification of weak labels. The loss is calculated on the prediction results of the student model to ensure that the student model can directly fit the high-quality labels.

[0082] The soft-label distillation loss is calculated using KL divergence to determine the loss between the probability distribution of the student model's output and the soft label, guiding the student model to learn the fine-grained probability structure of the teacher model within the confidence region.

[0083] Feature distillation loss establishes an alignment relationship between the intermediate layer of the student model and the reliable feature dimension of the teacher model. By minimizing the distance between the two in the corresponding dimension, the stable and reliable discriminative features of the teacher model are transferred to the student model.

[0084] Furthermore, the hard label, soft label, and feature distillation loss are combined in a weighted manner into a joint loss function, and the student model is trained end-to-end through backpropagation.

[0085] Step 6, Model Inference.

[0086] In this step, which serves as the inference phase, only the pre-trained student model is loaded and deployed. For the input text, the student model outputs a hierarchical multi-label probability distribution (and the probability distribution of the text across each level of label), selecting the final attribution label based on a preset threshold. Simultaneously, combining the learned features related to evidence fragments during training, sentences or fragments in the text are scored and ranked, selecting one or more fragments with the highest correlation to each label as key evidence sentences, which are output along with the predicted labels. This achieves low-cost, high-efficiency, hierarchical, and interpretable multi-label classification and attribution.

[0087] As further explanation, in this step, when outputting key evidence fragments, the attention mechanism or inner product scoring function inside the student model is used to evaluate the importance of each fragment of the input text. For each predicted label, one or more fragments with the highest scores are selected as key evidence fragments for that label and output together with the predicted label.

[0088] The hierarchical multi-label attribution method based on the above scheme, which integrates atomic rule-driven trusted features and knowledge distillation, has the following technical characteristics compared to existing technologies:

[0089] (1) The method provided by this invention changes the traditional approach that relies solely on manual annotation or is completely unsupervised. By constructing an atomic rule base, it formalizes domain expert experience and business rules into computable atomic rules, enabling automatic matching and weak annotation of text. Each atomic rule contains triggering conditions, associated labels, and weights, and can simultaneously generate candidate labels and corresponding evidence fragments during the annotation process. This not only significantly reduces the cost of manual annotation but also provides an initial "anchor point" for subsequent credible feature selection and interpretable output.

[0090] (2) The method provided by this invention abandons the traditional knowledge distillation method's unconditional acceptance of all outputs from the teacher model and introduces a reliable feature selection step. Specifically, this invention performs multi-dimensional reliability evaluation on the soft label probability distribution and intermediate layer feature representation output by the teacher model (large language model), including: calculating the reliability score of the soft label by combining the rule-based weak label score and the manual hard label; and selecting stable and reliable feature dimensions by the feature importance on statistical samples. Only soft labels and feature dimensions that pass the preset reliability threshold will be used to guide the training of the student model. This mechanism effectively suppresses the transfer of unstable and unreliable knowledge learned by the teacher model from noisy data to the student model, significantly improving the robustness and generalization ability of the final model.

[0091] (3) The method provided by this invention constructs a distillation framework of "teacher large model - student small model" and designs an innovative joint loss function. This loss function integrates knowledge from three sources: first, hard label loss from manual annotation to ensure the basic accuracy of the student model; second, soft label loss after reliable screening to guide the student model to learn the fine-grained probability structure of the teacher model in the reliable region; and third, feature distillation loss from the intermediate layer of the teacher model to directly transfer the stable and reliable discriminative features of the teacher model to the student model. At the same time, the top layer of the student model is designed with a hierarchical output structure consistent with the label system, and the inclusion relationship between parent and child labels is explicitly encoded through the structural constraint matrix to enforce the parent-child label consistency constraint.

[0092] The hierarchical multi-label attribution method that integrates atomic rule-driven trusted features and knowledge distillation, as presented in this example, can be configured into a corresponding software program to form a hierarchical multi-label attribution system that integrates atomic rule-driven trusted features and knowledge distillation. When this software program runs, it executes the aforementioned hierarchical multi-label attribution method that integrates atomic rule-driven trusted features and knowledge distillation, and stores the results in a suitable storage medium for the processor to retrieve and execute.

[0093] See Figure 2The resulting hierarchical multi-label attribution system, which integrates atomic rule-driven credible features and knowledge distillation, comprises the following six components: data preparation module 100, teacher model reasoning module 200, credible feature screening module 300, student model construction module 400, model distillation training module 500, and model reasoning module 600.

[0094] The data preparation module 100 is configured to be responsible for building a hierarchical labeling system, building an atomic rule base, and generating weakly labeled samples.

[0095] The teacher model inference module 200 is configured to call the large language model teacher to correct and supplement weakly labeled samples, and extract soft labels and intermediate layer features.

[0096] The credible feature screening module 300 is configured to perform reliability assessment on the output of the teacher model and screen out credible knowledge for distillation.

[0097] The student model building module 400 is configured to build lightweight student models with hierarchical output structures.

[0098] The Model Distillation Training Module 500 is configured to perform end-to-end training of the student model using a joint training objective function.

[0099] The model inference module 600 is configured to deploy the trained student model, predict new input text, and output attribution results.

[0100] In the resulting hierarchical multi-label attribution system, the original text data first enters the data preparation module 100. After label system construction, atomic rule matching, and weak labeling processing, a weakly labeled sample set (including candidate labels and evidence fragments) is generated. These weakly labeled samples, along with the original text, are sent to the teacher model inference module 200, where the large language model teacher corrects and supplements them, outputting soft label probability distributions and intermediate layer feature representations. Subsequently, the credible feature selection module 300 performs consistency evaluation and feature importance analysis on the teacher model's output, selecting credible soft labels and a subset of credible feature dimensions. The student model construction module 400 constructs a lightweight student model with a hierarchical output structure and reserves a feature space aligned with the teacher's credible feature dimensions. The model distillation training module 500 comprehensively utilizes manually labeled hard labels, credible soft labels and a subset of credible feature dimensions output by the credible feature selection module, and manually labeled hard labels output by the data preparation module to construct a joint loss function for end-to-end training of the student model. Finally, the model inference module 600 deploys the trained student model to perform hierarchical multi-label prediction and output key evidence fragments on new input text.

[0101] The following section provides a further explanation of the specific configuration schemes of each component module in this system.

[0102] The data preparation module 100 in this system specifically includes a tag system construction unit 101, an atomic rule base construction unit 102, a text preprocessing unit 103, and a weak annotation generation unit 104.

[0103] The tag system construction unit 101 is configured to transform the classification system in the business into a computer-processable hierarchical structure. Specifically, it defines a multi-level tag structure, configuring structured information for each tag node, including tag ID, Chinese name, tag description, and a list of parent tag IDs. For subclasses with the same name but different semantics, different internal IDs are used for differentiation, and their respective parent tags are recorded through a parent-child mapping function. This structure is stored in the form of an adjacency list or a parent-child index matrix for subsequent hierarchical constraint modeling. This unit further supports a directed acyclic graph (DAG) tag system, allowing multiple inheritance relationships, and achieves efficient parent-child relationship lookup through the parent-child index matrix, providing efficient data structure support for subsequent hierarchical consistency constraints.

[0104] The atomic rule base construction unit 102 is configured to decompose domain expert experience, book classification rules, and typical expression patterns into computable atomic rules. Each atomic rule is formalized as a triggering condition, an associated set of target labels, and a rule weight. The triggering conditions include keyword matching, regular expressions, phrase templates, and simple syntactic patterns. This unit transforms loosely defined, implicit expert knowledge into structured, computable, formalized rules. Each rule is independent, composable, and interpretable, providing a traceable source of evidence for weakly supervised annotation.

[0105] The text preprocessing unit 103 is configured to clean and standardize the original text. For example, the preprocessing includes: removing HTML tags, special control characters, and redundant spaces; standardizing the encoding format; converting Traditional Chinese to Simplified Chinese; and segmenting long texts into paragraphs of natural length or fixed length for subsequent fine-grained rule matching and feature extraction. This unit preserves the positional mapping between the text segments and the original text during preprocessing, enabling evidence fragments to be accurately traced back to their specific locations in the original text.

[0106] The weak annotation generation unit 104 is configured to automatically weakly annotate text using an atomic rule base. Specifically, this unit performs the following processing: for each text segment, it iterates through each rule in the atomic rule base, checking if the triggering condition is met; if so, it records the hit and accumulates the score based on the rule weight. For all labels, it converts the aggregated score using a Sigmoid function to obtain a weak annotation score. A candidate label set is generated based on a preset threshold, and all hit text segments and their corresponding rules and labels are retained as an evidence fragment set. Simultaneously, some samples undergo manual fine-tuning to form manually annotated labels. This unit filters low-confidence weak annotation results through a threshold screening mechanism, avoiding the introduction of excessive noise; the combination of manual fine-tuning and weak annotation significantly reduces the cost of manual annotation while ensuring annotation quality.

[0107] The data preparation module 100 thus constitutes outputs a set of weakly labeled samples (including a set of candidate labels, a set of evidence fragments, and a weakly labeled score vector) and manually labeled hard labels for some samples during operation; among them, the weakly labeled samples and the original text are transmitted to the teacher model inference module 200, and the manually labeled hard labels are transmitted to the model distillation training module 500.

[0108] The teacher model reasoning module 200 in this system, when specifically implemented, includes an input construction unit 201, a model calling unit 202, a soft label extraction unit 203, and an intermediate feature extraction unit 204.

[0109] The input construction unit 201 is configured to construct structured cue words suitable for large language models from the original text and weakly labeled samples.

[0110] As an example, for each sample in this unit, the constructed input text includes: original content (such as book title, synopsis, key chapter titles, etc.); candidate labels generated by rule-based weak labeling and their corresponding label descriptions; representative short sentences from the rule-based evidence fragments; and a summary of the label hierarchy structure. This content is organized in the form of a structured prompt template to guide the teacher model in making corrections based on existing weak labeling information. The prompts constructed in this unit explicitly contain candidate labels and evidence fragments, leveraging the teacher model's powerful reasoning ability while reducing its search range in the irrelevant label space, thus improving output quality and efficiency.

[0111] The model invocation unit 202 is configured to invoke large language model teachers, supporting batch and asynchronous invocation, and can efficiently process large-scale weakly labeled samples.

[0112] The soft label extraction unit 203 is configured to parse the soft label probability distribution of each level of labels from the output response of the teacher model, including the probability values ​​of first-level, second-level, and third-level labels. By extracting the probability distribution of each level separately, this unit preserves the independence information between levels, providing fine-grained soft label data for subsequent level constraints and reliable screening.

[0113] The intermediate feature extraction unit 204 is configured to acquire intermediate feature representations generated by the teacher model during inference, including the deep semantic vector of each text segment, the embedding representation of each label, and the relevance score between the text segment and the label calculated by the inner product scoring function. This unit not only extracts the output of the teacher model but also retains the intermediate state features during the inference process.

[0114] The teacher model reasoning module 200, thus formed, obtains the original text, weakly labeled sample set (candidate labels and evidence fragments), and label system description information from the data preparation module 100. First, the input construction unit 201 constructs the original text, candidate labels and evidence fragments in the weakly labeled samples, and label system description information into structured task instruction prompts. Then, the model calling unit 202 calls a large language model (such as DeepSeek-R1) for reasoning. The soft label extraction unit 203 extracts the soft label probability distribution (including the probability values ​​of first-level, second-level, and third-level labels) on each level of labels from the model output. At the same time, the intermediate feature extraction unit 204 extracts the intermediate layer feature representations generated by the model during the reasoning process, including the deep semantic vector of each text segment, the embedding representation of each label, and the relevance score between the text segment and the label calculated by the inner product scoring function. This generates the output soft label probability distribution (probability values ​​of first-level, second-level, and third-level labels) and intermediate layer feature representations (text segment semantic vector matrix, label embedding vector, and text segment-label relevance score), and transmits them to the credible feature filtering module 300.

[0115] In its specific implementation, the credible feature screening module 300 in this system includes a soft tag credibility evaluation unit 301, a feature stability analysis unit 302, and a credibility dimension screening unit 303.

[0116] The soft-label credibility assessment unit 301 is configured to evaluate the reliability of the soft labels output by the teacher model. Specifically, for each sample and label, this unit calculates a credibility score using a weighted function, integrating the teacher model's prediction probability, rule-based weak labeling scores, and consistency information with manually labeled hard labels. When the credibility score reaches a preset threshold, the label prediction is included in the set of credible soft labels. This unit effectively identifies stable and reliable soft labels in the teacher model through a fusion evaluation mechanism of three information sources (teacher prediction, rule labeling, and manual labeling).

[0117] The feature stability analysis unit 302 is used to analyze the stability performance of the intermediate layer features of the teacher model under different samples and perturbations. Specifically, this unit is configured to perform multiple inferences on the same sample, calculate the variance of the same feature dimension in multiple inferences, and the smaller the variance, the more stable the feature dimension is; at the same time, it analyzes the mutual information between features and labels, and calculates the correlation between each feature dimension and the label.

[0118] The credible dimension filtering unit 303 is used to filter out key feature dimensions that need distillation from the intermediate layer features of the teacher model. Specifically, this unit is configured to, for each label, collect the text segment representations of the teacher model from all positive samples and calculate the average vector; analyze the variance contribution or absolute mean of this average vector on each dimension, and select the top-ranked dimensions as the subset of feature dimensions highly correlated with that label; after performing the above operation on all labels, take the union of all subsets as the final credible feature dimension subset. This unit, through label-related feature dimension filtering, focuses the distillation target on the feature dimensions most valuable for the classification task, avoiding the computational overhead and potential feature interference caused by distillation across the entire feature space.

[0119] The credible feature screening module 300 thus constitutes a credible feature screening module. When working, it obtains the soft label probability distribution and intermediate layer feature representation from the teacher model inference module 200. The credible feature screening module 301 calculates the credibility score by combining teacher prediction, rule labeling and manual labeling, and screens credible soft labels. At the same time, the feature stability analysis unit 302 analyzes the stability of features under different samples and perturbations. The credible dimension screening unit (303) counts the feature dimensions that are highly related to each label to form a credible feature dimension subset. The generated credible soft labels and credible feature dimension subset are then transmitted to the model distillation training module 500.

[0120] In its specific implementation, the student model building module 400 in this system includes a text encoder unit 401, an attention aggregation unit 402, a hierarchical label embedding unit 403, a prediction output unit 404, and a feature space alignment unit 405.

[0121] The text encoder unit 401 is used to encode each segment of the input text into a semantic vector. For example, a lightweight pre-trained model can be used as the encoder.

[0122] Attention aggregation unit 402 is used to aggregate the representations of all text segments into a global representation of the entire sample. This unit is configured to calculate the attention weight of each text segment, and then calculate the weighted sum to obtain the global representation of the sample.

[0123] The hierarchical label embedding unit 403 is used to construct a learnable label embedding matrix for all labels in the label system. Each label embedding is optimized through backpropagation during training.

[0124] The prediction output unit 404 is configured to calculate the prediction probability for each label. Specifically, for each label, the prediction output unit 404 performs an inner product or linear transformation between its label embedding and the global representation of the sample, and then activates it with the Sigmoid function to obtain the prediction probability of that label.

[0125] The feature space alignment unit 405 is configured to reserve a feature space aligned with the reliable feature dimensions of the teacher model in the intermediate layers of the student model. Specifically, this unit is configured to set up a linear mapping layer to map student features to the teacher feature space dimensions; simultaneously, during distillation training, alignment constraints are applied only to the selected reliable feature dimensions. This unit achieves the low-dimensional to high-dimensional mapping of student features to the teacher feature space through the linear mapping layer; furthermore, by aligning only on the reliable feature dimensions, it avoids the overfitting risk associated with distillation across the entire feature space, while significantly reducing the computational cost of feature distillation.

[0126] The student model building module 400 thus generates a student model with a hierarchical output structure (including model structure definition, initial parameters, and feature mapping matrix), and transmits the student model and its related parameters to the model distillation training module 500.

[0127] In its specific implementation, the model distillation training module 500 in this system includes a hard label loss calculation unit 501, a soft label distillation loss calculation unit 502, a feature distillation loss calculation unit 503, a joint loss fusion unit 504, and a backpropagation optimization unit 505.

[0128] The hard-label loss calculation unit 501 is configured to calculate the loss based on manually labeled hard labels, ensuring the basic classification accuracy of the student model. Specifically, for the sample set with manual labeling, this unit uses multi-label binary cross-entropy loss at each level.

[0129] The soft-label distillation loss calculation unit 502 is configured to calculate distillation loss based on trusted soft labels, guiding the student model to learn the fine-grained probability structure of the teacher model within the trusted region. Specifically, this unit calculates the loss using KL divergence for labels in the trusted label set that reach the trusted threshold, and uses the trusted score as the loss weight.

[0130] The feature distillation loss calculation unit 503 is configured to calculate the mean squared error loss of student features and teacher features on a subset of credible feature dimensions.

[0131] The joint loss fusion unit 504 is configured to interact with the hard label loss calculation unit 501, the soft label distillation loss calculation unit 502, and the feature distillation loss calculation unit 503 to perform data exchange, and to perform weighted summation of the three losses calculated by the three units to form the total loss.

[0132] The backpropagation optimization unit 505 is configured to perform end-to-end training of the student model.

[0133] The model distillation training module 500, as thus constructed, acquires manually labeled hard labels from the data preparation module 100, reliable soft labels and a subset of reliable feature dimensions from the reliable feature selection module 300, the student model and feature mapping matrix from the student model construction module 400, and the training dataset. The hard label loss calculation unit 501 calculates the binary cross-entropy loss based on manual annotation; the soft label distillation loss calculation unit 502 calculates the KL divergence loss based on reliable soft labels, using the reliability score as the weight; the feature distillation loss calculation unit 503 calculates the mean squared error loss of student features and teacher features on the subset of reliable feature dimensions; the joint loss fusion unit 504 then weights and sums the three losses; the backpropagation optimization unit 505 then uses an optimizer to perform end-to-end parameter updates on the student model; finally, a converged student model (including optimized model parameters, feature mapping matrix, and label embeddings) is trained.

[0134] In its specific implementation, the model reasoning module 600 in this system includes a label prediction unit 601, an evidence fragment output unit 602, and a hierarchical consistency verification unit 603.

[0135] The label prediction unit 601 is used to output the probability distribution of labels at each level of the text to be predicted and determine the attribution labels. Specifically, this unit is configured to input the preprocessed text to be predicted into the student model, obtain the predicted probabilities of labels at each level, and generate the final label set according to a preset threshold or Top-k rule.

[0136] The evidence fragment output unit 602 is used to output key evidence fragments for each predicted label. Specifically, this unit is configured to use the attention weights or inner product scoring function inside the student model to calculate the relevance score between each text fragment and the predicted label, and select the fragments with the highest scores as evidence.

[0137] The hierarchical consistency verification unit 603 is used to ensure that the prediction results satisfy the parent-child label inclusion relationship. Specifically, this unit is configured to check whether all its ancestor labels are also predicted as positive examples for each predicted child label; if a child label is predicted as a positive example but its parent label is not predicted, the parent label is forced to be added to the prediction result or the prediction probability of the child label is set to zero.

[0138] When the model reasoning module 600 is working, it generates hierarchical multi-label attribution results (including label IDs, label names, and prediction probabilities at each level) and a set of key evidence fragments (including text fragment content and corresponding label associations), which can be output to users or downstream business systems.

[0139] The following section uses book hierarchical multi-label classification as an example to explain in detail the application and implementation process of this invention, including the system composition and deployment, the detailed process of hierarchical multi-label attribution, and the technical features and effects of the entire solution.

[0140] I. System Composition and Deployment

[0141] In this embodiment, a hierarchical multi-label attribution system integrating atomic rule-driven trusted features and knowledge distillation is deployed on a standard machine learning and model service architecture. During the training phase, servers equipped with multiple high-performance GPUs are used for teacher model inference and student model training; during the inference phase, edge servers equipped with a single GPU are used for student model deployment. Each module is deployed as a microservice, with asynchronous data transfer between modules via message queues, supporting batch processing of large-scale samples. The data preparation module and teacher model inference module are deployed on the training server. After preparing the training data and performing teacher model inference, the output soft labels and features are stored in a distributed file system. The trusted feature selection module and student model building module run on the same server. After feature selection and model building, the model distillation training module trains the model, and the final student model is exported in a general format and deployed to the model inference module on the inference server, providing API services for business systems to call.

[0142] II. Detailed Implementation Process

[0143] See Figure 3 In this embodiment, we take the book hierarchical multi-tag classification as an example. The tag system has situations where the subclasses have the same name but different parent classes. For example, the "Biography" subclass under the "History" first-level tag has the same name as the "Biography" subclass under the "Social Sciences" first-level tag, but the semantic emphasis is different.

[0144] Combination Figure 3 As shown, the specific implementation process of this embodiment includes the following steps:

[0145] 1. Data preparation. This includes label system construction, sample collection and data preprocessing, atomic rule base construction, and atomic rule-driven weak labeling.

[0146] Tag system construction. Based on library classification standards, publisher classification systems, or custom business classifications, construct a multi-level tag system. The first-level tag set is denoted as:

[0147] (1)

[0148] For example, literature (L1), history (L2), philosophy (L3), social sciences (L4), etc. The set of secondary labels is denoted as:

[0149] (2)

[0150] Configure the parent tag ID for each second-level tag. For subclasses with the same name, use different internal IDs to distinguish them. For example, L2-3 represents History - Biographies, and L4-3 represents Social Sciences - Biographies. Their parent-child relationship mapping is as follows:

[0151] (3)

[0152] The subclass finds the parent class through the mapping function. The set of three-level tags is denoted as:

[0153] (5)

[0154] For each third-level label, both its second-level and first-level parent labels are recorded. For example, L2-3-1 represents History - Biographies - Political Figures Biographies, L2-3-2 represents History - Biographies - Scientists Biographies, and L4-3-1 represents Social Sciences - Biographies - Social Movement Figures Biographies. Then, one or more directed acyclic graphs can be constructed using the hierarchical parent-child relationships and stored as adjacency lists or parent-child index matrices for subsequent hierarchical constraint modeling. Finally, structured information is configured for each label node: {Label ID; Chinese name; Label description; List of parent label IDs; Optional: Set of keywords related to the label, examples of typical text fragments, etc.}.

[0155] Sample collection and data preprocessing. A sample instance is constructed for each book, denoted as:

[0156] (6)

[0157] in, For the book title, For Rong's introduction, For the table of contents, This includes metadata (publisher, publication date, author, etc.). Preprocessing includes text cleaning, unified encoding, and simplified / traditional Chinese conversion. Text cleaning includes removing HTML tags, special control characters, and redundant spaces.

[0158] Atomic rule base construction. Domain expert experience, book classification rules, and typical expression patterns are broken down into atomic rules. Each atomic rule... Formalized as:

[0159] (7)

[0160] in, These are the rule triggering conditions, including keyword matching, regular expressions, phrase templates, simple syntax patterns, etc. It is the set of target tags associated with this rule. These are rule weights or confidence scores, used for subsequent weak annotation fusion. For example, rules... :{ The text contains keyword combinations such as "a certain historical period" or "the life of a certain historical figure." ; : 0.8.}; Rules :{ The text contains keywords such as "a certain entrepreneur" and "interaction between individuals and the market / institutional environment." ; : 0.8.}; The specific text fragments matched by the rule will also be recorded as evidence for subsequent credible feature mining and interpretable output.

[0161] Atomic rules drive weak labeling. For each book traverse its text segments With rule base Perform rule matching: if the text segment Satisfy the rules Triggering conditions If it is, then it is considered a hit: Record the rule ID, text segment position, and other information of the matched rule. For each tag... Aggregate the scores of all rules that match the label to construct a weak label score:

[0162] (8)

[0163] in, For the Sigmoid function, This is an indicator function. Then, for each book sample... This yields a rule-based weakly labeled vector. According to the preset threshold Generate a set of candidate tags:

[0164] (9);

[0165] in, For the candidate tag set, For any tag in the hierarchical tagging system, For weakly labeled scores, The preset threshold;

[0166] And retain the set of evidence fragments that match the rule. Manual labeling was performed on some samples to create manually labeled tags. This serves as a hard label for subsequent training.

[0167] 2. Teacher Large-Scale Model Inference and Hierarchical Feature Extraction. The teacher large-scale model uses DeepSeek-R1, combining cue word engineering and weak labeling to generate high-quality soft labels and intermediate features. This includes model input construction, cue word design, and soft label and intermediate feature extraction.

[0168] Model input construction and prompt word design. For each book sample Build a teacher model input text This includes: some original content (title, introduction, key chapter titles, etc.) and candidate tags for weak labeling based on rules. and corresponding label descriptions, rule evidence fragments A summary of representative short phrases and tag hierarchy (explanation of the relationship between first-level, second-level, and third-level tags).

[0169] Organized in a structured prompt template, for example: "Based on the following book information, candidate tags, and tag definitions, determine the attribution of this book to the first-level / second-level / third-level tags, and give the probability (0–1) of each tag, as well as the main basis fragment number."

[0170] Teacher model for input Perform reasoning and output: the probability distribution of each level of label (soft label), the judgment of whether it is reasonable or not, and the intermediate feature representation (vector representation of the text).

[0171] Soft labeling and intermediate feature extraction. Define the teacher model's output at each level, including the first-level label probability. Secondary label probability Level 3 tag probability Meanwhile, the teacher model is a text segment. Output deep semantic representation This forms an intermediate feature matrix:

[0172] (10)

[0173] It can also embed tags. This is used to analyze the correlation between tags and text fragments.

[0174] (11)

[0175] in, This is the inner product scoring function.

[0176] 3. Trustworthy Feature Selection. Stable and reliable soft labels and features are selected from the teacher model output to serve as a trustworthy source for subsequent knowledge distillation. This includes soft label trustworthiness assessment and trustworthy feature dimension selection.

[0177] Soft label credibility assessment. For each sample And label l, combined with teacher probability (* represents level 1 / 2 / 3), weak rule annotation , and manual hard labels The consistency and other reliable features are used for screening.

[0178] Using a soft-label confidence function:

[0179] (12)

[0180] in, These are weight parameters that can be tuned based on the validation set. When the confidence level exceeds a certain threshold, i.e. At that time, it was believed that the teacher was responsible for each sample. If the label l is predicted to be trustworthy, it is included in the set of trustworthy soft labels.

[0181] Confidential feature dimension selection. Intermediate semantic representation of teachers. and tag embedding This involves statistically analyzing the correlation across a large sample size. The direction of label correlation is calculated as follows:

[0182] (13)

[0183] in, This is for statistical sample size analysis. The main dimensions were analyzed using methods such as variance contribution and importance scoring to obtain a subset of feature dimensions highly correlated with label l. For all labels, collect their feature dimension sets and obtain the reliable feature dimension set:

[0184] (14)

[0185] 4. Construct a student model with a hierarchical output structure. The student model uses Chinese-RoBERTa-base as the encoder and constructs a structure for hierarchical multi-label output. This includes text encoding, hierarchical label embedding, and reliable feature space alignment.

[0186] Text encoding. Book samples. Each text segment Input student encoder :

[0187] (15)

[0188] Aggregate the entire sample to obtain a sample-level representation. Employing attention convergence:

[0189] (16)

[0190] (17)

[0191] in These are learnable parameters.

[0192] Hierarchical tag embedding. Construct a tag embedding matrix for the student side for all tags:

[0193] (18)

[0194] in, The prediction of first-level, second-level, and third-level labels adopts a hierarchical structure. For example, second-level label prediction:

[0195] (19)

[0196] Trustworthy feature space alignment. A feature subspace is reserved at a certain layer of the student encoder to align with the teacher's trustworthy feature dimensions. Specifically, student features can be projected onto the teacher's dimensional space using a linear mapping:

[0197] (20)

[0198] in, This is a learnable matrix. Further details will follow. and In terms of credibility Align it with the top.

[0199] 5. Distillation Training. When training the student model, hard labels, reliable soft labels, and reliable features are used simultaneously to construct a joint loss function. This includes hard label loss, soft label distillation loss, feature distillation loss, and the joint loss during training.

[0200] Hard-label loss. For a set of labeled samples... Multi-label binary cross-entropy loss is used at each level. For example, at the second-level label level:

[0201] (twenty one)

[0202] Similar to level 1 and level 3 tags, the overall hard tag loss is:

[0203] (twenty two)

[0204] Soft-label distillation loss. Distillation is performed on all samples using teacher-trusted soft labels. Taking second-level labels as an example, distillation is performed on the set of trusted labels. The labels are calculated using KL divergence:

[0205] (twenty three)

[0206] in, .

[0207] Feature distillation loss. During the feature distillation stage, the focus is on aligning the teacher's trustworthy feature dimensions. For each sample and each text segment, minimize the L2 distance along these dimensions:

[0208] (twenty four)

[0209] in, It is the component of teacher characteristics in dimension d. It is the component of the student's mapped features in dimension d.

[0210] Joint Loss and Training. Combining the above losses, the overall training objective is constructed as follows:

[0211] (25)

[0212] in, These are hyperparameters used to balance the influence of different loss terms. During training, the Adam optimizer is used for batch gradient descent, iteratively updating the student model parameters and mapping parameters on the training set. Tag embedding Wait until convergence on the validation set or the early stopping condition is met.

[0213] 6. Model Inference. Attribution results are output by deploying a pre-trained student model. This includes label predictions and evidence fragment outputs.

[0214] Tag prediction. For the new book x to be predicted, it is preprocessed and segmented into text segments. Input the student encoder to get With sample representation Calculate the predicted probabilities for first-level, second-level, and third-level labels respectively. For each level, a final tag set is generated based on a threshold or Top-k rule. For example, a two-level tag level:

[0215] (26)

[0216] For the same subclasses, History-Biography (L2-3) and Social Science-Biography (L4-3), each has its own predictive probability. The three-level labels are used for prediction under the premise of satisfying the parent-child consistency constraint.

[0217] Evidence fragments. To improve the interpretability of the classification results, this embodiment utilizes the student model's internal attention or fragment scoring mechanism to assess the importance of each text fragment:

[0218] (27)

[0219] in, This can be an inner product or a small feedforward network. For each predicted label l, select the highest-scoring segments as evidence:

[0220] (28)

[0221] in, Let l be the set of key evidence fragments, and l be the predicted label. Let j be the j-th text segment after the input text has been segmented. Assess the importance of a text segment with a score.

[0222] In summary, for the predicted social science biographies, the following evidence fragments may be selected: {"Examining the life of a certain person from the perspective of social structure and institutional change"; "Analyzing the interaction between individuals and social class mobility and policy environment"}. The final output includes: {Predicted results and probabilities for each level of label; key evidence fragments.}

[0223] To address the aforementioned hierarchical multi-label attribution method that integrates atomic rule-driven trusted features and knowledge distillation, this invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the aforementioned hierarchical multi-label attribution method.

[0224] This invention also provides a processor for running a program, wherein the program executes the steps of the hierarchical multi-label attribution method described above.

[0225] This invention also provides a terminal device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. The program code is loaded and executed by the processor to implement the steps of the above-described hierarchical multi-label attribution method.

[0226] The present invention also provides a computer program product, which, when executed on a data processing device, is adapted to perform the steps of the above-described hierarchical multi-label attribution method.

[0227] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0228] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A hierarchical multi-label attribution method that integrates atomic rule-driven credible features and knowledge distillation, characterized in that, Includes the following steps: Step 1, Data Preparation: Construct a hierarchical tagging system containing multi-level parent-child tags; construct an atomic rule library consisting of several atomic rules, each atomic rule containing the rule triggering condition and its associated target tag; use the atomic rule library to automatically match the text, generate candidate tags according to the matched rules, and extract the text fragments that trigger the rules as evidence fragments of the corresponding tags to form a weakly labeled sample set. Step 2, Teacher Model Inference: Using a large language model as the teacher model, the text and the weakly labeled samples are constructed as task instruction prompts and input into the teacher model. The soft label probability distribution of the teacher model output on each level of labels, as well as the intermediate layer feature representation generated during the inference process, are obtained. Step 3, Trustworthy feature selection: The consistency, stability and business relevance of the soft label probability distribution and intermediate layer feature representation output by the teacher model are evaluated, and trustworthy soft labels and trustworthy feature dimension subsets that meet the preset trust thresholds are selected respectively. Step 4, Constructing a student model: Construct a student model using a lightweight pre-trained model. The top layer of the student model has a hierarchical output structure that matches the hierarchical labeling system, and its intermediate layers reserve a feature space that is aligned with the subset of the credible feature dimensions. Step 5, Model Distillation Training: Construct a joint training objective function, and use labeled data, trusted soft labels, and trusted features to perform end-to-end training on the student model through backpropagation to minimize the joint training objective function; Step 6, Model Inference: Load and deploy only the trained student model, output the probability distribution of the text to be processed on each level of label, select the final attribution label according to the preset threshold, and combine the learning of the relevant features of the evidence fragments during training to output one or more fragments with the highest correlation with each label as key evidence sentences.

2. The method according to claim 1, characterized in that, The specific method for constructing the atomic rule base in step 1 is as follows: domain expert experience, classification rules, typical expression pattern rules, tag definition rules, or keyword table rules are decomposed into atomic rules. Each atomic rule is formalized as a triggering condition, an associated set of target tags, and a rule weight or confidence level. The associated set of target tags is a subset of tags at one or more levels in the hierarchical tag system.

3. The method according to claim 2, characterized in that, The specific method for generating weakly labeled samples using the atomic rule base in step 1 is as follows: traverse the atomic rule base for each text segment in the text and match them. If a text segment meets the triggering condition of a certain atomic rule, it is considered a hit. For each label, aggregate the weights of all rules that hit the label and convert them into weak label scores. Generate a candidate label set according to a preset threshold, and retain the text segments that hit the rules and their corresponding rules and labels as a set of evidence fragments. At the same time, manually label some samples to form manually labeled labels as hard labels for subsequent training.

4. The method according to claim 1, characterized in that, The credible feature selection in step 3 includes: First, combining the weak annotations generated by atomic rules, performing consistency analysis on the teacher model prediction results, and selecting stable label predictions as credible soft labels; Second, evaluating the feature importance of the teacher intermediate representations, and selecting a subset of features with stable importance and high correlation with the correct labels as credible feature dimensions; Then, combining the consistency analysis results, the degree of consistency with the annotations, and the feature importance stability index, selecting each level, each label, and the corresponding feature dimension, and retaining only the soft label distribution and features that reach the preset credibility threshold as credible soft labels and credible features, thus forming a credible feature library.

5. The method according to claim 1, characterized in that, In step 4, the hierarchical output structure of the student model maps the hierarchical labels to a unified vector space and explicitly encodes the parent-child relationship through the parent-child constraint structure, enabling the model to output multi-level, multi-label probability distributions simultaneously.

6. The method according to claim 1, characterized in that, The joint training objective function in step 5 includes hard label loss, soft label loss, and feature distillation loss. The hard label loss is calculated based on the student model's prediction results after manual verification of weak labels, ensuring the student model can directly fit high-quality annotations. The soft label loss uses KL divergence to calculate the loss between the probability distribution output by the student model and the soft labels, guiding the student model to learn the fine-grained probability structure of the teacher model within the trustworthy region. The feature distillation loss establishes an alignment relationship between the intermediate layer of the student model and the trustworthy feature dimensions of the teacher model, minimizing the distance between them in the corresponding dimensions to transfer stable and reliable discriminative features from the teacher model to the student model. The hard label loss, soft label loss, and feature distillation loss are combined in a weighted manner to form the joint training objective function.

7. A hierarchical multi-label attribution system that integrates atomic rule-driven credible features and knowledge distillation, characterized in that, include: The data preparation module (100) is configured to construct a hierarchical label system containing multi-level parent and child labels; construct an atomic rule library consisting of several atomic rules, each atomic rule containing the rule triggering condition and its associated target label; use the atomic rule library to automatically match the text, generate candidate labels according to the hit rules, and extract the text fragments that trigger the rules as evidence fragments of the corresponding labels to form a weakly labeled sample set. The teacher model reasoning module (200) is configured to use a large language model as the teacher model, construct the text and the weakly labeled samples as task instruction prompts and input them into the teacher model, obtain the soft label probability distribution of the teacher model output on each level of labels, and the intermediate layer feature representation generated during the reasoning process; The credible feature filtering module (300) is configured to evaluate the consistency, stability and business relevance of the soft label probability distribution and intermediate layer feature representation output by the teacher model, and respectively filter out credible soft labels and credible feature dimension subsets that meet the preset credible threshold. The student model building module (400) is configured to build a student model using a lightweight pre-trained model. The top layer of the student model has a hierarchical output structure that matches the hierarchical label system, and its intermediate layers reserve a feature space that is aligned with the subset of the credible feature dimensions. The model distillation training module (500) is configured to construct a joint training objective function and, using labeled data, trusted soft labels and trusted features, perform end-to-end training on the student model through backpropagation to minimize the joint training objective function; The model reasoning module (600) is configured to provide the trained student model with the probability distribution of the text to be processed on each level of labels, select the final attribution label according to a preset threshold, and combine the learning of the relevant features of the evidence fragments during training to output one or more fragments with the highest correlation with each label as key evidence sentences.

8. The system according to claim 7, characterized in that, The teacher model inference module (200) includes: an input construction unit for constructing task instruction prompts from text and weakly labeled samples; a soft label extraction unit for obtaining the soft label probability distribution output by the teacher model at each level of labels; and an intermediate feature extraction unit for obtaining the intermediate layer feature representations generated by the teacher model during the inference process, including the deep semantic vectors of the text segments and the label embedding representations.

9. The system according to claim 7, characterized in that, The credible feature screening module (300) includes: a soft label credibility evaluation unit, which is used to calculate credibility scores and screen credible soft labels by integrating teacher model prediction probability, rule weak label scores and manual hard label consistency information; and a feature dimension screening unit, which is used to statistically analyze teacher intermediate layer feature dimensions that are highly correlated with each label to form a subset of credible feature dimensions.

10. The system according to claim 7, characterized in that, The model distillation training module (500) includes: a hard label loss calculation unit for calculating the binary cross-entropy loss between student model predictions and manual annotations; a soft label distillation loss calculation unit for calculating the KL divergence loss between student model predictions and credible soft labels, using the credibility score as the weight; a feature distillation loss calculation unit for calculating the mean squared error loss between student-mapped features and teacher-original features on a subset of credible feature dimensions; and a joint optimization unit for weighted summation of the three losses to form a total loss and performing backpropagation optimization on the model accordingly.

Images