A knowledge base based model distillation method and apparatus

Abstract

The application provides a knowledge base-based model distillation method and device, and belongs to the technical field of model distillation. The method provided by the application comprises the following steps: extracting features of each layer from a plurality of pre-trained large models to form a distillation knowledge base; determining task content corresponding to a student model; matching target knowledge from the distillation knowledge base based on the task content, wherein the task content comprises a plurality of ordered task links, each task link corresponds to at least one target knowledge, the task description of the task link is matched with a plurality of task types to determine target features under a target task type, and the knowledge corresponding to the target features is determined as the target knowledge; and guiding the training of the student model based on the target knowledge, wherein the size of the student model after training is smaller than that of the large model corresponding to the target knowledge. The knowledge base-based model distillation method and device provided by the application are used to provide a targeted distillation strategy, so that a lightweight model can better adapt to an actual application scenario.

Description

Technical Field

[0001] This application relates to the field of model distillation technology, and in particular to a knowledge base-based model distillation method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence technology in the field of Natural Language Processing (NLP), large models, with their massive number of parameters and complex network structures, have demonstrated outstanding performance in tasks such as text understanding, semantic analysis, and dialogue generation. They have been widely applied in scenarios such as intelligent customer service, data platform interaction, and domain knowledge question answering. Especially in professional fields such as power and finance, the demand for accuracy and adaptability of large NLP models continues to increase. However, in the actual deployment process, large models face problems such as high consumption of computing resources, high storage consumption, and slow inference response, making them difficult to directly apply to edge devices or resource-constrained business systems.

[0003] Currently, distillation methods for large NLP models have a certain research foundation. These methods primarily utilize a teacher-student architecture to transfer knowledge from a large model (teacher model) to a smaller model (student model). Common distillation approaches include response-based distillation (based on soft labels), feature-based distillation (based on feature mapping), and relation-based distillation (based on relation matching). Simultaneously, some technical solutions incorporate model pruning and quantization compression techniques to further reduce the computational and storage costs of the smaller model. Furthermore, recording some data during the distillation process assists in subsequent model optimization. In addition, many related systems employ a B / S architecture to build user interfaces, supporting users in uploading datasets, configuring distillation parameters, and viewing training results to meet basic distillation task requirements.

[0004] However, existing technologies still have significant shortcomings. On the one hand, the knowledge extraction and management processes are incomplete. Most solutions can only extract single types of knowledge, such as the probability distribution of large models, and lack effective extraction and storage of key knowledge such as attention weights and intermediate layer features. Furthermore, they have not built a reusable knowledge base containing structured attributes such as knowledge base ID, level, and creation information, and cannot support users to customize the selection of knowledge to be transferred, resulting in insufficient targeting and reusability of knowledge transfer. On the other hand, the traceability and experimental comparison capabilities of the distillation process are lacking. Existing solutions do not systematically record key data such as configuration parameters, model version information, and evaluation results of distillation experiments, making it difficult to achieve horizontal comparative analysis of different distillation strategies and backtracking optimization of historical experiments. This makes it difficult to meet the requirements of professional fields for system stability and ease of operation. Summary of the Invention

[0005] In view of this, this application provides a knowledge base-based model distillation method and apparatus to provide targeted distillation strategies, enabling lightweight models to better adapt to real-world application scenarios.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a knowledge base-based model distillation method, the method comprising:

[0008] Features from each layer are extracted from multiple pre-trained large models to form a distillation knowledge base, wherein each piece of knowledge in the distillation knowledge base corresponds to a feature, and the multiple pre-trained large models correspond to multiple task types.

[0009] Determine the task content corresponding to the student model;

[0010] Based on the task content, target knowledge is matched from the distillation knowledge base. The task content includes multiple ordered task steps, each task step corresponds to at least one target knowledge. The task description of the task step is matched with multiple task types to determine the target features under the target task type. The knowledge corresponding to the target features is determined as the target knowledge.

[0011] The student model is trained based on the target knowledge, and the size of the trained student model is smaller than the large model corresponding to the target knowledge.

[0012] A second aspect of this application provides a knowledge-based model distillation apparatus, the apparatus comprising a construction module, a determination module, and a training module; wherein,

[0013] The construction module is used to extract features from each layer of multiple pre-trained large models to form a distillation knowledge base, wherein each piece of knowledge in the distillation knowledge base corresponds to a feature, and the multiple pre-trained large models correspond to multiple task types.

[0014] The determining module is used to determine the task content corresponding to the student model;

[0015] The training module is used to match target knowledge from the distillation knowledge base based on the task content, wherein the task content includes multiple ordered task steps, each task step corresponds to at least one target knowledge, the task description of the task step is matched with multiple task types to determine the target features under the target task type, and the knowledge corresponding to the target features is determined as the target knowledge.

[0016] The training module is also used to guide the training of the student model based on the target knowledge, and the size of the trained student model is smaller than the large model corresponding to the target knowledge.

[0017] The knowledge-based model distillation method and apparatus provided in this application achieve model distillation based on a knowledge base through an interactive interface. This means the model distillation process can be interactively intervened through a software interface. Furthermore, model distillation does not require a fixed set of parameters; it can be divided into matching distillation knowledge with each feature and level, refining the granularity of the model distillation parameters and improving the distillation effect. Specifically, by extracting features from each layer of pre-trained large models corresponding to multiple task types to construct a distillation knowledge base, the limitations of single task and single model knowledge are broken, providing multi-source and structured knowledge support for student model training. First, the student model task content is determined, and then target knowledge is matched to guide the student model training, enabling precise positioning of the key features required for the task, avoiding interference from irrelevant knowledge, and significantly improving the distillation's relevance. Simultaneously, the size of the trained student model is smaller than the large model corresponding to the target knowledge, solving the problems of high computational resource consumption, high storage usage, and slow inference response when deploying large models. This makes it adaptable to edge devices or resource-constrained business systems, enhancing the practical application adaptability of lightweight models. Attached Figure Description

[0018] Figure 1 A flowchart of an embodiment of the knowledge-based model distillation method provided in this application;

[0019] Figure 2 This is a schematic diagram of the structure of Embodiment 2 of the knowledge base-based model distillation apparatus provided in this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0023] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0024] Figure 1 This is a flowchart of an embodiment of the knowledge-based model distillation method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0025] S101. Extract features from each layer of multiple pre-trained large models to form a distillation knowledge base, wherein each piece of knowledge in the distillation knowledge base corresponds to a feature, and the multiple pre-trained large models correspond to multiple task types.

[0026] Specifically, at least two pre-trained large models belonging to different task types are selected. All features (including word vector features, syntactic features, semantic features, etc.) from the input layer to the output layer of each pre-trained large model are extracted. A unique identifier is assigned to each feature and associated with the task type information of its source model. These are then stored in a structured manner to form a distillation knowledge base. For example, in one embodiment, task types may include basic text understanding, semantic generation and interaction, and vertical domain-specific task types. Basic text understanding may include text classification: such as power fault work order classification (classifying work orders according to fault types such as "equipment short circuit," "line overload," and "signal anomaly"), named entity recognition: such as power field equipment entity recognition (extracting "transformer model," "switchgear number," and "transmission line name" from the text), and text matching: such as power dispatch instruction consistency verification (determining whether two dispatch instructions have logical conflicts). Semantic generation and interaction may include question-and-answer generation: such as power field intelligent customer service question-and-answer (generating corresponding answers based on "equipment tripping reasons"), and text summarization: such as power fault report summarization (from long texts). Extracting core information such as "fault time, location, cause, and solution" from fault handling records and generating dialogues: such as power operation and maintenance assistant dialogues (interacting with operation and maintenance personnel to gradually investigate the "abnormal temperature in the substation" problem); vertical domain special task types can include power dispatch instruction parsing: breaking down instructions such as "10kV Chengdong Line #3 tower switch tripping operation, safety measures check must be completed before 15:00" into "operation object (10kV Chengdong Line #3 tower switch), operation action (tripping), time requirement (before 15:00), prerequisite task (safety measures check)"; financial risk text warning: identifying text fragments that may cause financial risks, such as "risk of broken capital chain" and "illegal guarantee information" from corporate annual reports and news reports, and marking the risk level, etc.

[0027] Furthermore, the features of the pre-trained large model refer to the parameters and related data that reflect the core capabilities of a specific layer of the pre-trained large model. They are the key information carriers learned by the large model during training and used to adapt to the corresponding task. Each feature corresponds to an independent piece of knowledge in the distillation knowledge base. Features include at least the model's basic parameter features (such as word vector parameters, network layer weight parameters, and layer normalization parameters), attention mechanism-related features (such as self-attention distribution, cross-attention features, etc.), intermediate layer semantic and structural features, and task decision layer features.

[0028] Furthermore, pre-trained large models refer to deep learning models that have undergone initial training on large-scale general data through unsupervised or self-supervised learning methods, possessing strong general capabilities and typically having hundreds of millions or even trillions of parameters. After pre-training, these large models have already developed the ability to model general language rules and semantic logic. They do not need to be trained from scratch for specific tasks. The pre-trained large models can be used as a base model to further adapt to different scenarios. For example, in the field of power information, the pre-trained large models can be fine-tuned using data from areas such as power dispatch text and equipment fault descriptions to give them power-related semantic understanding capabilities. It should be noted that multiple pre-trained large models cover multiple task types (such as text classification, named entity recognition, question answering generation, etc.). Their core role is as a knowledge source. By extracting features at various levels (such as word vector features, syntactic features, and semantic features) to construct a distilled knowledge base, they provide multi-source, structured knowledge support for the targeted training of subsequent student models, avoiding performance shortcomings caused by insufficient data or low training starting points in student models.

[0029] Furthermore, leveraging the feature extraction capabilities of pre-trained large models for specific tasks, feature representations at each layer can be obtained through the output interfaces of the model's intermediate layers. For example, word vector features from the embedding layer, local features from the convolutional layer, and abstract features from the fully connected layer can be extracted from a text classification model; semantic encoding features from the encoder and generative features from the decoder can be extracted from a machine translation model. Feature extraction interfaces (such as PyTorch's hook mechanism) can be used to intercept the output tensors of each layer of the model, which, after standardization, can be used as knowledge units. Each knowledge unit contains metadata such as feature vector data, source model ID, layer number, and corresponding task type label. It's important to note that many pre-trained large models are trained using dedicated data specific to their respective task types. For example, in basic text understanding, the text classification model (e.g., power fault work order classification, financial customer complaint classification) uses "text-category label" data, while the named entity recognition model (e.g., power equipment recognition, financial entity recognition) uses standardized labeled entity data. In semantic generation and interaction, the question-answering generation model (e.g., power customer service Q&A, financial knowledge Q&A) uses "question-answer-reference text" triples, while the text summarization model (e.g., power fault report summary, financial statement summary) uses "raw text-manually labeled summary" pairs. In vertical domain-specific tasks, the power dispatch instruction parsing model uses "instruction text-structured elements" data, and the financial risk text warning model uses "text fragment-risk type-risk level" data. Dedicated data must meet the requirements of data-task target alignment, inclusion of domain-specific information, and a certain scale and high-quality labeling to ensure that the features of each layer of the large model accurately adapt to the core capabilities of the corresponding task, providing a high-quality knowledge source for the distillation knowledge base and supporting subsequent target knowledge matching.

[0030] Specifically, the process begins by selecting large pre-trained models (such as text classification, named entity recognition, and question answering systems) that cover the main tasks of the target domain. Features are then extracted from each model and layer by layer using feature extraction tools, and their quality is verified. Finally, the data is organized, and a fast retrieval index is established to form a distilled knowledge base.

[0031] Furthermore, it can be understood that the distillation knowledge base contains multiple pieces of knowledge, each corresponding to a feature from each layer of features in different pre-trained large models. The distillation knowledge base is composed of multiple pre-trained large models, multiple layers within a single pre-trained large model, and multiple features at each layer. A pre-trained large model corresponds to a task type, which refers to the core task that the pre-trained large model was originally adapted to. For example, text classification tasks (such as power fault work order classification), named entity recognition tasks (such as recognizing equipment models and fault types in power text), question-answering generation tasks (such as intelligent question answering for power customer service), and text matching tasks (such as consistency verification of power dispatch instructions). Each task type corresponds to a large model that solves a specific problem.

[0032] In this way, by aggregating the hierarchical features of multiple task-based large models, the knowledge limitations of a single model can be broken, enabling the distillation knowledge base to have cross-task knowledge transfer capabilities. Furthermore, each piece of knowledge corresponds to an independent feature, achieving refined knowledge management and flexible retrieval. Multi-task type coverage ensures adaptability to different student model tasks, laying the foundation for subsequent accurate matching. Compared with single-source knowledge distillation, it improves the generalization ability of student models.

[0033] Furthermore, the implementation steps for constructing the distillation knowledge base include:

[0034] (1) For each task type, select multiple pre-trained large models as knowledge sources, and extract all features from the input layer to the output layer of each pre-trained large model. The features include model parameters and attention weights.

[0035] Specifically, for each task type, select no fewer than two pre-trained large models as knowledge sources, and extract all features from the input layer to the output layer of each model using model parsing tools. The feature set must include the parameter matrix (weights and biases) of each layer of the model and the weight distribution data generated by the attention mechanism, and the extraction process must maintain the mapping relationship between the features and the corresponding layers.

[0036] Furthermore, multiple suitable pre-trained large models are matched for each task type. For example, for text classification tasks, BERT-large, RoBERTa, XLNet, etc. can be selected. The selected models are required to have an F1 score of ≥0.85 on the standard test set of the task type, and the model structure must include a Transformer encoder layer (to ensure that the attention mechanism is resolvable). The attention weights of each layer are obtained, and the extraction granularity is set to a three-dimensional structure of "layer-head-position" (such as the weight value of the second attention head of the third layer for the fifth token of the text). The extracted features are verified in two rounds. The first round checks the completeness of the parameter dimensions, and the second round passes the model reconstruction test (the model is reconstructed using the extracted parameters, and the output results are verified to be ≥99.9% consistent with the original model).

[0037] By extracting features from multiple large models, we can cover different model representation perspectives for the same task type. The feature set contains the complete cognitive chain of the model from basic semantics to task decision-making, providing a redundant verification basis for subsequent knowledge fusion.

[0038] (2) Construct a knowledge architecture with task type as the main node and hierarchical information as the branch node, which is the same as the hierarchical relationship of the model structure;

[0039] Specifically, the task type is used as the root node of the knowledge architecture. Multi-level branch nodes are constructed according to the hierarchical structure of the pre-trained large model (input layer → hidden layer → output layer). The hierarchical division must correspond one-to-one with the actual network structure of the model (such as "Embedding layer → Encoder layer 1 → ... → Encoder layer N → output layer" in the Transformer model). Each branch node must contain metadata such as hierarchical number, functional description, and associated model type.

[0040] Furthermore, based on the task type classification system, a top-level master node is created for each task type, such as text classification, named entity recognition, and question answering generation. Each master node needs to be associated with the standard test dataset commonly used for that task, as well as the core indicators for evaluating model performance (such as the accuracy of text classification and the F1 score of named entity recognition), to facilitate subsequent evaluation of knowledge quality. Branch nodes are constructed according to the forward propagation order of the model. Taking a model with an encoding layer as an example, the branch node chain is as follows: input layer sub-nodes for word encoding, position encoding, and sentence encoding → first encoding layer sub-nodes for attention calculation and feature transformation → second encoding layer (structure same as the first) → ... → output layer sub-nodes for feature summarization and result calculation, resulting in multiple branch nodes. Using a structure comparison tool, check whether the number of levels and the order of levels in the knowledge architecture are completely consistent with the corresponding model, ensuring that each level of the model has a unique corresponding node in the architecture, and that the order of the levels is consistent with the model's data processing flow. For example, the output node of encoding layer 1 must correspond to the input node of encoding layer 2.

[0041] (3) All features are classified into the knowledge architecture to form a triplet of knowledge consisting of main nodes, branch nodes and feature content, and stored in the distillation knowledge base.

[0042] Specifically, all extracted features are categorized according to a triplet structure of main node (task type) - branch node (hierarchical information) - feature content. Each triplet needs to satisfy the hierarchical function matching between the feature content and the branch node, and generate a unique knowledge ID through a hash algorithm. A distributed database is used to store triplet knowledge, supporting multi-dimensional retrieval by task type, hierarchical index, and feature type.

[0043] Furthermore, an automatic classification rule engine is established to clarify the attribution of different types of features. For example, numerical vectors representing word semantics are classified into word encoding sub-nodes in the input layer, and weight matrices reflecting the correlation strength of different parts of the text are classified into attention calculation sub-nodes in the corresponding encoding layer. The automatic matching accuracy of the rule engine needs to reach over 98%, and features that fail to match are subject to manual review to ensure that all features can be correctly classified. Each feature is organized into a triplet form of "task type-hierarchical path-feature content". For example, for the attention weight feature of the encoding layer 2 of a certain model in the named entity recognition task, the triple can be represented as "named entity recognition", "encoding layer 2 / attention calculation", {"feature value": [...], "source model identifier": "model A", "extraction time": "September 14, 2024"}, fully recording the key information of the feature; features classified into the same task type and the same level node are deduplicated. If the features extracted by different models are highly similar (similarity not less than 99%), they are merged into one knowledge item, and the number of their source models is recorded; if the features are complementary (e.g., different models capture different semantic focuses at the same level), association tags are established for these features to form a correspondence table of features and complementary features, which is convenient for subsequent comprehensive use; a database dedicated to storing vector data is used to store the specific numerical values ​​of the features, supporting fast calculation of the similarity between features; at the same time, a relational database is used to store the basic information of the triple (such as task type, level path, knowledge identifier), and the two databases can be linked for querying through an interface. This ensures that the response time of a single knowledge query is short and can support multiple concurrent query requests per second, meeting the needs of efficient retrieval.

[0044] Standardized feature storage can assign a unique identifier to each knowledge unit, allowing traceability of the original model, extraction time, and other information, facilitating knowledge management and problem identification. It also supports flexible multi-dimensional retrieval, enabling quick location of required knowledge. Furthermore, it provides an efficient data interface for knowledge matching in the subsequent distillation stage, reducing matching time. Ultimately, it forms a visualized knowledge relationship graph, which can intuitively display the relationships between task types, model levels, and feature content, aiding in the understanding of knowledge structures.

[0045] In addition, the distillation knowledge base also includes:

[0046] (1) Classify according to task type, and each category is allocated an independent task type storage unit in the distillation knowledge base;

[0047] Specifically, based on task type as the core dimension, the storage space for the distillation knowledge base is divided into several independent categories. Each category serves as an independent storage unit, containing metadata for that task type (such as task description, typical application scenarios, and feature requirement tags). For example, the power dispatch text classification category records task descriptions used to identify the operation type (such as closing or opening) in dispatch instructions, as well as requirement tags that should prioritize operational terminology features and instruction logic features.

[0048] By dividing tasks into top-level categories, the problem of mixed knowledge across multiple domains is solved, enabling subsequent knowledge matching to directly locate the category consistent with the student model task, thus avoiding interference from cross-task irrelevant knowledge.

[0049] (2) Multiple first-level child nodes are set under each task type storage unit. Each first-level child node corresponds to a pre-trained large model. Multiple child nodes are stored in order of high to low according to the performance index of the pre-trained large model on the task type.

[0050] Specifically, under each task type category, a corresponding first-level child node is set for each pre-trained large model. The first-level child node stores the core attributes of the model (such as model name, number of parameters, and performance indicators on the task: accuracy, F1 score, and inference speed). All first-level child nodes are sorted in descending order according to the model's comprehensive performance score on the current task type (such as accuracy weighted at 60% + F1 score weighted at 40%). For example, under the power fault classification category, model A node with a performance score of 92 is ranked before model B node with a performance score of 90.

[0051] The performance ranking mechanism ensures that high-quality model knowledge is retrieved first. When training student models, features from larger models with better performance can be matched first, reducing the impact of low-quality features on the distillation effect and indirectly improving the starting point of student model training.

[0052] (3) Multiple second-level sub-nodes are set under each first-level sub-node, and each second-level sub-node corresponds to a knowledge. The multiple second-level sub-nodes are sorted according to the hierarchical order. Among them, horizontal associations are established between second-level sub-nodes with the same hierarchical number under different first-level sub-nodes, and the knowledge similarity coefficient is marked. All second-level sub-nodes under the same task type are connected through the association degree to form a semantic association network.

[0053] Specifically, multiple second-level child nodes are set under each first-level child node, and each second-level child node corresponds to a knowledge point. They are arranged in the order of model hierarchy (from input layer → embedding layer → encoding layer → output layer). For example, the second-level child nodes of model A are stored in the order of Layer1-word vector features-Layer2-syntactic features-Layer3-semantic features-Layer4-decision features to ensure that the hierarchical logic of features is clear.

[0054] Establishing horizontal associations between second-level child nodes with the same level number under different first-level child nodes is represented as follows: For example, between second-level child nodes with the same level number under different first-level child nodes (different models) (e.g., Layer 3 of model A and Layer 3 of model B), the similarity of knowledge vectors is calculated using cosine similarity, and a similarity coefficient between 0 and 1 is assigned. A semantic association network refers to all second-level child nodes under the same task type category, which are associated through automatically labeled feature tags, forming a network structure.

[0055] Hierarchical sorting allows student models to match knowledge at corresponding levels according to their own network structure, achieving hierarchical alignment distillation and avoiding knowledge transfer losses caused by feature mismatch with network structure. The similarity coefficient of horizontal association provides a reference for knowledge selection; when a model lacks knowledge, features from other models at the same level with high similarity coefficients can be selected as substitutes, ensuring the integrity of knowledge. The semantic association network supports cross-model and cross-level feature association retrieval. For example, when retrieving knowledge related to tripping, features corresponding to all knowledge related to that term in different models and levels can be obtained simultaneously, enhancing the comprehensiveness of knowledge matching. This tree-like storage structure combines structured retrieval efficiency with associative knowledge mining capabilities. On the one hand, through three-level localization of task type-model performance-order, the average time for knowledge matching is shortened; on the other hand, horizontal association and the semantic network break down the knowledge barriers of single models and single levels, enabling student models to aggregate high-quality features from multiple models, providing clear structural support for the dynamic updating of the knowledge base.

[0056] Furthermore, the distillation knowledge base also includes deletion and addition mechanisms. Specifically, the deletion mechanism includes: when the pre-trained large model corresponding to a first-level child node experiences performance failure (e.g., a drop of more than 15% in core metrics on the task's standard test set), data pollution (training data is confirmed to contain mislabeled data), or is completely replaced by a better model of the same task type (performance metrics improved by more than 10%), the first-level child node is marked as pending deletion; when the knowledge corresponding to a second-level child node has extraction errors (e.g., feature values ​​deviate from the original model parameters by more than 5%), is out of touch with the current task requirements (has not been retrieved or used for 6 consecutive months), or has a semantic relevance of less than 0.2 (isolated knowledge), it is marked as pending deletion. The system automatically generates a deletion warning report, including the ID, source, reason for deletion, and scope of impact of related knowledge of the node to be deleted; then, a manual review team (including algorithm engineers and domain experts) reviews the report, and after confirming that there is no risk of dependency, the deletion operation is performed; finally, a deletion log is recorded (including deletion time, executor, and node backup path), and all associations between the node and other nodes are automatically disconnected to avoid affecting the integrity of the knowledge base.

[0057] Furthermore, the deletion mechanism also includes the deletion of associations. Association deletion is triggered when the knowledge similarity coefficient of a horizontal association drops below 0.3 due to knowledge updates (such as optimization of a model feature), or when the association degree between nodes in the semantic association network falls below 0.15 due to changes in task requirements (such as adjustments to the core task objective). The system periodically scans all associations, calculates the current similarity coefficient or association degree, filters out associations that do not meet the threshold, and generates a list. Invalid associations are automatically deleted without manual review, and the association record table is updated. Simultaneously, corresponding connection lines are removed in real-time from the semantic association network visualization interface to ensure that the network structure accurately reflects the knowledge association status.

[0058] Furthermore, the new mechanism includes the addition of first-level sub-nodes and second-level sub-nodes. The addition of a first-level sub-node is triggered when a pre-trained large model with better performance under the same task type (with core indicators exceeding the existing best model by more than 5%), a new special model adapted to special scenarios (such as a model optimized for short texts on power dispatching) is added, or a major update version of the existing model family is released (with significant upgrades in parameter structure or training data). The addition of a second-level sub-node is triggered when the model corresponding to the existing first-level sub-node has unextracted hierarchical features (such as missing attention weights of a certain hidden layer), the trained student model generates new knowledge with stronger adaptability (such as semantic features optimized for financial risk texts), or a blind spot in the coverage of existing knowledge is discovered (such as the lack of output layer decision features under a certain task type).

[0059] After triggering the addition of a first-level child node, the basic information of the new model (number of parameters, training data scale, publishing institution) and performance data (metric scores on the task standard test set and domain-specific dataset) are collected, and features of each layer are extracted. Then, according to the performance index from high to low, the insertion position is determined in the first-level child node sequence of the corresponding task type storage unit, and a new first-level child node is created. Finally, the node is associated with the corresponding second-level child node (initially the extracted feature knowledge of each layer), and the metadata of the storage unit is updated (such as the number of models, the latest update time). After triggering the addition of a second-level child node, the quality of the new knowledge is verified (such as the student model features need to be verified to have a correlation degree ≥ 0.6 with the target task), and the metadata (knowledge source, extraction time, applicable scenario) is supplemented. Then, under the corresponding first-level child node, the position of the new second-level child node is determined according to the hierarchical order (such as inserting it between "level 3 - semantic features" and "level 4 - decision features"). Finally, the horizontal association of the second-level child node with the same level under other first-level child nodes is established (the similarity coefficient is calculated), and it is integrated into the semantic association network (the correlation degree with other nodes is calculated, and corresponding connections are established), completing the addition.

[0060] Furthermore, when fine-tuning the student model for a specific task, the core knowledge types (such as semantic features of the encoding layer and probability distribution features of the output layer) can be determined first based on the fine-tuning requirements (such as optimizing the sentiment recognition accuracy of text classification tasks). Then, the knowledge base retrieval function can be used to filter existing knowledge that meets the requirements by multiple dimensions of "task type - model performance - level range" (such as semantic feature knowledge of levels 5-8 in the top 3 performing models under the text classification storage unit). Finally, a knowledge location list is generated, annotating each knowledge with its ID, source model, core advantages (such as "Model A level 6 features: improves the accuracy of negative sentiment text recognition by 8%), and associations (similar knowledge, strongly associated knowledge) for reference in the fine-tuning scheme. For the located existing knowledge, if its matching degree with the fine-tuning target is ≥0.7 (calculated by feature semantic similarity), it is directly included in the fine-tuning knowledge pool as the core knowledge to guide the training of the student model, without the need for repeated extraction; if the matching degree is between 0.5 and 0.7, the knowledge is reused after adaptive adjustments (such as adjusting feature weights) to ensure that it is adapted to the fine-tuning requirements.

[0061] Furthermore, during the fine-tuning process, if potential connections are found between reused existing knowledge (e.g., the synergistic effect of level 5 features of model A and level 7 features of model B is significant in the fine-tuning task, and the correlation increases), then a connection between the two is added to the semantic association network to mark the correlation in the current fine-tuning scenario; if the newly generated knowledge after fine-tuning (e.g., the level features after the student model optimization) has a strong correlation with existing knowledge (similarity ≥ 0.75), then a derived association label between the new knowledge and existing knowledge is established (e.g., derived from level 5 features of model A), and horizontal associations are added according to the similarity coefficient to enrich the association dimensions between knowledge and achieve network optimization after knowledge reuse.

[0062] S102. Determine the task content corresponding to the student model.

[0063] Specifically, the task type (such as power dispatch text classification, equipment fault diagnosis), task objectives (such as accuracy requirements, inference speed requirements), and application scenarios (such as edge terminal deployment, cloud services) of the student model are obtained through user input, task configuration files, or automatic recognition methods, forming a structured task description.

[0064] Furthermore, the system uses natural language understanding technology to parse the task requirement text or collects key parameters through preset task templates. For example, for the task of building an automatic classification model for power distribution network fault work orders, the task type is parsed as text classification, the domain is power distribution network, and the core object is fault work orders. A task metadata model is constructed, including fields such as task ID, task name, domain label, input / output format, and performance indicators. Task information is received through an interactive interface or API interface, and after data cleaning and standardization, it is stored as a machine-understandable task description vector. Users select the task category (such as classification, recognition, or generation) and fill in the domain information through a visual interface. The system calls the domain dictionary to perform entity recognition and intent parsing on the task description; it automatically associates the typical feature types required for the task (such as power equipment entity features and fault terminology features) to form a task feature spectrum.

[0065] Precise task content definition provides a clear basis for knowledge matching, avoiding distillation noise caused by the introduction of irrelevant knowledge; structured task description enables the machine to automatically parse task requirements, improving the efficiency and accuracy of knowledge matching; task definition combined with application scenarios can constrain the lightweight goal of student models in advance, ensuring that training results meet actual deployment requirements.

[0066] Furthermore, after determining the corresponding task content, the task content can be broken down into multiple logically coherent steps. Each step corresponds to an independent operation unit in the task execution process. Each step meets the requirements of independence (with clear input and output boundaries), adaptability (matching with the hierarchical features of the knowledge architecture), and scalability (supporting the breakdown of complex steps into sub-steps). Each step corresponds to at least one piece of knowledge in the distillation knowledge base. The number of knowledge layers can also be configured according to the complexity. Simple steps (such as "word segmentation") are equipped with 1 layer of knowledge, while complex steps (such as "financial risk level determination") are equipped with 3-5 layers of complementary knowledge (including low-level professional terminology features, mid-level correlation features, and high-level decision features). When determining the target knowledge, the search scope is narrowed by using the steps as anchors (such as only retrieving relevant mid-level semantic feature knowledge in the "entity type judgment" step). The knowledge layer is adapted according to the complexity of the steps. The linkage of knowledge in different steps of the same task is activated through the knowledge association network (such as establishing a connection between the knowledge in the "fault cause analysis" and "solution generation" steps), ensuring accurate matching of target knowledge and that the student model can learn the complete task logic.

[0067] S103. Match target knowledge from the distillation knowledge base based on the task content, wherein the task content includes multiple ordered task steps, each task step corresponds to at least one target knowledge, and the task description of the task step is matched with multiple task types to determine the target features under the target task type, and the knowledge corresponding to the target features is determined as the target knowledge.

[0068] Specifically, the task content is converted into a feature requirement vector, its similarity to each feature in the distillation knowledge base is calculated, candidate knowledge is selected based on the relevance of the task domain, and the final target knowledge set is determined by ranking the features according to their importance.

[0069] Furthermore, relying on multiple ordered task segments derived from the task content (e.g., power fault diagnosis task is broken down into "fault text parsing → fault entity recognition → fault cause determination → solution generation"), each task segment is used as an independent matching unit to avoid cross-segment knowledge interference. For each segment, the core requirements in its task description are extracted, and the task description is semantically matched with multiple pre-defined task types in the distillation knowledge base (e.g., text classification, named entity recognition, question answering, etc.). By calculating the semantic similarity between the segment description and the task type definition, the target task type corresponding to that segment is determined. Precise matching based on task content ensures that the student model only learns key knowledge related to the target task, reducing redundant information interference and improving distillation efficiency. The multi-dimensional matching mechanism takes into account both the relevance and effectiveness of knowledge. Compared with random knowledge selection, it can improve the task accuracy of the student model, and flexible adjustment of matching parameters supports the optimization of knowledge selection strategies in different scenarios.

[0070] Furthermore, based on the target task type, the corresponding task type storage unit in the distillation knowledge base is located. Within this unit, target features that match the requirements of the current stage are selected: if the stage requires extracting entity boundary information, mid-level semantic features related to "entity boundary detection" (such as entity position weight features in the encoding layer) are selected within the storage unit; if the stage requires determining entity categories, high-level decision features related to "entity type labeling" (such as entity category probability distribution features in the output layer) are selected. During the selection process, the functional priority of the stage needs to be considered (e.g., in the "fault cause determination" stage, features related to causal relationships are prioritized), and redundant features with a correlation coefficient of less than 0.6 with the stage requirements are excluded to ensure a strong match between the target features and the stage functions. The knowledge corresponding to the selected target features is determined as the target knowledge for that stage. If a single stage requires multiple layers of knowledge support (e.g., in the "solution generation" stage, semantic association features and domain rule features are required simultaneously), the knowledge corresponding to multiple target features is integrated into the target knowledge set for that stage. At the same time, based on the orderliness of the task stages, the sequential relationship of the target knowledge of each stage is maintained, forming a target knowledge system that covers all task stages and is consistent with the task logic, providing accurate knowledge support for subsequent hierarchical training of the student model.

[0071] Optionally, there are multiple target knowledge sets, each of which is used to train a layer corresponding to a task stage. The number of target knowledge sets matches the number of layers in the student model, and the distillation weight of each target knowledge set is determined according to the layer of the pre-trained large model to which it belongs.

[0072] Specifically, there are multiple target knowledge sets, each corresponding one-to-one with a layer in the student model. Each target knowledge set is specifically used to train the corresponding task layer within the student model. Specifically, if the student model contains N layers, then the number of target knowledge sets matched from the distillation knowledge base is also N. The i-th target knowledge set directly corresponds to the i-th layer in training the student model, ensuring precise alignment between knowledge transfer and the model's hierarchical structure. The distillation weight of each target knowledge set is determined based on its hierarchical position within the pre-trained large model. Target knowledge sets corresponding to features at basic levels (such as the input layer and lower encoding layers) are assigned lower distillation weights because they carry the basic semantic understanding of the text. Target knowledge sets corresponding to features at intermediate levels (such as the middle encoding layer) are assigned medium distillation weights because they are responsible for capturing the structural relationships and domain semantics of the text. Target knowledge sets corresponding to features at higher levels (such as the output layer and decision layer) are assigned higher distillation weights because they are directly related to the generation logic of the task results. By using hierarchical adaptation and weight differentiation settings, it is possible to ensure that each layer of the student model learns the core capabilities of the corresponding stage, and to highlight the influence of key knowledge through weight adjustment, so that the student model can efficiently inherit the task adaptation capabilities of the pre-trained large model with a limited number of parameters.

[0073] Furthermore, when the target knowledge and the student model hierarchy are mapped one-to-one, each target knowledge independently corresponds to a certain layer of the student model, forming a hierarchical training unit. Target knowledge that strongly matches the function of the task is selected from the distillation knowledge base, and the target knowledge is directly mapped to the specified layer of the student model. The output distribution of the student model at that layer is made to approximate the feature distribution of the target knowledge through the knowledge distillation algorithm.

[0074] Furthermore, when the target knowledge possesses cross-level guidance value, multiple consecutive levels of the student model can be trained simultaneously using a single target knowledge, forming a hierarchical training chain for knowledge reuse. For example, the knowledge of "semantic association of financial risk" (including composite information such as domain terminology association and risk feature transmission) can simultaneously train layers 4-6 of the student model: layer 4 learns the basic associations between terms, layer 5 learns the transmission logic of risk features, and layer 6 learns the quantitative representation of association strength. By decomposing the target knowledge into features, sub-features adapted to different levels are extracted (such as extracting "term co-occurrence features," "transmission path features," and "intensity weight features" from the associated knowledge), and these are bound to the training objectives of the corresponding levels. Distillation weights employ a gradient allocation strategy: levels matching the core sub-features of the knowledge (such as layer 5 matching "transmission path features") are assigned weights of 0.6-0.7, while levels matching secondary sub-features (such as layers 4 and 6) are assigned weights of 0.3-0.4, ensuring that the core value of the knowledge is strengthened at key levels. It is suitable for scenarios with complex task stages that require multi-level collaboration. It can reduce the amount of target knowledge and decrease the redundancy of model training by reusing knowledge.

[0075] Furthermore, the specific implementation steps for matching target knowledge from the distillation knowledge base based on task content include:

[0076] (1) Determine the knowledge matching range based on the task content, and select large model features corresponding to the task type with the same type as the task content from the distillation knowledge base according to the knowledge matching range;

[0077] Specifically, the task content of the student model is analyzed to extract core task type tags. Using these tags as search criteria, the corresponding task type storage unit is located from the top-level category of the distillation knowledge base. Features of all pre-trained large models under that unit are selected, while features of other task types are excluded, forming a preliminary knowledge matching range. For example, when the task content is transformer fault description entity recognition, the matching range is limited to all model features under the power equipment entity recognition category in the knowledge base.

[0078] (2) For each layer of the student model, within the knowledge matching range, compare the performance metrics of the corresponding layer features of different pre-trained large models on the associated tasks of each layer of the network.

[0079] Specifically, for the network structure of the student model, the performance indicators of the corresponding layer features of different pre-trained large models are compared layer by layer within the determined knowledge matching range; for the student model embedding layer, the performance of the embedding layer features of each pre-trained large model is compared; for the student model encoding layer, the performance of the encoding layer features of each pre-trained large model is compared: the correlation coefficient represents the matching degree between the feature and the current level associated task; the transfer efficiency score represents the ease with which the feature is adapted to the student model on small-scale data.

[0080] By comparing the student model layers with the main model layers one by one, the problem of mismatch between knowledge and network structure is solved, ensuring that the selected features are consistent with the functional requirements of the corresponding student model layers (e.g., the student model output layer needs decision features, not lower-level word vector features). At the same time, the dual-index evaluation takes into account the relevance and transferability of features, avoiding the selection of features with excellent performance but difficult for the student model to learn.

[0081] (3) Select the best large model feature at each level as the target knowledge of the corresponding level, wherein the performance index includes the correlation coefficient between the feature and the task and the feature transfer efficiency score.

[0082] Specifically, for each layer of the student model, a comprehensive score is calculated by combining the correlation coefficient of the comprehensive features and the transfer efficiency score. The feature corresponding to the layer of the pre-trained large model with the highest comprehensive score is selected as the target knowledge for that layer. It should be noted that the weights of the correlation coefficient and transfer efficiency score are set according to actual needs, and are not restricted in this embodiment.

[0083] For example, if the student model's encoding layer 2 needs to match semantic features, and the relevance coefficient of the encoding layer 2 of model A is 0.9, the transfer efficiency score is 85, and the weighted comprehensive score is 87, and the relevance coefficient of the encoding layer 2 of model B is 0.85, the transfer efficiency score is 90, and the weighted comprehensive score is 87, then the feature of model A that is more similar to the existing features of the student model is selected as the target knowledge by using the feature similarity coefficient (taken from the horizontal association data of the knowledge base).

[0084] By selecting the optimal features at each level, the student model learns the most suitable knowledge at each layer. This ensures that the base layer acquires high-quality basic features (such as accurate word vectors for electrical terminology), the intermediate layers acquire efficient structural features (such as syntactic features of fault sentence structures), and the high-level layers acquire accurate decision features (such as probability distribution features of fault categories). Compared to randomly selecting or using all features, this improves the convergence speed of the student model's training while ensuring efficient knowledge transfer and reducing training data requirements. This approach guarantees the relevance of knowledge to the task, achieves adaptability between knowledge and the student model structure, and balances knowledge transferability with performance and lightweight requirements.

[0085] In addition to the method described above, the implementation steps of matching target knowledge from the distillation knowledge base based on the task content also include:

[0086] (1) Convert the text description of each step into a task feature vector and determine the feature vector of each piece of knowledge;

[0087] Specifically, the text descriptions of each task stage are converted into fixed-dimensional task feature vectors using a pre-trained semantic encoding model. The vector dimension must be consistent with the feature vectors of knowledge in the distillation knowledge base. For each piece of knowledge in the distillation knowledge base, its core features are extracted and converted into knowledge feature vectors using the same semantic encoding model, ensuring that both are in the same vector space, providing a foundation for subsequent relevance calculations. Furthermore, during the conversion process, the mapping relationship between the vectors and the original text / knowledge must be preserved to avoid vector confusion.

[0088] (2) Calculate the correlation between the task feature vector and the feature vector of each piece of knowledge;

[0089] Specifically, the cosine similarity algorithm is used to calculate the correlation between the task feature vector and each knowledge feature vector. For example, the correlation between the task feature vector of the "entity recognition" step and the knowledge vector of "named entity recognition task / model B / level 4 - entity type feature" in the knowledge base is 0.82, indicating a high degree of matching. For complex knowledge containing multi-level features, the correlation between each of its sub-feature vectors and the task vector needs to be calculated separately, and the average value is taken as the final correlation between the knowledge and the task, ensuring that the calculation result reflects the overall matching degree between the knowledge and the task.

[0090] (3) Select knowledge with a relevance higher than a preset threshold as candidate target knowledge;

[0091] Specifically, a relevance threshold is set (dynamically adjusted according to the task type; typically 0.6 for basic tasks and 0.7 for vertical domain tasks), and knowledge with a relevance higher than this threshold is selected as candidate target knowledge. For example, the threshold for financial risk identification tasks is set to 0.75, and only knowledge with a relevance ≥ 0.75 is retained in the candidate pool. After screening, the candidate knowledge needs to be deduplicated: if the similarity of feature vectors from different sources is ≥ 0.95, it is judged as duplicate knowledge, and only the knowledge corresponding to the better-performing model is retained (such as retaining the model knowledge with a higher F1 score), reducing the redundancy of candidate knowledge.

[0092] (4) Based on the hierarchical sequence relationship and semantic association network of the distillation knowledge base, determine the target knowledge that matches the task content.

[0093] Specifically, based on the hierarchical sequence relationship of the distillation knowledge base, knowledge matching the hierarchical requirements of the task stages is selected from candidate knowledge. Simultaneously, a semantic association network is used to analyze the correlation between candidate knowledge and the selected knowledge, defining correlations greater than a threshold (≥0.6 for valid correlation) to ensure logical connections between target knowledge corresponding to different stages of the same task (e.g., a strong correlation between "fault cause analysis" and "solution generation" knowledge). The final determined target knowledge must meet the following requirements: coverage of all task stages, hierarchical distribution matching the student model structure, and correlation between knowledge points conforming to task logic, forming a complete target knowledge chain.

[0094] S104. The student model is trained based on the target knowledge, and the size of the trained student model is smaller than the large model corresponding to the target knowledge.

[0095] Specifically, a distillation loss function that integrates target knowledge is constructed, a small-parameter network structure is used as the student model, and the model parameters are optimized through backpropagation. While ensuring performance, model compression techniques (such as pruning and quantization) are used to make the number of parameters and computation of the student model smaller than that of the source model.

[0096] Furthermore, utilizing a knowledge distillation-based teacher-student learning paradigm, multiple pre-trained large models are used as teacher models, with target knowledge serving as soft labels or feature constraints to guide student models in learning the knowledge representation of the large models. A joint loss function is designed, incorporating feature loss (the difference between student model features and target knowledge) and output loss (the loss from task prediction results). A small-sized network architecture (e.g., reducing the number of Transformer layers and the dimensionality of hidden layers) is adopted as the student model. During training, the knowledge distillation weights are dynamically adjusted, with the proportion of task loss gradually increased in later stages.

[0097] Furthermore, a student model with 1 / 10 to 1 / 5 the number of parameters of the large model can be initialized; the target knowledge is input into each layer of the student model according to the corresponding level; the joint loss is calculated and the parameters are updated in each round of training; after training, channel pruning is used to remove redundant neurons, and INT8 quantization is combined to further compress the model size.

[0098] Distillation training guided by target knowledge enables small models to inherit the key capabilities of large models, maintaining performance levels with a significant reduction in the number of parameters; lightweight design can improve the inference speed of student models, reduce storage consumption, and adapt to resource-constrained scenarios such as edge devices; compared with directly training small models, the knowledge distillation strategy can accelerate model convergence and has a more significant performance improvement on few-sample tasks.

[0099] Furthermore, the specific implementation steps for guiding student model training based on target knowledge include:

[0100] (1) Assign a loss weight to each target knowledge;

[0101] Specifically, the loss weights are dynamically determined based on the relevance and feature importance of the target knowledge and the student model's task content. The higher the relevance and importance, the greater the weight. By tilting the weights, the student model can be guided to prioritize learning key knowledge, avoid excessive consumption of training resources on secondary knowledge, and balance the learning priority of knowledge at different levels to prevent insufficient basic semantic understanding caused by emphasizing high-level layers and neglecting basic layers.

[0102] (2) Calculate the distillation loss of the student model for each of the multiple target knowledge areas;

[0103] Specifically, the distillation loss for each target knowledge is the feature difference between the student model's output at the corresponding level and the target knowledge. For basic feature-type target knowledge, mean squared error loss is used as the distillation loss; for decision-type target knowledge, KL divergence loss is used; and for semantic association-type target knowledge, cosine similarity loss is used. By calculating the loss by type, the inadequacy of a single loss function for different types of knowledge is avoided. For example, using mean squared error loss to calculate probability distribution ignores the normalization characteristics of probability, and using KL divergence loss to calculate word vectors amplifies small numerical differences. The targeted loss function can accurately capture the core deviations between each target knowledge and the student model, providing a reliable error signal for subsequent weight allocation and parameter optimization.

[0104] (3) The distillation losses corresponding to multiple target knowledge are weighted and summed according to their respective loss weights to obtain the total distillation loss;

[0105] Specifically, for each target knowledge, its corresponding distillation loss and loss weight are determined. The total distillation loss is obtained by weighted summation of all target knowledge values. This weighted summation merges multiple independent loss values ​​into a single total loss signal, giving the student model a clear optimization objective. It eliminates the need to optimize the loss of each target knowledge separately; instead, the deviation between each target knowledge and the student model is reduced simultaneously through a decrease in the total loss. Furthermore, the influence of the weights is directly reflected in the total loss, ensuring that the optimization of the loss of key knowledge is prioritized and reflected in the total loss, guiding the training direction to align with task requirements.

[0106] (4) Optimize the student model parameters with the goal of minimizing the total distillation loss and complete the student model training.

[0107] Specifically, an adaptive learning rate optimizer is used to set an initial learning rate. A gradient decay strategy is used to avoid oscillations in the later stages of training. After calculating the total distillation loss, the gradient is backpropagated through an automatic differentiation framework to update all trainable parameters of the student model, so that the total distillation loss gradually decreases. Training is terminated when the student model's task performance (such as fault classification accuracy) on the validation set reaches a preset threshold.

[0108] This approach enables efficient fusion and accurate transfer of multi-source knowledge. Compared to single-target knowledge distillation, it significantly improves the task generalization ability of the student model. Compared to unweighted average loss optimization, it enhances the key task performance of the student model. At the same time, it accelerates training convergence speed and reduces training resource consumption in small-sample scenarios in the power field, providing key technical support for the lightweight implementation of large NLP models in the power information field.

[0109] Furthermore, after training the student model based on the target knowledge, the process includes:

[0110] (1) Extract the features of each layer of the trained student model as new knowledge;

[0111] Specifically, the same techniques used for feature extraction of pre-trained large models (such as model layer output interception) are employed to extract features from each layer of the trained student model from the input layer to the output layer (including optimized word vectors adapted to the task, domain semantic features, task decision features, etc.) as new knowledge. Each piece of new knowledge is assigned a feature ID containing the student model identifier, and the task content label, target knowledge source information used in training, and model performance indicators corresponding to each piece of new knowledge are recorded.

[0112] Transforming the training results of student models into structured knowledge units not only preserves the optimized features of student models on specific tasks (which are more suitable for specific scenarios than the features of the original large model), but also enables knowledge traceability through metadata records. This allows for the subsequent tracing of the derivation path of a certain knowledge (such as which target knowledge was used to train it) and its actual effects (such as the performance of the corresponding student model), providing a basis for knowledge reuse and selection.

[0113] (2) Determine the task type storage unit based on the task type of the trained student model;

[0114] (3) If the task type exists in the task type storage unit, update the newly added knowledge to the task type storage unit;

[0115] (4) If the task type does not exist in the task type storage unit, a new task type storage unit is added to record the task type, and the newly added knowledge is updated to the new task type storage unit;

[0116] Specifically, based on the task type of the student model, the distillation knowledge base is searched for categories to determine the corresponding task type storage unit. If a corresponding task type storage unit already exists, the new knowledge is inserted into the second-level child node of the corresponding unit in hierarchical order (stored alongside the features of the larger model at the same level). If no corresponding task type storage unit exists, a new task type storage unit is created, metadata (task description, feature requirements) is defined, and the new knowledge is stored in it. Through this flexible storage strategy of updating existing units and creating new units, it ensures that new knowledge is accurately categorized, avoiding cross-task mixing and supporting the dynamic expansion of the knowledge base as new tasks are added, enabling the knowledge base to continuously cover diverse needs within the domain.

[0117] (5) Establish derivative association tags between new knowledge and target knowledge, update feature similarity coefficients and semantic association networks, and update the distillation knowledge base.

[0118] Specifically, derived labels are added to both the newly added knowledge and the target knowledge used during training (e.g., the feature annotations of the student model Layer3 are derived from Model_A_Layer4), clarifying the parent-child relationship between the two and facilitating the tracing of the knowledge evolution path. The vector similarity between the newly added knowledge and other knowledge of the same task type (including features of the large model and features of other student models) is calculated, and the coefficients of the horizontal association are updated. Based on the task label and feature vector of the newly added knowledge, it is integrated into the existing semantic association network, supplementing new association edges to make the network cover more comprehensive semantic relationships.

[0119] The updating of the association network ensures that newly added knowledge is not isolated, and the derived tags strengthen the inheritance relationship between knowledge, facilitating the subsequent selection of effective derived knowledge. The updated similarity coefficient provides a quantitative basis for cross-model and cross-source knowledge comparison. The expansion of the semantic network enhances the associative retrieval capability of the knowledge base, enabling it to cover more highly relevant derived features when matching target knowledge, thus improving the overall richness and depth of knowledge association. By incorporating optimized features from the student model, the knowledge base not only includes the general knowledge of the original large model but also accumulates task-validated adaptive knowledge, improving the matching accuracy for specific scenarios. The dynamic expansion of task types ensures that the knowledge base can adapt to new needs within the domain. Continuous optimization of the association network strengthens the intrinsic connections between knowledge, making knowledge retrieval more comprehensive and accurate. In the long term, as the student model training iterations increase, the knowledge base will gradually build a three-dimensional knowledge system covering multiple tasks, multiple levels, and multiple derived relationships, providing higher-quality and more suitable knowledge sources for subsequent distillation training.

[0120] Furthermore, the knowledge-based model distillation method provided in this embodiment also includes:

[0121] (1) Provide at least three distillation method options through the interactive interface, namely full-level distillation, key-level distillation and custom-level distillation. The interactive interface synchronously displays the knowledge scope preview and estimated distillation effect indicators corresponding to each distillation method.

[0122] Specifically, the visual interactive interface features three main entry points, each corresponding to a distillation method, accompanied by textual descriptions (e.g., full-level distillation: uses all matching level features, adapting to complex tasks; critical-level distillation: automatically selects core levels, balancing efficiency and performance; custom-level distillation: manually specifies levels to meet personalized needs), helping users quickly understand the differences. When each option is selected, the interface displays the knowledge scope of the corresponding distillation method in real time (e.g., full-level distillation displays a list of levels from input layer to embedding layer to encoding layer to output layer, annotating the feature types included in each level); simultaneously, based on historical distillation data from the knowledge base, it generates estimated performance metrics (e.g., accuracy, inference speed, model size), providing data references for user decision-making.

[0123] Through visual interaction and effect prediction, the technical distillation level selection is transformed into an operation that users can understand, avoiding strategy misselection due to users' unfamiliarity with the model structure; at the same time, the knowledge scope preview allows users to clearly know the knowledge source corresponding to the selected method, enhancing the transparency and controllability of the distillation process, especially suitable for the operational needs of non-technical operation and maintenance personnel in the power field.

[0124] (2) When full-level distillation is selected, all level features in the distillation knowledge base that match the task content will be automatically included in the knowledge scope;

[0125] Specifically, when a user selects full-level distillation, the system first matches the corresponding task type storage unit in the knowledge base based on the student model's task content. Then, it automatically traverses the hierarchical structure of all pre-trained large models under that unit (from the input layer to the output layer), filtering out all task-related features (excluding cross-task irrelevant features) to form a complete knowledge scope. Full-level distillation covers the complete knowledge chain from basic features to decision features, making it suitable for complex tasks in the power field that require high semantic understanding (such as multi-dimensional parsing of dispatch instructions and in-depth analysis of fault causes). This ensures that the student model can inherit the comprehensive knowledge of the large model, avoids incomplete semantic understanding due to missing levels, and guarantees distillation performance in complex task scenarios.

[0126] (3) When selecting key level distillation, the feature level range that plays a decisive role in the task content is automatically identified and defined based on the task content;

[0127] Specifically, based on the core requirements of the student model task content, key levels are automatically identified. First, the task type is analyzed to extract the core task objectives. Then, combined with the historical contribution data of each level of features under this task type in the knowledge base, the range of key levels is defined. Finally, features within this range are selected from the knowledge base as the knowledge scope. Key level distillation automatically filters core levels and eliminates redundant levels with low contribution to the task (such as low-level basic features). While ensuring core performance (such as fault classification accuracy), it reduces the amount of knowledge and training computation, adapting to scenarios in the power industry where high model inference speed is required.

[0128] (4) When selecting custom level distillation, the features of the corresponding level are selected as the knowledge scope based on the level range specified by the user in the interactive interface.

[0129] Specifically, the interactive interface provides layer selection controls (such as sliders or checkboxes) to display all layers under the corresponding task type (labeling the layer name and feature function, such as Encoding Layer 2: capturing fault phrase associations). Users can manually select layer ranges (such as Encoding Layer 3 → Output Layer) according to their actual needs (such as only optimizing the fault decision layer and retaining the existing basic semantic layer). Based on the user's selection, the system extracts the features of the corresponding layer from the knowledge base matching task type storage unit to form a custom knowledge range. Custom layer distillation gives users flexible adjustment space, which is suitable for personalized scenarios in the power industry. For example, when the performance of the basic semantic layer of the existing student model has reached the standard and only the decision layer needs to be optimized, users can select only the output layer features for distillation to avoid the waste of resources caused by repeatedly training the basic layer; or for special equipment (such as storage-limited transformer area maintenance terminals), high-level large-volume features can be manually removed to further compress the model size and meet customized deployment needs.

[0130] In addition to the existing hierarchical selection controls (slider, checkbox), users can also manually modify the target knowledge. After selecting a hierarchical range, the interface automatically lists detailed information on all feature knowledge within that range, including knowledge ID, source model (including performance metrics such as F1 score), feature type (such as attention weight / semantic features), applicable scenarios, and historical reuse effects, helping users quickly determine knowledge suitability. Each piece of knowledge is equipped with three operation buttons: "Keep," "Replace," and "Delete." Clicking "Replace" displays a list of other alternative knowledge at the same level (sorted in descending order of relevance to the current task, with similarity coefficients indicated), allowing users to select better knowledge to replace existing knowledge. Clicking "Delete" displays a pop-up indicating the impact of deleting the knowledge on related knowledge (e.g., "Deleting the semantic features of encoding layer 4 will cause a 30% decrease in the relevance of the decision features of encoding layer 5"). After user confirmation, the deletion operation is performed to prevent accidental disruption of the knowledge logic chain. Each time a user completes a knowledge modification (replacement / deletion), the interface updates the current custom knowledge set in real time, intuitively displaying the hierarchical distribution of the knowledge, the total amount of data, and the estimated distillation effect, allowing the user to clearly perceive the impact of the modification on the distillation results.

[0131] Furthermore, when a user manually modifies a target piece of knowledge, the system automatically triggers a knowledge recommendation process based on the horizontal and semantic association networks of the distilled knowledge base, providing the user with combination suggestions. Specifically, based on the attributes of the modified knowledge (such as level, feature type, and source model), the system retrieves two types of related knowledge from the knowledge base: first, horizontally related knowledge, i.e., similar features at the same level under different models; and second, semantically related knowledge, i.e., cross-level knowledge with the modified knowledge. The similarity between horizontally and semantically related knowledge is determined, and knowledge is filtered based on a similarity threshold, retaining knowledge with a similarity greater than the threshold to form an initial recommendation pool. The knowledge in the recommendation pool is sorted from high to low similarity. After sorting, the top M knowledge items are selected as recommendation results and displayed on the interface. For the recommended knowledge, the system further analyzes its combination effect with existing knowledge, generates combination schemes (such as "adopting recommendation 1 + recommendation 3 can improve model accuracy by 4% without increasing volume"), and provides a one-click adoption button. After clicking, the user can add the recommended knowledge to their custom knowledge scope, and the system automatically updates the knowledge association network and the estimated effect. If a user rejects a recommendation, the system records the reason for rejection and prioritizes excluding similar knowledge in subsequent recommendations to optimize recommendation accuracy.

[0132] By manually modifying target knowledge through a customizable hierarchical distillation interface, users can fine-tune the knowledge according to their actual needs, avoiding resource waste and adaptation bias. After a user modifies a target knowledge item, the system can quickly retrieve and recommend related knowledge based on the horizontal relationships and semantic networks of the knowledge base, and also generate combination schemes and a one-click adoption function, preserving the user's decision-making power while reducing operational complexity. It can accurately adapt to personalized scenarios in fields such as power, quickly forming a small-volume, highly adaptable knowledge scope, ensuring that the distilled model meets customized deployment requirements while guaranteeing core functional performance, achieving the dual goals of personalized needs and efficient knowledge combination.

[0133] The interactive distillation method selection process achieves a balance between technological flexibility and user-friendliness. Full-level distillation ensures performance for complex tasks, critical-level distillation balances efficiency, and custom-level distillation meets individual needs, covering distillation requirements across various power sector scenarios (such as complex analysis in dispatch centers, rapid identification of edge terminals, and customized deployment of special equipment). Simultaneously, knowledge scope preview and effect prediction help users make optimal decisions, reducing trial-and-error costs. Compared to a fixed, single distillation method, this process improves distillation adaptability, increases the deployment success rate of student models in different power scenarios, and significantly enhances user operational efficiency, lowering the barrier to technology implementation.

[0134] The knowledge base-based model distillation method provided in this embodiment extracts full-level features from pre-trained large models of multiple task types, constructing a distillation knowledge base with a tree-structured storage and multi-dimensional association index. This breaks the knowledge limitations of a single model and task, enabling refined knowledge management and efficient retrieval. Furthermore, it incorporates optimized features from post-trained student models through a dynamic update mechanism, forming a virtuous cycle of knowledge accumulation and model optimization, continuously improving the knowledge base's adaptability to different scenarios. In the distillation matching stage, target knowledge is accurately selected based on task content through vector calculation and hierarchical comparison. It also provides three distillation methods: full-level, key-level, and custom-level, balancing performance, efficiency, and personalization for complex tasks. This approach significantly enhances the relevance and operational flexibility of distillation by addressing diverse needs. During the training phase, the categorized loss calculation and dynamic weight allocation based on multi-objective knowledge ensure that student models efficiently inherit the key capabilities of large models. Furthermore, the model size is significantly reduced after training, effectively solving the problems of high computational resource consumption, high storage consumption, and slow inference response when deploying large models. It is adaptable to edge devices and resource-constrained business systems, especially in professional fields such as power, where it can accurately match the task requirements of scheduling instruction parsing and fault diagnosis. While ensuring high model accuracy, it also meets the requirements of scenarios such as real-time response from edge terminals, providing key technical support for the lightweight and scenario-based deployment of large NLP models. This significantly enhances the practical application adaptability and generalization ability of lightweight models.

[0135] Corresponding to the aforementioned embodiment of a knowledge-based model distillation method, this application also provides an embodiment of a knowledge-based model distillation apparatus.

[0136] Figure 2 This is a schematic diagram of the structure of Embodiment 2 of the knowledge-based model distillation apparatus provided in this application. Please refer to... Figure 2 The apparatus provided in this embodiment includes a construction module 210, a determination module 220, and a training module 230; wherein,

[0137] The construction module 210 is used to extract features from each layer of multiple pre-trained large models to form a distillation knowledge base, wherein each piece of knowledge in the distillation knowledge base corresponds to a feature, and the multiple pre-trained large models correspond to multiple task types.

[0138] The determining module 220 is used to determine the task content corresponding to the student model;

[0139] The training module 230 is used to match target knowledge from the distillation knowledge base based on the task content, wherein the task content includes multiple ordered task steps, each task step corresponds to at least one target knowledge, and the task description of the task step is matched with multiple task types to determine the target features under the target task type, and the knowledge corresponding to the target features is determined as the target knowledge.

[0140] The training module 230 is also used to guide the training of the student model based on the target knowledge, wherein the size of the trained student model is smaller than the large model corresponding to the target knowledge.

[0141] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0142] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0143] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A knowledge-based model distillation method, characterized in that, The method includes: Features from each layer are extracted from multiple pre-trained large models to form a distillation knowledge base, wherein each piece of knowledge in the distillation knowledge base corresponds to a feature, and the multiple pre-trained large models correspond to multiple task types. The distillation knowledge base comprises: For each task type, multiple pre-trained large models are selected as knowledge sources, and all features from the input layer to the output layer of each pre-trained large model are extracted. All features include model parameters and attention weights. With task type as the main node and hierarchical information as the branch nodes, a knowledge architecture with the same hierarchical relationship as the model structure is constructed. All features are categorized into the knowledge architecture to form a triplet of knowledge consisting of a main node, branch nodes, and feature content, and stored in the distillation knowledge base; Determine the task content corresponding to the student model; Based on the task content, target knowledge is matched from the distillation knowledge base. The task content includes multiple ordered task steps, each task step corresponds to at least one target knowledge. The task description of the task step is matched with multiple task types to determine the target features under the target task type. The knowledge corresponding to the target features is determined as the target knowledge. The student model is trained based on the target knowledge, and the size of the trained student model is smaller than the large model corresponding to the target knowledge. The step of guiding the student model training based on the target knowledge includes: Assign a loss weight to each target knowledge; Calculate the distillation loss of the student model separately for multiple target knowledge; The total distillation loss is obtained by weighting and summing the distillation losses corresponding to multiple target knowledge points according to their respective loss weights. The student model parameters were optimized with the goal of minimizing total distillation loss, and the student model training was completed.

2. The method according to claim 1, characterized in that, There are multiple target knowledge points, and each target knowledge point is used to train a layer corresponding to a task stage. The number of target knowledge points matches the number of layers in the student model. The distillation weight of each target knowledge point is determined according to the layer of the pre-trained large model to which it belongs.

3. The method according to claim 1, characterized in that, The matching of target knowledge from the distillation knowledge base based on the task content includes: Based on the task content, a knowledge matching range is determined, and large model features corresponding to task types of the same type as the task content are selected from the distillation knowledge base according to the knowledge matching range. For each layer of the student model, within the knowledge matching range, compare the performance metrics of the corresponding layer features of different pre-trained large models on the network association task at each layer. The feature with the best performance index at each level is selected as the target knowledge for the corresponding level.

4. The method according to claim 1, characterized in that, The distillation knowledge base also includes: The categories are divided according to task type, and each category is allocated an independent task type storage unit in the distillation knowledge base. Multiple first-level child nodes are set under each task type storage unit. Each first-level child node corresponds to a pre-trained large model. Multiple child nodes are stored in order of high to low according to the performance index of the pre-trained large model on the task type. Each primary child node has multiple secondary child nodes, and each secondary child node corresponds to a piece of knowledge. The multiple secondary child nodes are sorted according to the hierarchical order. Among them, secondary child nodes with the same hierarchical number under different primary child nodes are horizontally associated and the knowledge similarity coefficient is marked. All secondary child nodes under the same task type are connected through the association degree to form a semantic association network.

5. The method according to claim 4, characterized in that, Matching target knowledge from the distillation knowledge base based on the task content also includes: The text description of each step is converted into a task feature vector, and the feature vector of each piece of knowledge is determined. Calculate the correlation between the task feature vector and the feature vector of each piece of knowledge; Filter knowledge with a relevance higher than a preset threshold as candidate target knowledge; Based on the hierarchical sequence relationship and semantic association network of the distillation knowledge base, target knowledge that matches the task content is determined.

6. The method according to claim 1, characterized in that, After training the student model based on the target knowledge, the process includes: Extract features from each layer of the trained student model as new knowledge; The task type storage unit is determined based on the task type of the trained student model; If the task type exists in the task type storage unit, the newly added knowledge will be updated to the task type storage unit; If the task type does not exist in the task type storage unit, a new task type storage unit is created to record the task type, and the newly added knowledge is updated to the new task type storage unit; Establish derivative association tags between newly added knowledge and target knowledge, update feature similarity coefficients and semantic association networks, and update the distillation knowledge base.

7. The method according to claim 1, characterized in that, The method further includes matching target knowledge from the distillation knowledge base based on the task content, and the method also includes: The interactive interface provides at least three distillation method options: full-level distillation, critical-level distillation, and custom-level distillation. The interactive interface also displays a preview of the knowledge scope and an estimated distillation effect index for each distillation method. When full-level distillation is selected, all level features in the distillation knowledge base that match the task content will be automatically included in the knowledge scope. When selecting key level distillation, the feature level range that plays a decisive role in the task content is automatically identified and defined based on the task content; When selecting custom-level distillation, features of the corresponding level are selected as the knowledge scope from the matching task type based on the level range specified by the user in the interactive interface.

8. A knowledge-based model distillation apparatus, characterized in that, The device includes a construction module, a determination module, and a training module; wherein... The construction module is used to extract features from each layer of multiple pre-trained large models to form a distillation knowledge base, wherein each piece of knowledge in the distillation knowledge base corresponds to a feature, and the multiple pre-trained large models correspond to multiple task types. The distillation knowledge base comprises: For each task type, multiple pre-trained large models are selected as knowledge sources, and all features from the input layer to the output layer of each pre-trained large model are extracted. All features include model parameters and attention weights. With task type as the main node and hierarchical information as the branch nodes, a knowledge architecture with the same hierarchical relationship as the model structure is constructed. All features are categorized into the knowledge architecture to form a triplet of knowledge consisting of a main node, branch nodes, and feature content, and stored in the distillation knowledge base; The determining module is used to determine the task content corresponding to the student model; The training module is used to match target knowledge from the distillation knowledge base based on the task content, wherein the task content includes multiple ordered task steps, each task step corresponds to at least one target knowledge, the task description of the task step is matched with multiple task types to determine the target features under the target task type, and the knowledge corresponding to the target features is determined as the target knowledge. The training module is also used to guide the training of the student model based on the target knowledge, and the size of the trained student model is smaller than the large model corresponding to the target knowledge; The step of guiding the student model training based on the target knowledge includes: Assign a loss weight to each target knowledge; Calculate the distillation loss of the student model separately for multiple target knowledge; The total distillation loss is obtained by weighting and summing the distillation losses corresponding to multiple target knowledge points according to their respective loss weights. The student model parameters were optimized with the goal of minimizing total distillation loss, and the student model training was completed.

Images