Knowledge distillation methods, apparatus, equipment and storage media

Abstract

This application discloses a knowledge distillation method, apparatus, device, and storage medium, relating to the field of knowledge distillation technology. The disclosed knowledge distillation method includes: parsing task text data corresponding to a task to be executed to construct a task execution thought chain, where the task to be executed includes a device fault diagnosis task; extracting keywords from the task text data and the task execution thought chain to obtain highly relevant keywords; constructing a knowledge subnetwork based on a task execution knowledge graph and highly relevant keywords; and performing knowledge transfer and distillation on a student model through the knowledge subnetwork. Applying the above technical solution solves the technical problem in existing knowledge distillation methods that struggle to express and transfer complex knowledge within large models, leading to student models' inability to fully understand and master this knowledge.

Description

Technical Field

[0001] This application relates to the field of knowledge distillation technology, and in particular to knowledge distillation methods, apparatus, equipment and storage media. Background Technology

[0002] While large-scale language models (LLMs) excel at many tasks, they typically require significant computational and storage resources, making them expensive and difficult to deploy in real-world applications. Furthermore, the lengthy training and inference times of these models hinder applications requiring rapid response. Knowledge distillation, a method that transfers knowledge from a large model to a smaller one, aims to mitigate these drawbacks. Through knowledge distillation, knowledge from a high-performance but resource-intensive large model (the teacher model) can be transferred to a smaller, more efficient model (the student model). This reduces the computational and storage requirements of the model without significantly sacrificing performance, making it more suitable for practical applications.

[0003] However, traditional knowledge distillation methods have difficulty expressing and conveying complex knowledge within large models, especially when it comes to abstract concepts and complex reasoning, where student models may not be able to fully understand and master this knowledge.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a knowledge distillation method, apparatus, device, and storage medium, which aims to solve the technical problem that existing knowledge distillation methods have difficulties in expressing and transmitting complex knowledge within large models, resulting in student models being unable to fully understand and master this knowledge.

[0006] To achieve the above objectives, this application proposes a knowledge distillation method, which includes:

[0007] The task text data corresponding to the task to be executed is parsed to construct the task execution thought chain corresponding to the task to be executed, the task to be executed includes: equipment fault diagnosis task;

[0008] Keyword extraction is performed on the task text data and the task execution thought chain to obtain highly relevant keywords;

[0009] A knowledge subnetwork is constructed based on the task execution knowledge graph and the highly relevant keywords;

[0010] Knowledge transfer and distillation are performed on the student model through the aforementioned knowledge subnet.

[0011] In one embodiment, the step of parsing the task text data corresponding to the task to be executed in order to construct the task execution thought chain corresponding to the task to be executed includes:

[0012] Semantic analysis is performed on the task text data corresponding to the task to be executed in order to extract all semantic units in the task text data, as well as the semantic information between each semantic unit;

[0013] The logical relationships between the semantic units are determined based on the semantic information.

[0014] Based on the logical relationship, construct the task execution thought chain corresponding to the task to be executed.

[0015] In one embodiment, the step of extracting keywords from the task text data and the task execution thought chain to obtain highly relevant keywords includes:

[0016] Keyword extraction is performed on the task text data and the task execution thought chain to obtain several target keywords;

[0017] The target keywords are used to perform relationship analysis and background expansion on the task execution knowledge graph.

[0018] Upon completion of the expansion, the target keywords are scored for relevance using a teacher model.

[0019] Highly relevant keywords are identified from the target keywords based on the scoring results.

[0020] In one embodiment, the step of constructing a knowledge subnetwork based on the task execution knowledge graph and the highly relevant keywords includes:

[0021] Based on the highly relevant keywords, a knowledge node search is performed in the task execution knowledge graph to determine the target knowledge nodes and the node relationships between the target knowledge nodes.

[0022] A knowledge subnet is constructed based on the target knowledge nodes and the relationships between the nodes.

[0023] In one embodiment, after the step of knowledge transfer and distillation of the student model through the knowledge subnet, the method further includes:

[0024] The teacher model generates predicted labels based on the task text data.

[0025] The teacher model is used to obtain explanatory information corresponding to the prediction results of the task to be performed.

[0026] The student model is optimized based on the predicted labels, the explanatory information, and the task execution knowledge graph.

[0027] In one embodiment, the step of optimizing the student model based on the predicted labels, the explanatory information, and the task execution knowledge graph includes:

[0028] Based on the explanatory information and the task execution knowledge graph, determine the cross-entropy loss between the predicted label and the target label;

[0029] Based on the predicted labels and the cross-entropy loss, the student model is optimized using a preset optimization loss function;

[0030] The preset optimization loss function is:

[0031]

[0032] In the formula, L label Let N be the preset optimized loss function, N be the number of samples, and L be the cross-entropy loss. Let f be the predicted label, f be the student model, and x be the predicted label. i This is the input data for the student model.

[0033] In one embodiment, the step of determining the cross-entropy loss between the predicted label and the target label based on the explanatory information and the task execution knowledge graph includes:

[0034] The interpretation generation loss is determined based on the interpretation information;

[0035] The knowledge graph information generation loss is determined based on the key information in the task execution knowledge graph.

[0036] The cross-entropy loss between the predicted label and the target label is determined based on the explanation generation loss and the knowledge graph information generation loss.

[0037] Furthermore, to achieve the above objectives, this application also proposes a knowledge distillation apparatus, the apparatus comprising:

[0038] The thought chain construction module is used to parse the task text data corresponding to the task to be executed in order to construct the task execution thought chain corresponding to the task to be executed, the task to be executed including: equipment fault diagnosis task;

[0039] The keyword extraction module is used to extract keywords from the task text data and the task execution thought chain to obtain highly relevant keywords.

[0040] The knowledge network construction module is used to construct knowledge subnetworks based on the task execution knowledge graph and the highly relevant keywords;

[0041] The knowledge distillation module is used to perform knowledge transfer and distillation on the student model through the knowledge subnet.

[0042] In addition, to achieve the above objectives, this application also proposes a knowledge distillation apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the knowledge distillation method as described above.

[0043] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the knowledge distillation method described above.

[0044] This application provides a knowledge distillation method. It discloses parsing task text data corresponding to a task to be executed, constructing a task execution thought chain for that task, which includes: equipment fault diagnosis. Keyword extraction is performed on the task text data and the task execution thought chain to obtain highly relevant keywords. A knowledge subnetwork is constructed based on the task execution knowledge graph and the highly relevant keywords. Knowledge transfer and distillation are performed on a student model through the knowledge subnetwork. Because this invention obtains highly relevant keywords based on the task text data and task execution thought chain of the task to be executed, and performs knowledge transfer and distillation on the student model through the knowledge subnetwork constructed based on the task execution knowledge graph and the highly relevant keywords, it solves the technical problem in existing knowledge distillation methods that have difficulty expressing and transferring complex knowledge within large models, leading to the student model's inability to fully understand and master this knowledge. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic flowchart of the knowledge distillation method of this application, provided in Embodiment 1.

[0048] Figure 2 This is an example diagram illustrating the construction of the thought chain in the knowledge distillation method of this application;

[0049] Figure 3This is a framework diagram of the knowledge distillation method of this application;

[0050] Figure 4 This is a schematic flowchart of the knowledge distillation method embodiment two of this application;

[0051] Figure 5 This is an example diagram illustrating the extraction of knowledge graph-related knowledge in the knowledge distillation method of this application;

[0052] Figure 6 This is a schematic diagram of knowledge transfer and distillation in the knowledge distillation method of this application;

[0053] Figure 7 This is a schematic flowchart of the knowledge distillation method embodiment three of this application;

[0054] Figure 8 This is a schematic diagram of the modular structure of the knowledge distillation apparatus according to an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the hardware operating environment involved in the knowledge distillation method in the embodiments of this application.

[0056] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0058] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of this application embodiment is: to parse the task text data corresponding to the task to be executed in order to construct the task execution thinking chain corresponding to the task to be executed, wherein the task to be executed includes: equipment fault diagnosis task; to extract keywords from the task text data and the task execution thinking chain to obtain highly relevant keywords; to construct a knowledge subnetwork based on the task execution knowledge graph and the highly relevant keywords; and to perform knowledge transfer and distillation on the student model through the knowledge subnetwork.

[0060] This application provides a solution that can obtain highly relevant keywords based on the task text data and task execution thought chain of the task to be executed, and perform knowledge transfer and distillation on the student model through a knowledge subnetwork constructed based on the task execution knowledge graph and highly relevant keywords. This solves the technical problem that the knowledge distillation method in the prior art has difficulty in expressing and transferring complex knowledge inside the large model, which leads to the student model being unable to fully understand and master this knowledge.

[0061] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or knowledge distillation device capable of performing the above functions. The following description uses a knowledge distillation device (hereinafter referred to as the device) as an example to illustrate this embodiment and the subsequent embodiments.

[0062] Based on this, embodiments of this application provide a knowledge distillation method, referring to... Figure 1 , Figure 1 This is a schematic flowchart of the first embodiment of the knowledge distillation method of this application.

[0063] In this embodiment, the knowledge distillation method includes steps S10 to S40:

[0064] Step S10: Parse the task text data corresponding to the task to be executed in order to construct the task execution thought chain corresponding to the task to be executed, the task to be executed includes: equipment fault diagnosis task.

[0065] It should be understood that the above-mentioned task to be performed can be any task in an application scenario that can use a large-scale language model, such as: equipment fault diagnosis task, that is, the task of diagnosing the faults of equipment in production. This embodiment does not limit this.

[0066] It is understood that the above-mentioned task text data can be the text data required during the task execution process. For example, for a device fault diagnosis task, the corresponding task text data may include: the purpose of this fault diagnosis, the device identifier corresponding to the faulty device, the possible causes of the fault, etc. This embodiment does not limit this.

[0067] It should be noted that the aforementioned task execution thought chain can be a complete reasoning chain formed by meticulously recording each reasoning step experienced by a large model when solving a problem. These steps include not only the final conclusion but also the intermediate assumptions, reasoning logic, and the knowledge points used. By simulating the human thought process, the thought chain breaks down complex tasks into a series of simple steps. In this way, student models can clearly understand the thought process of the large model, thereby better learning and imitating its reasoning methods. The construction of the thought chain not only helps student models master complex reasoning abilities but also enhances the model's interpretability, enabling it to reason more accurately when faced with complex tasks.

[0068] Further, step S10 includes: performing semantic analysis on the task text data corresponding to the task to be executed, so as to extract all semantic units in the task text data and the semantic information between each semantic unit; determining the logical relationship between each semantic unit based on the semantic information; and constructing the task execution thought chain corresponding to the task to be executed based on the logical relationship.

[0069] It should be understood that semantic analysis is the process of deeply understanding and analyzing natural language text to identify semantic information such as entities, relationships, and sentiment in task text data. Correspondingly, the aforementioned semantic unit can be a unit consisting of a word or phrase obtained after semantic analysis of the task text data.

[0070] In practical applications, the construction of thought chains plays a crucial role in the knowledge distillation process. By analyzing the semantic associations and logical connections of the input data, a complete thought chain is formed. This thought chain not only demonstrates the teacher model's understanding and reasoning path of the input data but also effectively transfers this knowledge and logical relationship to the student model, significantly improving the latter's learning effectiveness and performance. In this embodiment, the construction process of the task execution thought chain mainly includes four steps: data preprocessing, semantic analysis, logical relationship construction, and thought chain optimization. First, data preprocessing ensures data quality and consistency through text cleaning, word segmentation, and part-of-speech tagging. Second, semantic analysis extracts semantic units and their interrelationships through a deep learning model, thereby identifying semantic connections between words. Then, logical relationship construction, based on these semantic units, demonstrates the reasoning path and logical structure between knowledge points. Finally, the thought chain is output through a large model to ensure its accuracy and completeness.

[0071] In the specific implementation, refer to Figure 2 , Figure 2 This is an example diagram illustrating the construction of the thought chain in the knowledge distillation method of this application. For example... Figure 2As shown, this embodiment uses GPT-3.5 as the teacher model to construct the thought chain, and verifies it using the CQA dataset (an open-source question-and-answer dataset). First, the CQA dataset is obtained. The text content in the dataset can be: A beaver is famous for its construction skills. Where do its materials come from? (A) British Columbia; (B) Water; (C) Forest region; (D) Paying debts; (E) Zoo. The CQA dataset is preprocessed, specifically by removing meaningless characters, stop words, and other noisy data. Then, semantic analysis, logical relationship construction, and thought chain optimization are performed by calling the GPT-3.5 API interface. In this embodiment, GPT-3.5 is guided by examples to progressively parse the question and generate the reasoning process corresponding to the correct answer. In this embodiment, the task text data corresponding to the task to be executed can be input into the teacher model. Then, the teacher model can guide the output of the above task execution thought chain based on the thought chain examples. Specifically, for... Figure 2 The text input shown to the teacher model can be guided by the following thought chain example: `cot_prompt_template = Give a question, options, and correct answer, provide a detailed plan and explanation explaining why the correct answer is the best choice, ensuring that the explanation and key words are explicitly included.` The output should then be in the following format: Plan: 1. Restate the question and list the options; 2. Analyze each option in the context of the question; 3. Explain why the correct answer is the best choice; Explanation: The explanation is here. The teacher model can then output the thought chain: Beavers are known for their ability to build complex dams and burrows using branches, logs, and mud. They gather these materials from woodlands near water and build their homes there. The woodlands provide the necessary resources for beavers to demonstrate their architectural skills and create habitats.

[0072] Step S20: Extract keywords from the task text data and the task execution thought chain to obtain highly relevant keywords.

[0073] It should be noted that the aforementioned highly relevant keywords can be keywords that are highly relevant to the task text data and the task execution thought process. In this embodiment, a teacher model can be used to score the relevance of keywords in the task text data and the task execution thought process, and based on the scoring results, keywords with higher relevance scores are identified as highly relevant keywords.

[0074] Step S30: Construct a knowledge subnetwork based on the task execution knowledge graph and the highly relevant keywords.

[0075] It should be noted that the aforementioned task execution knowledge graph can be a structured information storage method, where each knowledge node represents a concept or entity, and the edges between nodes represent the relationships between them. In this embodiment, the knowledge graph presents knowledge nodes and their relationships in a structured manner, preventing key information from being buried in noise. This structured information presentation can significantly enhance the student model's understanding of the relationships between words, thereby improving the coherence and systematic nature of knowledge. Furthermore, the knowledge graph can also help capture and represent relationships in longer texts, enabling the student model to systematically learn and transfer complex reasoning abilities, thus better addressing new and unseen problems.

[0076] It should be noted that the aforementioned knowledge subnetwork can be a network used to display the nodes and relationships between them in a task execution knowledge graph. In practical applications, relevant knowledge nodes and relationships can be searched in the task execution knowledge graph using highly relevant keywords to construct a knowledge network closely related to the reasoning process, i.e., the aforementioned knowledge subnetwork. This method innovatively ensures that the knowledge nodes and relationships extracted from the knowledge graph are highly relevant through effective keyword extraction and relevance scoring. This not only helps to construct a concise and practical knowledge subgraph but also effectively avoids student models encountering too many irrelevant knowledge points during the learning process, reducing error accumulation and performance degradation caused by noise. Furthermore, this method not only solves the problem of difficulty in identifying useful information from a large knowledge graph but also addresses the difficulty of large models expanding their background information by incorporating the structure of the knowledge graph. Providing these highly relevant knowledge points as background information to student models not only enriches their knowledge base but also helps them better understand and apply these knowledge points.

[0077] Step S40: Perform knowledge transfer and distillation on the student model through the knowledge subnet.

[0078] In practical applications, the device can efficiently transfer knowledge from the teacher model to the student model through a knowledge subnet. In this embodiment, knowledge transfer and distillation can systematically transfer the thought chains extracted from the large model and relevant knowledge from the knowledge graph to the student model. This process employs a step-by-step transfer method to ensure that the student model can gradually absorb and digest this knowledge. First, the reasoning process of the large model is demonstrated through thought chains, enabling the student model to understand and imitate its reasoning logic. Then, the background knowledge provided by the knowledge graph further enriches the student model's knowledge base. Finally, combining these two aspects of information, the system innovatively not only enables the student model to learn from the teacher model's output but also encourages it to learn from the teacher model's reasoning ability and its understanding of globally important information. This not only ensures the comprehensiveness and systematic nature of knowledge transfer but also improves the student model's reasoning and knowledge application abilities, thereby enhancing its performance in practical applications.

[0079] In the specific implementation, refer to Figure 3 , Figure 3 This is a framework diagram of the knowledge distillation method of this application. (For example...) Figure 3 As shown, firstly, the task text data corresponding to the task to be executed can be input into the teacher model. The teacher model performs semantic analysis, logical relationship construction, and thought chain optimization on the input data to construct the aforementioned task execution thought chain. Then, keywords can be extracted from the task execution thought chain and the input task text data, and these keywords are fed into the knowledge graph for relationship analysis and background expansion, enriching the associated information in the knowledge graph. Next, these keywords and their relationships are passed to the teacher model for relevance scoring, thereby selecting keywords and relationships with high relevance scores to construct knowledge subnetworks. Finally, the thought chain extracted from the large model and the relevant knowledge in the knowledge graph can be systematically passed to the student model.

[0080] This embodiment provides a knowledge distillation method. The method discloses parsing the task text data corresponding to the task to be executed to construct a task execution thought chain. The task to be executed includes: equipment fault diagnosis task; keyword extraction from the task text data and task execution thought chain to obtain highly relevant keywords; construction of a knowledge subnetwork based on the task execution knowledge graph and highly relevant keywords; and knowledge transfer and distillation of the student model through the knowledge subnetwork. Because this embodiment obtains highly relevant keywords based on the task text data and task execution thought chain of the task to be executed, and performs knowledge transfer and distillation of the student model through the knowledge subnetwork constructed based on the task execution knowledge graph and highly relevant keywords, it solves the technical problem in existing knowledge distillation methods that have difficulty expressing and transferring complex knowledge within large models, leading to the student model's inability to fully understand and master this knowledge.

[0081] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a schematic flowchart of the knowledge distillation method embodiment two of this application.

[0082] In this embodiment, step S20 includes steps S201 to S204:

[0083] Step S201: Extract keywords from the task text data and the task execution thought chain to obtain several target keywords.

[0084] It should be understood that the aforementioned target keywords can be keywords in the steps or concepts of the task text data and the task execution thought chain, such as keywords in the questions, options and explanations extracted from the task text data and the task execution thought chain. This embodiment does not limit this.

[0085] Step S202: Perform relationship analysis and background expansion on the task execution knowledge graph using the target keywords.

[0086] It should be noted that performing relational analysis on the task execution knowledge graph can be an analysis of the correlation between nodes in the knowledge graph; extending the background of the task execution knowledge graph can be a technique for applying the knowledge graph to a wider range of fields, such as semantic role labeling, entity linking, entity embedding, etc. This embodiment does not limit this, and this process can enrich the relationships in the task execution knowledge graph.

[0087] Step S203: Upon completion of the expansion, the target keywords are relevance-scored using a teacher model.

[0088] It is understandable that the above relevance score can be used to score the degree of relevance between target keywords.

[0089] Step S204: Based on the scoring results, determine the highly relevant keywords from the target keywords.

[0090] Further, step S30 includes: searching for knowledge nodes in the task execution knowledge graph based on the highly relevant keywords to determine the target knowledge nodes and the node relationships between the target knowledge nodes; and constructing a knowledge subnetwork based on the target knowledge nodes and the node relationships.

[0091] In this embodiment, the target knowledge node can be a node in the task execution knowledge graph. After obtaining highly relevant keywords, related knowledge nodes and their corresponding relationships can be searched in the task execution knowledge graph using these keywords. Based on the knowledge nodes and their relationships, a knowledge network closely related to the reasoning process can be constructed.

[0092] It's important to note that applying knowledge graphs effectively addresses the problem of key information in the teacher model being buried in noise during knowledge distillation. Traditional knowledge extraction methods often struggle to capture relationships between words and complex relationships from longer texts. Knowledge graphs, however, visually represent the relationships between nodes using triples (node ​​1 - relation - node 2). For example, using the formula (node ​​1, relation, node 2), knowledge graphs can break down complex textual information into clear relational chains, making it easier for student models to understand and learn these relationships. This structured information presentation helps student models capture key information from the text and enhances their understanding and application of knowledge through the relationships between nodes.

[0093] It should be understood that knowledge graphs not only help student models understand complex relationships but also significantly improve the quality of knowledge distillation. By providing student models with highly relevant knowledge points and their relationships as background information, knowledge graphs enrich the students' knowledge base, enabling them to better understand and apply these knowledge points. Specifically, through the interpretation of relationships, student models can more clearly capture key relationships in the input information, improving their reasoning ability and overall performance. For example, using the formula (e1, r, e2) to represent the relationship r between entities e1 and e2, student models can more clearly identify which information is key, thereby improving the efficiency and quality of knowledge extraction. In this way, student models can not only avoid interference from noisy information but also exhibit stronger adaptability and generalization ability when facing new or complex tasks. Combining the advantages of thought chains and knowledge graphs, this solution achieves comprehensiveness and systematicity in knowledge transfer, making student models more accurate and efficient in knowledge mastery and application.

[0094] In the specific implementation, refer to Figure 5 , Figure 5 This is an example diagram illustrating the knowledge extraction from knowledge graphs in the knowledge distillation method of this application. For example... Figure 5As shown, the task text data (A beaver is famous for its building skills. Where do they get their materials? (A) British Columbia; (B) Water; (C) Forest; (D) Paying debts; (E) Zoo) and the data in the task execution thought chain (Beavers are famous for their ability to build complex dams and dens with branches, logs, and mud. They collect these materials from woodlands near water and build their homes there. Woodlands provide the necessary resources for beavers to demonstrate their building skills and create habitats) are first input into the teacher model. The teacher model extracts keywords from this data using keyword examples (keywords_prompt_template = "List keywords in the questions, options, and explanations. Keywords: Extract keywords here"). The extracted keywords can include: beaver, building skills, materials, dam, woodland, etc. These keywords can then be fed into the knowledge graph for relationship analysis and background expansion to enrich the association information of the knowledge graph. Subsequently, these keyword machine relationships are fed into the teacher model for relevance scoring to filter out keywords and relationships with high relevance scores. Finally, these highly relevant keywords are used to search for related knowledge nodes and relationships in the knowledge graph, constructing a knowledge network closely related to the reasoning process. For example... Figure 5 As shown, the output at this point can be: A[Beaver] -> | Show | B[Construction Skill]; A[Beaver] -> | Requires | B[Materials]; C[Materials] -> | Obtained from | D[Woodland]; D[Woodland] -> | Provides | E[Twigs]; D[Woodland] -> | Provides | F[Logs]; D[Woodland] -> | Provides | G[Soil]; B[Construction Skill] -> | Used to build | H[Dam]; B[Construction Skill] -> | Used to build | I[Nest]; H[Dam] -> | Located nearby | J[Water Area]; J[Nest] -> | Located nearby | J[Water Area]; J[Water Area] -> | Required | K[Habitat].

[0095] It should be noted that after constructing the knowledge subnet, knowledge transfer and distillation can be performed on the student model through the knowledge subnet. Knowledge transfer and distillation involve efficiently transferring knowledge from the teacher model to the student model by combining relevant knowledge from the knowledge graph and thought chains. In this embodiment, not only can the student model learn the output of the teacher model, but it is also encouraged to learn the reasoning ability of the teacher model and its understanding of globally important information. Specifically, this scheme can train the model with prefix annotation according to different given tasks, and output in different ways.

[0096] Specifically, refer to Figure 6 , Figure 6 This is a schematic diagram of knowledge transfer and distillation in the knowledge distillation method of this application. Figure 6This paper demonstrates the knowledge transfer and distillation process based on thought chains and knowledge graphs in this scheme. The method leverages the reasoning capabilities of large language models (LLMs) to train smaller models in a data-efficient manner. The process begins with the teacher model (LLM) predicting input data, generating labeled outputs, and providing detailed explanations of the predictions. These explanations are provided in natural language, containing the rationale behind the model's label predictions. Simultaneously, the teacher model extracts keywords from the input data and explanations, combining them with the knowledge graph to obtain highly relevant core information and knowledge points. Next, the student model learns and trains using the labels, explanations, and knowledge graphs provided by the teacher model. This scheme adds three task prefixes—answer, explanation, and knowledge graph—before the input examples and trains the model for multi-task learning based on these prefixes.

[0097] It should be noted that this solution, by combining thought chains and knowledge graphs, ensures that student models not only learn the correct output results but also understand the reasoning process and key knowledge points of the teacher model. This multi-layered knowledge transfer and distillation mechanism improves the learning efficiency and reasoning ability of student models, significantly enhancing their performance in practical applications. By efficiently utilizing the reasoning capabilities of large models and rich knowledge graph resources, this invention achieves comprehensive and systematic knowledge transfer, ensuring efficient training and excellent performance of student models.

[0098] In this embodiment, keyword extraction is performed on task text data and task execution thought chain to obtain several target keywords; relationship analysis and background expansion are performed on the task execution knowledge graph through the target keywords; upon completion of the expansion, the target keywords are scored for relevance using a teacher model; based on the scoring results, highly relevant keywords are determined from the target keywords, thereby ensuring that the knowledge nodes and relationships extracted in the knowledge graph are highly relevant, effectively avoiding the student model from encountering too many irrelevant knowledge points during the learning process, and reducing error accumulation and performance degradation caused by noise introduction.

[0099] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 7 , Figure 7 This is a schematic flowchart of the knowledge distillation method embodiment three of this application.

[0100] In this embodiment, after step S40, the knowledge distillation method further includes steps S50 to S70:

[0101] Step S50: Generate predicted labels based on the task text data using the teacher model.

[0102] It should be noted that the predicted labels mentioned above can be labels generated by the teacher model after predicting the input data.

[0103] Step S60: Obtain the explanatory information corresponding to the prediction result of the task to be performed through the teacher model.

[0104] It should be noted that the above explanatory information can be obtained after the teacher model has provided a detailed explanation of the prediction results.

[0105] Step S70: Optimize the student model based on the predicted labels, the explanatory information, and the task execution knowledge graph.

[0106] Specifically, step S70 includes:

[0107] Step S701: Determine the cross-entropy loss between the predicted label and the target label based on the explanation information and the task execution knowledge graph.

[0108] Step S702: Based on the predicted label and the cross-entropy loss, optimize the student model using a preset optimization loss function.

[0109] The preset optimization loss function is as follows:

[0110]

[0111] In the formula, L label Let N be the preset optimized loss function, N be the number of samples, and L be the cross-entropy loss. Let f be the predicted label, f be the student model, and x be the predicted label. i This is the input data for the student model.

[0112] It should be noted that by optimizing the loss function, the student model maximizes the learning and representation of knowledge from the teacher model within limited computational resources and data volume. In the absence of manually labeled data, task-specific distillation methods can use large language models (LLMs) to generate pseudo-noise training labels. Replace real labels (y i Smaller models f are trained by minimizing the label prediction loss, and the optimization loss function can be defined as:

[0113]

[0114] Further, step S701 includes: determining the explanation generation loss based on the explanation information; determining the knowledge graph information generation loss based on the key information in the task execution knowledge graph; and determining the cross-entropy loss between the predicted label and the target label based on the explanation generation loss and the knowledge graph information generation loss.

[0115] It should be understood that the above-mentioned explanation generation loss can be a loss used to measure the difference between the predicted explanation information and the actual explanation information; the above-mentioned knowledge graph information generation loss can be a loss used to measure the difference between the key information of the knowledge graph and the actual key information.

[0116] In practical applications, in order to input (x) i ) and prediction labels To create more explicit connections between them, this scheme can use the extracted interpretations. and key information of knowledge graphs As additional supervisory information, the problem of learning explanations and knowledge graph information can be framed as a multi-task problem in this invention. Specifically, the model can be trained... It not only predicts task labels but also generates corresponding explanations, key knowledge points, and relationships. The loss function (i.e., the cross-entropy loss mentioned above) is defined as follows:

[0117] L=λ1L label +λ2L explanation +(1-λ1-λ2)L knowledge-graph ;

[0118] In the formula, λ1 and λ2 are both weighting parameters, and L label To pre-define the optimization loss function, L explanation To explain the generation loss, L knowledge-graph The loss is generated for knowledge graph information.

[0119] In this embodiment, the interpretation generation loss can be defined as:

[0120]

[0121] In the formula, This indicates the interpretation of the student model extraction.

[0122] The loss for generating knowledge graph information is defined as:

[0123]

[0124] In the formula, It represents the key information of the knowledge graph.

[0125] It should be understood that, after learning and training, the student model is able to predict better new input questions. This knowledge transfer and distillation method based on thought chains and knowledge graphs effectively utilizes the ability of large language models to generate explanations, significantly improving the performance and generalization ability of the student model and providing a new solution for practical deployment.

[0126] In this embodiment, a method is disclosed that a teacher model generates predictive labels based on task text data; the teacher model obtains explanatory information corresponding to the prediction results of the task to be executed; and the student model is optimized based on the predictive labels, explanatory information, and task execution knowledge graph, so that the student model can predict better new input questions.

[0127] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the knowledge distillation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0128] This application also provides a knowledge distillation apparatus, please refer to... Figure 8 The knowledge distillation apparatus includes:

[0129] The thought chain construction module 10 is used to parse the task text data corresponding to the task to be executed in order to construct the task execution thought chain corresponding to the task to be executed, the task to be executed including: equipment fault diagnosis task;

[0130] Keyword extraction module 20 is used to extract keywords from the task text data and the task execution thought chain to obtain highly relevant keywords;

[0131] Knowledge network construction module 30 is used to construct a knowledge subnetwork based on the task execution knowledge graph and the highly relevant keywords;

[0132] The knowledge distillation module 40 is used to perform knowledge transfer and distillation on the student model through the knowledge subnet.

[0133] The knowledge distillation apparatus provided in this application, employing the knowledge distillation method described in the above embodiments, can solve the technical problem that existing knowledge distillation methods have difficulty in expressing and transmitting complex knowledge within large models, leading to student models being unable to fully understand and master this knowledge. Compared with the prior art, the beneficial effects of the knowledge distillation apparatus provided in this application are the same as those of the knowledge distillation method provided in the above embodiments, and other technical features in the knowledge distillation apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0134] This application provides a knowledge distillation apparatus, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the knowledge distillation method in Embodiment 1 above.

[0135] The following is for reference. Figure 9The diagram illustrates a structural schematic of a knowledge distillation device suitable for implementing embodiments of this application. The knowledge distillation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The knowledge distillation apparatus shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0136] like Figure 9 As shown, the knowledge distillation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the knowledge distillation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the knowledge distillation device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show knowledge distillation devices with various systems, it should be understood that implementing or possessing all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0137] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0138] The knowledge distillation apparatus provided in this application, employing the knowledge distillation method described in the above embodiments, can solve the technical problem of knowledge distillation. Compared with the prior art, the beneficial effects of the knowledge distillation apparatus provided in this application are the same as those of the knowledge distillation method described in the above embodiments, and other technical features of the knowledge distillation apparatus are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0139] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0140] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0141] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the knowledge distillation method described in the above embodiments.

[0142] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0143] The aforementioned computer-readable storage medium may be included in the knowledge distillation apparatus; or it may exist independently and not assembled into the knowledge distillation apparatus.

[0144] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the knowledge distillation device, the knowledge distillation device performs the following actions: parses the task text data corresponding to the task to be executed to construct a task execution thought chain corresponding to the task to be executed, wherein the task to be executed includes: a device fault diagnosis task; extracts keywords from the task text data and the task execution thought chain to obtain highly relevant keywords; constructs a knowledge subnetwork based on the task execution knowledge graph and the highly relevant keywords; and performs knowledge transfer and distillation on the student model through the knowledge subnetwork.

[0145] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0147] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0148] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described knowledge distillation method. This solves the technical problem in existing knowledge distillation methods where it is difficult to express and convey complex knowledge within large models, leading to student models being unable to fully understand and master this knowledge. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the knowledge distillation method provided in the above embodiments, and will not be repeated here.

[0149] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A knowledge distillation method, characterized in that, The method includes: The task text data corresponding to the task to be executed is parsed to construct the task execution thought chain corresponding to the task to be executed, the task to be executed includes: equipment fault diagnosis task; Keyword extraction is performed on the task text data and the task execution thought chain to obtain highly relevant keywords; A knowledge subnetwork is constructed based on the task execution knowledge graph and the highly relevant keywords; Knowledge transfer and distillation are performed on the student model through the aforementioned knowledge subnetwork; The step of extracting keywords from the task text data and the task execution thought chain to obtain highly relevant keywords includes: Keyword extraction is performed on the task text data and the task execution thought chain to obtain several target keywords; The target keywords are used to perform relationship analysis and background expansion on the task execution knowledge graph. Upon completion of the expansion, the target keywords are scored for relevance using a teacher model. Based on the scoring results, highly relevant keywords are identified from the target keywords; The step of constructing a knowledge subnetwork based on the task execution knowledge graph and the highly relevant keywords includes: Based on the highly relevant keywords, a knowledge node search is performed in the task execution knowledge graph to determine the target knowledge nodes and the node relationships between the target knowledge nodes. Construct a knowledge subnetwork based on the target knowledge nodes and the node relationships; After the step of knowledge transfer and distillation of the student model through the knowledge subnet, the method further includes: The teacher model generates predicted labels based on the task text data. The teacher model is used to obtain explanatory information corresponding to the prediction results of the task to be performed. The student model is optimized based on the predicted labels, the explanatory information, and the task execution knowledge graph.

2. The method as described in claim 1, characterized in that, The step of parsing the task text data corresponding to the task to be executed in order to construct the task execution thought chain corresponding to the task to be executed includes: Semantic analysis is performed on the task text data corresponding to the task to be executed in order to extract all semantic units in the task text data, as well as the semantic information between each semantic unit; The logical relationships between the semantic units are determined based on the semantic information. Based on the logical relationship, construct the task execution thought chain corresponding to the task to be executed.

3. The method as described in claim 1, characterized in that, The step of optimizing the student model based on the predicted labels, the explanatory information, and the task execution knowledge graph includes: Based on the explanatory information and the task execution knowledge graph, determine the cross-entropy loss between the predicted label and the target label; Based on the predicted labels and the cross-entropy loss, the student model is optimized using a preset optimization loss function; The preset optimization loss function is: In the formula, Let N be the preset optimized loss function, N be the number of samples, and L be the cross-entropy loss. Let f be the predicted label, and f be the student model. This is the input data for the student model.

4. The method as described in claim 3, characterized in that, The step of determining the cross-entropy loss between the predicted label and the target label based on the explanatory information and the task execution knowledge graph includes: The interpretation generation loss is determined based on the interpretation information; The knowledge graph information generation loss is determined based on the key information in the task execution knowledge graph. The cross-entropy loss between the predicted label and the target label is determined based on the explanation generation loss and the knowledge graph information generation loss.

5. A knowledge distillation apparatus, characterized in that, The device includes: The thought chain construction module is used to parse the task text data corresponding to the task to be executed in order to construct the task execution thought chain corresponding to the task to be executed, the task to be executed including: equipment fault diagnosis task; The keyword extraction module is used to extract keywords from the task text data and the task execution thought chain to obtain highly relevant keywords. The knowledge network construction module is used to construct knowledge subnetworks based on the task execution knowledge graph and the highly relevant keywords; The knowledge distillation module is used to perform knowledge transfer and distillation on the student model through the knowledge subnet; The keyword extraction module is further used to extract keywords from the task text data and the task execution thought chain to obtain several target keywords; to perform relationship analysis and background expansion on the task execution knowledge graph through the target keywords; to score the relevance of the target keywords through a teacher model when the expansion is completed; and to determine highly relevant keywords from the target keywords based on the scoring results. The knowledge network construction module is also used to search for knowledge nodes in the task execution knowledge graph based on the highly relevant keywords, so as to determine the target knowledge nodes and the node relationships between the target knowledge nodes; and to construct a knowledge sub-network based on the target knowledge nodes and the node relationships. The knowledge distillation module is further configured to generate prediction labels based on the task text data using the teacher model; obtain explanatory information corresponding to the prediction results of the task to be executed using the teacher model; and optimize the student model based on the prediction labels, the explanatory information, and the task execution knowledge graph.

6. A knowledge distillation apparatus, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the knowledge distillation method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the knowledge distillation method as described in any one of claims 1 to 4.

Images