Knowledge distillation method, device, equipment, storage medium and computer program product

Abstract

The application discloses a knowledge distillation method, device, equipment, storage medium and computer program product, relates to the technical field of artificial intelligence, and the method comprises the following steps: inputting a detection text into a teacher model, outputting a thinking chain and a global keyword, the thinking chain is used for representing the understanding and reasoning process of the teacher model on the detection text, and the global keyword is used for representing core information and knowledge points of reasoning thinking in the detection text; and knowledge in the teacher model is transmitted to a student model based on the thinking chain and the global keyword. Through the thinking chain construction, not only the reasoning ability of the student model is improved, but also the accuracy of knowledge transmission is ensured. Through the extraction of the global keyword, the understanding and application ability of the model on the text content is enhanced. Through the introduction of the global keyword and the thinking chain, the knowledge expression ability and generalization ability of the student model are effectively enhanced under the condition of limited data, and the learning performance of the student model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a knowledge distillation method and device, equipment, a storage medium and a computer program product. BACKGROUND

[0002] Large language models (LLMs) perform well in few-shot learning capabilities, but their large size makes deployment in practical applications extremely challenging. In particular, as of 2024, OpenAI's GPT-4 is known to be the largest LLM, with over one trillion (1T) parameters. Such a large model requires significantly more memory and computing resources, not only increasing the difficulty of deployment, but also placing extremely high demands on infrastructure. For most product teams, such computing requirements are difficult to bear, especially for applications that require low-latency performance.

[0003] Currently, in order to circumvent the deployment challenges of large-scale models, practitioners usually choose to deploy smaller specialized models, but the lack of data for small models makes the generated label data biased, resulting in a decline in the learning performance of student models.

[0004] Therefore, how to improve the learning performance of student models is a problem that needs to be solved at present. SUMMARY

[0005] The main purpose of the present application is to provide a knowledge distillation method, device, equipment, storage medium and computer program product, aiming to solve the technical problem of the decline in the learning performance of student models.

[0006] To achieve the above-mentioned purpose, the present application provides a knowledge distillation method, which comprises:

[0007] inputting a detection text into a teacher model to output a thought chain and a global keyword, the thought chain being used to represent the understanding and reasoning process of the teacher model for the detection text, and the global keyword being used to represent the core information and knowledge points of reasoning thought in the detection text;

[0008] based on the thought chain and the global keyword, transferring the knowledge in the teacher model to a student model.

[0009] In an embodiment, the step of inputting a detection text into a teacher model to output a thought chain and a global keyword comprises:

[0010] inputting a detection text into a teacher model, analyzing the semantic relationship and logical connection of the detection text through the teacher model, and constructing a thought chain according to the analysis result;

[0011] recognize the detection text based on the thought chain through the teacher model, extract global keywords from the detection text according to the recognition result, and output the thought chain and the global keywords.

[0012] In an embodiment, the step of analyzing semantic relations and logical connections of the detection text through the teacher model and constructing a thought chain according to an analysis result comprises:

[0013] performing data preprocessing on the detection text to obtain preprocessed text;

[0014] performing semantic analysis on the preprocessed text by using a deep learning model to obtain semantic units and semantic connections between the semantic units;

[0015] constructing logical connections of the semantic units based on the semantic connections, and constructing a thought chain according to the logical connections and the semantic units.

[0016] In an embodiment, the step of transferring knowledge in the teacher model to a student model based on the thought chain and the global keywords comprises:

[0017] inputting the thought chain and the global keywords into a to-be-optimized student model for training to output a student thought chain;

[0018] correcting the student thought chain according to the global keywords to obtain a target student thought chain;

[0019] transferring the knowledge in the teacher model to the student model based on the target student thought chain.

[0020] In an embodiment, the step of correcting the student thought chain according to the global keywords to obtain a target student thought chain comprises:

[0021] correcting the student thought chain by using the global keywords, and inputting the corrected student thought chain and the thought chain into the to-be-optimized student model for iterative training until a target student thought chain is obtained, wherein a deviation between the target student thought chain and the thought chain is less than a preset threshold.

[0022] In an embodiment, the step of correcting the student thought chain by using the global keywords, and inputting the corrected student thought chain and the thought chain into the to-be-optimized student model for iterative training comprises:

[0023] correcting the student thought chain by using the global keywords, and inputting the corrected student thought chain and the thought chain into the to-be-optimized student model;

[0024] According to the modified student thought chain and the thought chain, a loss function of the to-be-optimized student model is calculated:

[0025]

[0026] wherein L label represents a loss function, N represents a number of thought chains, represents a cross-entropy loss between a predicted label and a target label, f represents a to-be-optimized student model, x i represents a modified student thought chain, represents a predicted label;

[0027] The to-be-optimized student model is iteratively trained by using the loss function.

[0028] In addition, to achieve the above object, the present application further provides a knowledge distillation device, which comprises:

[0029] a knowledge analysis module, configured to input a detection text into a teacher model, and output a thought chain and a global keyword, wherein the thought chain is used to represent an understanding and reasoning process of the teacher model on the detection text, and the global keyword is used to represent core information and a knowledge point of reasoning thought in the detection text;

[0030] a knowledge output module, configured to transfer knowledge in the teacher model to a student model based on the thought chain and the global keyword.

[0031] In addition, to achieve the above object, the present application further provides a knowledge distillation device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the knowledge distillation method as described above.

[0032] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, wherein the computer program is executed by a processor to implement the steps of the knowledge distillation method as described above.

[0033] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, wherein the computer program is executed by a processor to implement the steps of the knowledge distillation method as described above.

[0034] The one or more technical solutions provided by the present application have at least the following technical effects:

[0035] The detection text is input into the teacher model, and a thought chain and a global keyword are output, the thought chain is used to represent the understanding and reasoning process of the teacher model on the detection text, and the global keyword is used to represent the core information in the detection text and the knowledge point of the reasoning thought, knowledge in the teacher model is transferred to the student model based on the thought chain and the global keyword. By simulating the human thinking process, a complex task is decomposed into a series of simple steps, so that the student model can gradually learn and master the processing method of the complex task, thereby improving its performance in the complex task. By extracting the global keyword, the model can more comprehensively capture and transfer knowledge, and improve the learning effect of the student model. Through the introduction of the global keyword and the thought chain, the knowledge expression ability and the generalization ability of the student model are effectively enhanced under the condition of limited data, and the learning performance of the student model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0038] Figure 1 The schematic diagram of the total technical framework of the embodiments of the present application is shown in the figure.

[0039] Figure 2 The schematic diagram of the flow of the first embodiment of the knowledge distillation method of the present application is shown in the figure.

[0040] Figure 3 The schematic diagram of the flow of the second embodiment of the knowledge distillation method of the present application is shown in the figure.

[0041] Figure 4 The schematic diagram of the global keyword extraction process of the embodiments of the present application is shown in the figure.

[0042] Figure 5 The schematic diagram of the thought chain construction of the embodiments of the present application is shown in the figure.

[0043] Figure 6 The schematic diagram of the flow of the third embodiment of the knowledge distillation method of the present application is shown in the figure.

[0044] Figure 7 The schematic diagram of the knowledge transfer and distillation framework based on the global keyword and the thought chain of the embodiments of the present application is shown in the figure.

[0045] Figure 8 The schematic diagram of the module structure of the knowledge distillation device of the embodiments of the present application is shown in the figure.

[0046] Figure 9 Figure 1 is a schematic diagram of a device structure of a hardware running environment involved in a knowledge distillation method in an embodiment of the present application.

[0047] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0049] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Since large language models (LLMs) require significantly more memory and computing resources, not only does it increase the difficulty of deployment, but it also puts extremely high requirements on the infrastructure. For most product teams, this computing requirement is difficult to bear, especially for applications that require low-latency performance. In addition, LLMs still have limitations in fully understanding the context, especially in cases involving complex reasoning or requiring domain-specific knowledge, the logic and accuracy of the generated text are often affected. In order to circumvent the deployment challenges of large-scale models, practitioners often choose to deploy smaller specialized models. These smaller models are trained through two common paradigms: fine-tuning and distillation. Fine-tuning applies a pre-trained model to a specific task and uses human-labeled data for further training, allowing the model to perform better within a specific domain. However, the fine-tuning effect is highly dependent on the quality and quantity of the labeled data, and obtaining high-quality labeled data often requires a lot of manpower and time. Distillation, on the other hand, uses the labels generated by large-scale LLMs to train smaller models, using a large-scale model as a teacher model to generate pseudo-label data for training smaller student models. This method can improve the performance of smaller models by utilizing the knowledge of large-scale models, but it also faces the challenge of requiring computing resources to generate high-quality pseudo-label data, as well as the problem of label data bias and noise. The knowledge distillation process is highly dependent on a large amount of high-quality training data, and if the data quality is not high or the data quantity is insufficient, the distillation effect will be significantly affected, limiting the generalization ability of the student model. Therefore, how to make a smaller student model better express the knowledge output of a teacher model under limited data conditions is still a major challenge that needs to be addressed.

[0051] The present application provides a solution that detects the text input into the teacher model, outputs the thought chain and the global keyword, the thought chain is used to represent the understanding and reasoning process of the teacher model for the detected text, and the global keyword is used to represent the core information and the knowledge point of the reasoning thought in the detected text. Based on the thought chain and the global keyword, the knowledge in the teacher model is transferred to the student model. For example,Figure 1 As shown, the relationship between the three parts of thought chain construction, global keyword extraction, and knowledge transfer and distillation in this application is demonstrated. The knowledge distillation method based on global keywords and thought chains frames the learning problem of explanations and global keywords as a multi-task problem, training the model not only to predict task labels but also to generate corresponding explanations and global keywords. By increasing the supervision information, the generalization ability and prediction accuracy of the student model are improved. Through thought chain construction, the semantic relationship and logical connection of the input data are analyzed to form a complete thought chain, which not only improves the reasoning ability of the student model but also ensures the accuracy and integrity of knowledge transfer. Using natural language processing technology, global keywords are extracted from the input data and the output of the teacher model, further enhancing the model's understanding and application of text content. This method significantly reduces the computational resources and data requirements of model deployment, improving the performance and adaptability of small-scale models in practical applications, and providing a new solution for the practical deployment of large-scale language models. By generating detailed natural language explanations using large language models, the reasons supporting the model's predicted labels are provided, enabling the student model to better understand and apply this knowledge, and improving the transparency and credibility of the model.

[0052] Based on this, the embodiment of the application provides a knowledge distillation method, referring to Figure 2 , Figure 2 The flowchart of the first embodiment of the knowledge distillation method of the application is shown in the figure.

[0053] In this embodiment, the knowledge distillation method includes steps S10-S20:

[0054] Step S10, input the detection text into the teacher model, and output the thought chain and global keyword.

[0055] It should be noted that the detection text can be understood as the text to be processed input into the teacher model, the teacher model can be a large and complex pre-trained model, the thought chain is used to represent the understanding and reasoning process of the teacher model for the detection text, and the global keyword is used to represent the core information and knowledge points of reasoning thought in the detection text.

[0056] For example, the teacher model extracts key information from the text based on the text content and reasoning process, constructs a logically coherent sequence of reasoning steps, and outputs the key information, key information extraction logic, and reasoning step sequence.

[0057] Step S20, based on the thought chain and the global keyword, transfer the knowledge in the teacher model to the student model.

[0058] For example, the thought chain and global keyword can be input into the student model for iterative training, and the student model performance can be gradually enhanced in multiple iterations. This approach can also be replaced by other progressive optimization methods.

[0059] In this embodiment, the detection text is input into the teacher model, and the thought chain and the global keyword are output. The thought chain is used to represent the understanding and reasoning process of the teacher model for the detection text, and the global keyword is used to represent the core information in the detection text and the knowledge point of the reasoning thought. Based on the thought chain and the global keyword, the knowledge in the teacher model is transferred to the student model. By simulating the human thinking process, the complex task is decomposed into a series of simple steps, so that the student model can gradually learn and master the processing method of the complex task, thereby improving its performance in the complex task. By extracting the global keyword, the model can more comprehensively capture and transfer knowledge, and improve the learning effect of the student model. Through the introduction of the global keyword and the thought chain, the knowledge expression ability and the generalization ability of the student model are effectively enhanced under the condition of limited data, and the learning performance of the student model is improved.

[0060] The first embodiment adopts the methods of multi-task learning framework, thought chain construction and optimization, and global keyword extraction. These methods can be replaced by alternative solutions such as multi-perspective knowledge distillation, semantic relationship construction of graph neural network, or keyword extraction based on contrastive learning. The above extension methods do not constitute a limitation on the knowledge distillation method of the present application, but in order for the student model to more accurately and completely capture the knowledge points of the teacher model during the learning process, the second embodiment of the present application is proposed.

[0061] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the knowledge distillation method of the present application is shown in the above Figure 2 Based on the first embodiment shown in the above

[0062] In the second embodiment, the step S10 comprises:

[0063] Step S101, input the detection text into the teacher model, analyze the semantic relationship and logical connection of the detection text through the teacher model, and construct the thought chain according to the analysis result.

[0064] It should be noted that the semantic relationship can be understood as the meaning association between words, sentences or paragraphs in the text, such as cause-effect relationship, parallel relationship, transition relationship, etc., and the logical connection can be understood as the associativity between the parts of the text based on logical reasoning, such as sequentiality, hierarchy, conditionality, etc. The analysis result can be the result obtained by the teacher model analyzing the semantic relationship and logical connection of the detection text.

[0065] Exemplarily, the detection text can be pre-processed as necessary, such as removing noise (such as HTML tags, special characters, etc.), and the teacher model uses natural language processing techniques (such as dependency syntax analysis, semantic role labeling, etc.) to identify semantic relationships in the detection text, analyze the interaction and connection between words in the text, such as identifying subject, predicate, object, etc. sentence components, and their relationships such as modification, limitation, etc. Further, the teacher model analyzes the logical connections in the text, such as the order between paragraphs, the cause-and-effect relationship between sentences, the transition relationship, etc., constructs the relevance network between the parts of the text through logical reasoning, and constructs the thinking chain according to the analysis results.

[0066] In step S102, the detection text is identified by the teacher model based on the thinking chain, and global keywords are extracted from the detection text according to the identification result, and the thinking chain and the global keywords are output.

[0067] It should be noted that the global keyword extraction identifies key words and phrases by analyzing the input data and the understanding and reasoning process of the teacher model for the data.

[0068] Exemplarily, as shown in Figure 4 By example guiding GPT-3.5, the key words and phrases are identified by analyzing the input data and the understanding and reasoning process of the teacher model (such as GPT-3.5) for the data. These key words represent the core information in the text and the knowledge points of reasoning thinking. The global keyword extraction not only analyzes the input data, but also includes the reasoning information in the thinking chain. In the entire analysis process, the teacher model can identify the key words and phrases in the global information, and these key words accurately reflect the content and meaning of the text. The teacher model identifies the core information and key content in the detection text according to the constructed thinking chain, and these core information is usually closely related to the theme, central idea or main argument of the text. From the identified core information, the teacher model extracts global keywords that can summarize the core content and theme of the text, and these keywords should be representative and discriminative, which can help readers quickly understand the main content of the text. The teacher model constructs the thinking chain and extracts the global keywords as output results.

[0069] The embodiment inputs the detection text into the teacher model, analyzes the semantic relationship and logical connection of the detection text through the teacher model, constructs a thinking chain according to the analysis result, and identifies the detection text through the teacher model based on the thinking chain, extracts global keywords from the detection text according to the identification result, and outputs the thinking chain and the global keywords. The thinking chain can reflect the understanding and reasoning process of the teacher model for the input data, thereby improving the learning effect and performance of the student model. The global keyword extraction enables the model to quickly locate and understand the core information in the text, thereby reducing the processing time. The model's understanding and application ability of the text content are enhanced, so that the student model can more accurately capture the key knowledge points in the teacher model during the learning process. Through this method, not only the overall performance of the student model is improved, but also the ability to handle complex tasks is stronger.

[0070] In an embodiment, the step of analyzing the semantic relationship and logical connection of the detection text through the teacher model, and constructing a thinking chain according to the analysis result, comprises: data preprocessing of the detection text to obtain preprocessed text; performing semantic analysis on the preprocessed text using a deep learning model to obtain semantic units and semantic connections between the semantic units; constructing logical connections of each semantic unit based on the semantic connections, and constructing a thinking chain according to the logical connections and the semantic units.

[0071] For example, as shown in Figure 5 GPT-3.5 can be used as a teacher model to construct a thinking chain, and a CQA dataset can be used. First, the CQA dataset is preprocessed, including removing meaningless characters, stop words and other noise data. Then, the API interface of GPT-3.5 is called for semantic analysis, logical relationship construction and thinking chain optimization. By guiding GPT-3.5, it can analyze the problem step by step and generate the reasoning process of the correct answer.

[0072] In this embodiment, data preprocessing, semantic analysis, logical relationship construction and thinking chain optimization are performed. Data preprocessing is performed through text cleaning, word segmentation and part-of-speech tagging to ensure data quality and consistency. Semantic analysis uses a deep learning model to extract semantic units and their relationships to identify semantic connections between words. Logical relationship construction is based on these semantic units to show the reasoning path and logical structure between knowledge points. Finally, the large model outputs the thinking chain to ensure its accuracy and completeness.

[0073] Referring to Figure 6 , Figure 6 is a flowchart of the third embodiment of the knowledge distillation method of the present application, based on the second embodiment shown in Figure 3 The third embodiment of the knowledge distillation method of the present application is proposed based on the second embodiment shown in

[0074] In the third embodiment, the step S20 comprises:

[0075] Step S201, input the thought chain and the global keyword into the student model to be optimized for training, and output the student thought chain.

[0076] It should be noted that the student thought chain can be understood as a sequence of reasoning steps generated by the student model after receiving the thought chain input and processing it through its internal algorithm. It can be understood that due to the low initial performance of the student model, the generated thought chain may differ from the real or ideal thought chain.

[0077] Step S202, correct the student thought chain according to the global keyword, and obtain the target student thought chain.

[0078] It should be noted that the target student thought chain is a student thought chain that is closer to the real or ideal situation after being corrected by the global keyword, representing an optimized output of the student model on a specific task.

[0079] For example, first, the global keyword is obtained from the input text using natural language processing technology. Next, a preliminary reasoning step is generated using a large language model. We gradually generate reasoning steps through sample prompts, which preliminarily explain how to derive possible conclusions from the input text. Combining the global keyword, the preliminary generated reasoning steps are corrected and optimized. In this way, the generated thought chain not only reflects the reasoning ability of the large language model, but also combines key knowledge to ensure the accuracy and reliability of the reasoning process. Finally, the optimized reasoning steps are organized into a coherent thought chain. The thought chain contains all the logical steps from the input text to the final conclusion, and each step is based on key knowledge and reasonable reasoning paths.

[0080] Step S203, based on the target student thought chain, transfer the knowledge in the teacher model to the student model.

[0081] For example, the step-by-step distillation optimization method can be used to optimize the knowledge transfer process through multiple rounds of optimization. For example, Figure 7As shown, in each round of distillation training, the student model is trained using the answers provided by the teacher model while mimicking the reasoning process of the teacher model. This training method not only focuses on the accuracy of the final answer but also pays attention to the rationality and coherence of the reasoning process. The student model generates its own reasoning path by learning the reasoning chain of the teacher model, thereby understanding and mastering the knowledge in a specific domain. Through multiple iterations, the reasoning ability and answer accuracy of the student model gradually improve, approaching the level of the teacher model. After each round of training, the reasoning results of the student model are compared and corrected with the reasoning process of the teacher model. By combining new training data and updated thought chains, the student model is optimized.

[0082] In this embodiment, the thought chain and global keywords are input into the student model to be optimized for training, and the student thought chain is output. The target student thought chain is obtained by correcting the student thought chain according to the global keywords. Based on the target student thought chain, the knowledge in the teacher model is transferred to the student model. By combining the key knowledge in the global keywords, the reasoning steps generated initially are corrected and optimized. Each round of optimization not only improves the reasoning ability of the student model, but also avoids the reasoning bias caused by relying solely on data-driven through the correction of reasoning steps. This step-by-step correction and optimization method ensures the integrity and effectiveness of knowledge transfer, enabling the student model to perform well in problem solving in a specific domain. The introduction of the thought chain makes the reasoning process transparent, allowing users to clearly see the basis and logic of each reasoning step, thereby enhancing trust in the output results of the model. Through this technical solution, the model can provide more reliable and valuable results in actual applications.

[0083] In one embodiment, based on the third embodiment described above, the step S202 includes: correcting the student thought chain with the key information, and inputting the corrected student thought chain and the thought chain into the student model for iterative training until a target student thought chain is obtained, the deviation between the target student thought chain and the thought chain being less than a preset threshold.

[0084] It should be noted that the loss function of the student model can be calculated based on the corrected student thought chain and the thought chain, and the student model can be iteratively trained using the loss function. In the absence of human-labeled labels, the task-specific distillation method uses a large language model (LLM) to generate pseudo-noise training labels instead of real labels (y i ). Smaller models f are all trained by minimizing the label prediction loss. To create a more explicit connection between the input (x i ) and the predicted label , we use the extracted explanations and keywords as additional supervision information. The learning explanation and the answer to the question are framed as a multi-task problem. Specifically, the model is trained not only to predict the task label, but also to generate the corresponding explanation and keywords. Further, the overall cross-entropy loss can be defined as formula (1):

[0085] L = λ1L label + λ2L exp lanation + (1-λ1-λ2)L keyword (1)

[0086] where L is the cross-entropy loss between the predicted label and the target label, λ1and λ2represent the loss function calculation weights, L label represents the loss function, L explanation is the explanation generation loss, and L keyword is the global keyword generation loss.

[0087] L label The loss function can be represented as formula (2):

[0088]

[0089] In formula (2), L label is the loss function, N represents the number of thought chains, represents the cross-entropy loss between the predicted label and the target label, f represents the student model, x i represents the corrected student thought chain, and represents the predicted label.

[0090] In formula (1), L explanation is the explanation generation loss, defined as formula (3):

[0091]

[0092] In formula (3), L explanation is the explanation generation loss, N represents the number of thought chains, represents the cross-entropy loss between the predicted label and the target label, f represents the student model, x i represents the corrected student thought chain, and represents the extracted explanation.

[0093] In formula (1), L keyword is the global keyword generation loss, defined as formula (4):

[0094]

[0095] In formula (4), L keyword is the global keyword generation loss, N represents the number of thought chains, represents the cross-entropy loss between the predicted label and the target label, f represents the student model, x i represents the modified student thought chain, represents the keyword.

[0096] In this embodiment, by optimizing the loss function, the student model maximizes the learning and expression of the knowledge in the teacher model under limited computing resources and data volume. After learning and training, the student model can make predictions on better new input problems. This step-by-step distillation method through global keywords and thought chains effectively utilizes the ability of large language models to generate explanations, significantly improves the performance and generalization ability of the student model, and provides a new solution for actual deployment.

[0097] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the knowledge distillation method of the present application. More forms of simple changes based on this technical concept are within the protection scope of the present application.

[0098] The present application also provides a knowledge distillation device, please refer to Figure 8 , the knowledge distillation device comprises:

[0099] The knowledge analysis module 10 is used for inputting the detection text into the teacher model, outputting the thought chain and the global keyword, the thought chain is used to represent the understanding and reasoning process of the teacher model to the detection text, and the global keyword is used to represent the core information and the knowledge point of the reasoning thought in the detection text;

[0100] The knowledge output module 20 is used for transferring the knowledge in the teacher model to the student model based on the thought chain and the global keyword.

[0101] The knowledge distillation device provided by the present application adopts the knowledge distillation method in the above embodiment, which can solve the technical problem of performance decline of the student model. Compared with the prior art, the knowledge distillation device provided by the present application has the same beneficial effects as the knowledge distillation method provided by the above embodiment, and other technical features in the knowledge distillation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0102] The present application provides a knowledge distillation device, the knowledge distillation device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor to enable the at least one processor to execute the knowledge distillation method in the above embodiment one.

[0103] The following refers to Figure 9The diagram illustrates a structural schematic of a knowledge distillation device suitable for implementing embodiments of this application. The knowledge distillation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The knowledge distillation apparatus shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 9 As shown, the knowledge distillation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the knowledge distillation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the knowledge distillation device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 Knowledge distillation apparatus with various systems is shown; however, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0105] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0106] The knowledge distillation device provided by the present application adopts the knowledge distillation method in the above-mentioned embodiments, and can solve the technical problem of the decline of the learning performance of the student model. Compared with the prior art, the knowledge distillation device provided by the present application has the same beneficial effects as the knowledge distillation method provided by the above-mentioned embodiments, and other technical features in the knowledge distillation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0107] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0108] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0109] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the knowledge distillation method in the above-mentioned embodiments.

[0110] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination thereof.

[0111] The above computer readable storage medium may be contained in the knowledge distillation device, or may exist separately without being assembled into the knowledge distillation device.

[0112] The above computer readable storage medium carries one or more programs, when the one or more programs are executed by the knowledge distillation device, the knowledge distillation device is caused to: input the detection text into the teacher model, output the thought chain and the global keyword, the thought chain is used to represent the understanding and reasoning process of the teacher model to the detection text, and the global keyword is used to represent the core information and the knowledge point of the reasoning thought in the detection text, and based on the thought chain and the global keyword, the knowledge in the teacher model is transmitted to the student model.

[0113] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0114] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0115] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0116] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the knowledge distillation method described above, and can solve the technical problem of the decline of the learning performance of the student model. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the knowledge distillation method provided by the above embodiments, which will not be repeated here.

[0117] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the knowledge distillation method as described above.

[0118] The computer program product provided by the application can solve the technical problem of the decline of the learning performance of the student model. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the knowledge distillation method provided by the above-mentioned embodiments, and are not described here.

[0119] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or direct / indirect application in other related technical fields based on the technical concept of the application and the content of the specification and drawings are included in the patent protection scope of the application.

Claims

1. A knowledge distillation method, characterized in that, The method includes: The detection text is input into the teacher model, and the output is the thought chain and global keywords. The thought chain is used to represent the teacher model's understanding and reasoning process of the detection text, and the global keywords are used to represent the core information and knowledge points of reasoning in the detection text. Based on the aforementioned thought chain and global keywords, the knowledge in the teacher model is transferred to the student model; The steps of inputting the detected text into the teacher model and outputting the thought chain and global keywords include: The detected text is input into the teacher model, which analyzes the semantic relationships and logical connections of the detected text and constructs a thought chain based on the analysis results. Based on the thought chain, the teacher model is used to identify the detection text. Global keywords are extracted from the detection text according to the identification results, and the thought chain and the global keywords are output. The step of transferring knowledge from the teacher model to the student model based on the thought chain and the global keywords includes: The thought chain and global keywords are input into the student model to be optimized for training, and the student thought chain is output. The student's thought process chain is modified based on the global keywords to obtain the target student's thought process chain; Based on the target student's thought chain, the knowledge in the teacher model is transferred to the student model.

2. The method as described in claim 1, characterized in that, The steps of analyzing the semantic relationships and logical connections of the detected text using the teacher model and constructing a thought chain based on the analysis results include: The detected text is preprocessed to obtain preprocessed text; The preprocessed text is semantically analyzed using a deep learning model to obtain semantic units and semantic relationships between them. Based on the semantic relationships, logical connections are constructed between the semantic units, and a thought chain is constructed based on the logical connections and the semantic units.

3. The method according to any one of claims 1 to 2, characterized in that, The step of correcting the student's thought chain based on the global keywords to obtain the target student's thought chain includes: The student's thought process chain is corrected using the global keywords, and the corrected student thought process chain and the thought process chain are input together into the student model to be optimized for iterative training until the target student thought process chain is obtained, and the deviation between the target student thought process chain and the thought process chain is less than a preset threshold.

4. The method as described in claim 3, characterized in that, The step of using the global keywords to correct the student's thought process chain, and then inputting the corrected student thought process chain and the thought process chain together into the student model to be optimized for iterative training, includes: The student's thought process chain is corrected using the global keywords, and the corrected student thought process chain and the thought process chain are input into the student model to be optimized. The loss function of the student model to be optimized is calculated based on the revised student thought chain and the thought chain: in, Represents the loss function. Indicates the number of thought chains. This represents the cross-entropy loss between the predicted label and the target label. This indicates a student model that needs optimization. This represents the revised student thought process. Indicates the predicted label; The student model to be optimized is iteratively trained using the loss function.

5. A knowledge distillation apparatus, characterized in that, The device includes: The knowledge analysis module is used to input the detection text into the teacher model and output the thought chain and global keywords. The thought chain is used to represent the teacher model's understanding and reasoning process of the detection text, and the global keywords are used to represent the core information and knowledge points of reasoning in the detection text. The knowledge output module is used to transfer the knowledge in the teacher model to the student model based on the thought chain and the global keywords. The knowledge analysis module is also used to input the detection text into the teacher model, analyze the semantic relationships and logical connections of the detection text through the teacher model, and construct a thought chain based on the analysis results; based on the thought chain, the teacher model identifies the detection text, extracts global keywords from the detection text based on the identification results, and outputs the thought chain and the global keywords; The knowledge output module is also used to input the thought chain and global keywords into the student model to be optimized for training, and output the student thought chain; correct the student thought chain according to the global keywords to obtain the target student thought chain; and transfer the knowledge in the teacher model to the student model based on the target student thought chain.

6. A knowledge distillation apparatus, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the knowledge distillation method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the knowledge distillation method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the knowledge distillation method as described in any one of claims 1 to 4.

Images