Semi-supervised software defect prediction method based on graph representation learning and knowledge distillation

Abstract

The application discloses the following technical solutions, first, the graph structure and abstract syntax tree structure are extracted from the source code; then the word vector sequence is obtained by encoding the abstract syntax tree information as the input of the bidirectional recurrent neural network to learn the semantic features of the source code, and the obtained semantic features and the traditional static features are combined to be used as the state vector representation of the graph node; then the teacher integrated network model is pre-trained by using the complete graph representation of the source code; finally, the knowledge is extracted from the teacher integrated network model pre-trained previously and injected into the student integrated network model by the knowledge distillation technology. With the advancement of the graph neural network and the idea of the knowledge distillation technology, compared with the prior art, the generated student integrated network model can realize a higher software defect detection rate, and the design of the integrated network guarantees the robustness and the robustness of the model.

Description

TECHNICAL FIELD

[0001] The application relates to a semi-supervised software defect prediction method based on graph representation learning and knowledge distillation, and belongs to the field of software analysis and defect prediction in software engineering. BACKGROUND

[0002] Software defect prediction technology has been one of the most active contents in the field of software engineering since the 1970s, and plays an important role in analyzing software quality and balancing software cost. In recent years, with the availability of mass storage capacity, the emergence of high-speed Internet and Internet of Things devices, software has become an important factor affecting the national economy, politics, military and even social life. At the same time, the size and complexity of modern software systems are growing, making defects more deeply and complexly hidden and more challenging to detect. Therefore, the traditional software testing method based on manual review requires developers to invest more cost and effort in the testing or debugging process, and it is difficult to ensure the timeliness and accuracy of software testing. In this case, scholars have proposed software defect prediction technology to help developers determine whether there are possible defect problems in the software and guide developers to focus limited software quality assurance resources on high-risk defect modules. The related patents of software defect prediction mainly include: an instant software defect prediction method based on code representation learning (publication number CN 111858323); a software defect prediction model based on deep neural network and probability decision forest (publication number CN 109446090); a software defect prediction method based on convolutional neural network (publication number CN 108829607) and the like.

[0003] In previous studies, machine learning techniques have been widely applied to the defect prediction task of software, and the related work is mainly devoted to designing and extracting metric features in software systems, such as McCabe metric features for measuring code complexity, to build prediction classification models based on software metric features. With the advanced performance of graph neural networks in natural language processing, recommendation systems and many other fields, the representation and learning of source code on the graph have also been widely studied. In 2020, Wang et al. focused on the cyclic structure in source code through graph representation learning, and learned the representation of source code on the graph by constructing an interval graph neural network, which to some extent solved the oversmoothing problem in the process of extending the graph neural network to the large graph and the gradient disappearance problem caused by the message passing strategy. The experimental results show that their work effectively improves the performance of graph neural networks in the field of software engineering. However, the method based on graph neural network, due to the prediction mechanism of message passing strategy and complex aggregation operation, can not fully utilize the prior knowledge based on structure and features, and the prior knowledge based on structure and features is often very important in software defect prediction problems, for example, two modules with inheritance, calling and other dependency relationships are more inclined to have the same class label. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a semi-supervised software defect prediction method based on graph representation learning and knowledge distillation technology. The present application inherits the advantages of graph neural networks, and solves the problems of insufficient labels in supervised learning and the inability of graph neural networks to fully utilize prior knowledge based on structure and features by means of collaborative training and knowledge distillation.

[0005] The present application adopts the following technical solutions: a software defect prediction method for source code; a graph representation learning method for source code and a construction method of a semi-supervised software defect prediction model based on knowledge distillation; wherein the graph representation learning method for source code comprises the following steps:

[0006] Step A. Select a predetermined number of source codes of various types as samples, and extract abstract syntax tree structure and graph structure from the source code, and then obtain the static features of each module in the source code as the initial vector representation of the nodes in the graph structure and tree structure through a static code analysis tool, and then enter step B;

[0007] Step B. The abstract syntax tree structure is further divided into sentence-level tree input Word2vec model to encode as word vector, and then the word vector is input into the bidirectional recurrent neural network to learn the semantic features of the source code, and then enter step C;

[0008] Step C. The semantic features learned in step B are spliced with the traditional static features extracted in step A to form a new node state vector, and feature dimension reduction is performed to remove redundant features and noise to obtain the final representation of the module corresponding node, and then step D is entered;

[0009] Step D. The final node representation obtained in step C is embedded into the graph structure extracted in step A as the initial state vector of the node.

[0010] Based on the construction of the semi-supervised software defect prediction model of knowledge distillation, according to the following steps I to step IV, the knowledge is extracted from the teacher integrated network model and injected into the student integrated network model through the knowledge distillation technology, and then the trained student network model realizes more efficient and accurate software defect prediction:

[0011] Step I. Pre-train the teacher integrated network model by using only a small number of class labels based on the difference-based co-training algorithm, and the termination condition of pre-training is that the performance of all sub-classifiers in the teacher integrated network model no longer improves or the labeled data set no longer expands, and the termination condition is reached to enter step II;

[0012] Step II. Match the sub-classifiers in the teacher integrated network with the branches in the student integrated network one by one, and perform the first knowledge distillation to train the internal parameters of the branches in the student integrated network, and then enter step III;

[0013] Step III. The soft labels generated by the sub-classifiers in the teacher integrated network are weighted and aggregated as the final result of the teacher integrated network model, and the final result of the teacher integrated network is used for the second knowledge distillation with the final result of the student integrated network to train the weights of each branch in the student integrated network.

[0014] As a preferred technical solution of the present application, the step A extracts the abstract syntax tree structure and the graph structure of the source code according to the lexical analysis and the control flow graph of the source code, respectively, including the following sub-steps:

[0015] Step A1. The program source code is parsed into an abstract syntax tree structure by applying the javalang tool, and the function nodes are cut out from the abstract syntax tree structure;

[0016] Step A2. The third-party tool understand is applied to construct an adjacency matrix according to the control flow graph of the source code to represent the dependency relationship between modules to obtain the graph structure of the source code;

[0017] Step A3. The static analysis tool is applied to extract the static features of the nodes corresponding to the modules in the graph structure and the abstract syntax tree structure to initialize the node state vector.

[0018] As a preferred embodiment of the present invention, step B learns the semantic features of the source code using the Word2vec model and a bidirectional recurrent neural network according to steps B1 to B3 as follows:

[0019] Step B1. Traverse the abstract syntax tree structure obtained in step A1, map all user-defined names to fixed names, and transform the AST into a node sequence through pre-sorted traversal. Using this method, we extract function-level code, remove redundant information, and retain complete syntax information. This algorithm is very effective when processing a large number of programs.

[0020] Step B2. Map the sentence-level trees obtained in Step B1 to their word vector sequence representations using the word2vec model so that they can be directly trained by the neural network;

[0021] Step B3. Input the word vector sequence obtained in step B2 into a bidirectional recurrent neural network for training, and finally obtain a vector representation of the semantic features of the source code.

[0022] As a preferred embodiment of the present invention, step C is not a simple concatenation of traditional static features and semantic features learned in step B3, but is performed according to steps C1 to C2 below, in order to retain important feature information while removing noise and redundant features, thereby improving data processing speed. The specific sub-steps are as follows:

[0023] Step C1. Concatenate the traditional static features obtained in step A3 with the semantic features of a specified dimension learned through the bidirectional recurrent neural network in step B3 to form a new vector representation;

[0024] Step C2. Apply SPSS software to standardize the vector representation obtained in step C1. Next, determine the correlation between features, then set the number of principal components and perform principal component analysis according to the principal component expression. Finally, obtain the dimensionality-reduced feature vector as the final representation of the node.

[0025] As a preferred embodiment of the present invention, step I includes the following sub-steps:

[0026] Step I 1. Divide the graph nodes into two parts: a labeled dataset and an unlabeled dataset;

[0027] Step 1.2. Train the teacher ensemble network using the labeled dataset and use the trained teacher ensemble network to label the nodes in the unlabeled dataset with pseudo-labels;

[0028] Step I 3. Calculate the confidence of nodes with pseudo-labels. Add nodes with confidence scores higher than a preset threshold to the labeled dataset, and use their corresponding pseudo-labels as real labels in the next training. Nodes with confidence scores lower than the preset threshold are kept in the unlabeled dataset.

[0029] Step I4. Repeat Step I2 and Step I3 until the performance of all subclassifiers in the teacher ensemble network no longer improves or the labeled dataset no longer expands.

[0030] As a preferred embodiment of the present invention, step II includes the following sub-steps:

[0031] Step II 1. Match the subclassifiers in the pre-trained teacher ensemble network with the branches in the student ensemble network one by one;

[0032] Step II.2. Perform the first distillation using the differences between the soft labels generated by the subclassifiers in the pre-trained teacher ensemble network and the soft labels generated by each branch in the student ensemble network z. The distillation function is as follows:

[0033]

[0034] Where K is a hyperparameter representing the number of layers in the branch, ||||2 is the L2 norm, and Θ represents the set of trainable parameters.

[0035] As a preferred embodiment of the present invention, the specific sub-steps of step III are as follows:

[0036] Step III 1. Initialize the weights of each branch in the student ensemble network model using the mean;

[0037] Step III 2. Interpolate and aggregate the soft labels of each subclass in the teacher ensemble network as the final soft labels generated by the teacher ensemble network model, and weight and aggregate the soft labels of each branch in the student ensemble network as the final soft labels generated by the student ensemble network model.

[0038] Step III.3. Based on the differences between the soft labels generated by the teacher ensemble network model and the student ensemble network model obtained in Step III.2, a second ensemble knowledge distillation is performed to train the weights of each branch in the student ensemble network model. The ensemble distillation function is expressed as follows:

[0039] L KD =L KL (S A ,T A / T)+L MSE (S A ,T A )

[0040] in and These represent the Kullback-Leibler divergence loss and the mean square error loss, respectively. T is the distillation temperature hyperparameter, which controls the soft label output by the teacher subnetwork.

[0041] As a preferred embodiment of the present invention, the teacher ensemble network model is composed of graph neural networks with various different structures to utilize knowledge derived from advanced graph neural networks and learn knowledge from different sub-classifiers from different perspectives. The branch structure in the student ensemble network is a trainable combination of parameterized label propagation algorithm and multilayer perceptron to compensate for the shortcomings of graph neural networks in fully utilizing prior knowledge based on structure and features.

[0042] The beneficial effects of this invention are as follows:

[0043] This technical solution focuses on the problem of semi-supervised software defect prediction, and comprehensively utilizes graph representation learning and knowledge distillation techniques to improve the accuracy and efficiency of the semi-supervised software defect prediction model.

[0044] The technical reasons for achieving the above results in this technical solution are as follows: 1) Traditional code representation methods, whether treating code modules as isolated nodes or simply treating them as natural language processing, cannot fully utilize the unique features of the code itself. This invention, by extracting the graph structure of the code and embedding semantic features into the graph nodes, makes more comprehensive and full use of the information contained in the code; 2) Graph neural networks cannot fully utilize prior knowledge based on structure and features during the prediction process due to complex prediction mechanisms such as message passing strategies and aggregation operations. This invention, by carefully designing the structure of branches in the student ensemble network, incorporates the aforementioned prior knowledge using a parameterized label propagation algorithm and a trainable combination of multilayer perceptrons, and combines prior knowledge with the advanced features of graph neural networks through knowledge distillation technology, effectively compensating for the shortcomings of graph neural networks; 3) In the single-teacher distillation method, the student model is overly dependent on the performance of the teacher model. If the teacher model's performance is poor, there is a risk of misleading the student model. Although the traditional multi-teacher distillation method utilizes the knowledge of multiple teachers, simple aggregation operations weaken the differences between different teachers, affecting the training effect. This invention, through two distillations, trains the internal parameters of each branch in the student ensemble network and the weights of each branch in the overall network, making more comprehensive and effective use of knowledge from multiple teachers. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0046] Figure 1 This is a flowchart illustrating the semi-supervised software defect prediction method based on graph representation learning and knowledge distillation techniques.

[0047] Figure 2 This is a schematic diagram of the semantic feature extraction method in this embodiment.

[0048] Figure 3 This is a schematic diagram of the pre-trained teacher ensemble network model method in this embodiment of the method.

[0049] Figure 4 This is a schematic diagram of the framework of the multi-teacher distillation process in this embodiment of the method. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments:

[0051] The present invention provides a semi-supervised software defect prediction method based on graph representation learning and knowledge distillation techniques. This method comprehensively utilizes graph representation learning and knowledge distillation techniques, thereby improving the accuracy and efficiency of the semi-supervised software defect prediction model. Figure 1 and Figure 2 As shown, the specific embodiments of the present invention are as follows:

[0052] Step A. Select a preset number of source codes of various types as samples, and extract abstract syntax tree structure and graph structure from the source code. Then, use static code analysis tools to obtain the static features of each module in the source code as the initial vector representation of the nodes in the graph structure and tree structure, and then proceed to step B.

[0053] Step A1. Use the javalang tool to parse the program source code into an abstract syntax tree structure and extract function nodes from it;

[0054] Step A2. Use the third-party tool understand to construct an adjacency matrix based on the control flow graph of the source code to represent the dependencies between modules and obtain the graph structure of the source code;

[0055] Step A3. Use static code analysis tools to extract the static features of the modules corresponding to the nodes in the graph structure and abstract syntax tree structure, and initialize the node state vectors;

[0056] In practical applications, since program code is different from ordinary natural language text, this invention must design it into an effective encoding sequence that the language model can recognize. Therefore, we continue to perform the following step B.

[0057] Step B. Further divide the abstract syntax tree structure into statement-level trees and input them into the Word2vec model to be encoded as word vectors. Then, use the word vectors as input to a bidirectional recurrent neural network to learn the semantic features of the source code, and then proceed to step C.

[0058] Step B1. Traverse the abstract syntax tree structure obtained in step A1, map all user-defined names to fixed names, and transform the AST into a node sequence through pre-sorted traversal. Using this method, we extract function-level code, remove redundant information, and retain complete syntax information. This algorithm is very effective when processing a large number of programs.

[0059] Step B2. Map the sentence-level trees obtained in Step B1 to their word vector sequence representations using the word2vec model so that they can be directly trained by the neural network;

[0060] Step B3. Input the word vector sequence obtained in step B2 into a bidirectional recurrent neural network for training, and finally obtain a vector representation of the semantic features of the source code.

[0061] Step C. Concatenate the semantic features learned in Step B with the traditional static features extracted in Step A to form a new node state vector, and perform feature dimensionality reduction to remove redundant features and noise to obtain the final representation of the corresponding node of the module, and then proceed to Step D.

[0062] Step C1. Concatenate the traditional static features obtained in step A3 with the semantic features of a specified dimension learned through the bidirectional recurrent neural network in step B3 to form a new vector representation;

[0063] Step C2. Apply SPSS software to standardize the vector representation obtained in step C1. Next, determine the correlation between features, then set the number of principal components and perform principal component analysis according to the principal component expression. Finally, obtain the dimensionality-reduced feature vector as the final representation of the node.

[0064] Step D. Embed the final node representation obtained in Step C into the graph structure extracted in Step A as the initial state vector of the node;

[0065] like Figure 4 As shown, the process of extracting knowledge from the teacher ensemble network model and injecting it into the student ensemble network model is achieved by performing two distillations according to steps I to III:

[0066] Step I. As Figure 3 As shown, the teacher ensemble network model is pre-trained using only a small number of class labels through the difference-based collaborative training algorithm. The termination condition for pre-training is that the performance of all sub-classifiers in the teacher ensemble network model no longer improves or the labeled dataset no longer expands. Once the termination condition is met, the process proceeds to step II.

[0067] Step I 1. Divide the graph nodes into two parts: a labeled dataset and an unlabeled dataset;

[0068] Step 1.2. Train the teacher ensemble network using the labeled dataset and use the trained teacher ensemble network to label the nodes in the unlabeled dataset with pseudo-labels;

[0069] Step I 3. Calculate the confidence of nodes with pseudo-labels. Add nodes with confidence scores higher than a preset threshold to the labeled dataset, and use their corresponding pseudo-labels as real labels in the next training. Nodes with confidence scores lower than the preset threshold are kept in the unlabeled dataset.

[0070] Step I4. Repeat Step I2 and Step I3 until the performance of all subclassifiers in the teacher ensemble network no longer improves or the labeled dataset no longer expands.

[0071] Step II. Match the subclassifiers in the teacher ensemble network with the branches in the student ensemble network one by one to perform the first knowledge distillation to train the internal parameters of the branches in the student ensemble network, and then proceed to Step III;

[0072] Step II 1. Match the subclassifiers in the pre-trained teacher ensemble network with the branches in the student ensemble network one by one;

[0073] Step II.2. Perform the first distillation using the differences between the soft labels generated by the subclassifiers in the pre-trained teacher ensemble network and the soft labels generated by each branch in the student ensemble network z. The distillation function is as follows:

[0074]

[0075] Where K is a hyperparameter representing the number of layers in the branch, ||||2 is the L2 norm, and Θ represents the set of trainable parameters.

[0076] Step III. The soft labels generated by the sub-classifiers in the teacher ensemble network are weighted and aggregated as the final result of the teacher ensemble network model. The final result of the teacher ensemble network and the final result of the student ensemble network are subjected to a second knowledge distillation to train the weights of each branch in the student ensemble network.

[0077] Step III 1. Initialize the weights of each branch in the student ensemble network model using the mean;

[0078] Step III 2. Interpolate and aggregate the soft labels of each subclass in the teacher ensemble network as the final soft labels generated by the teacher ensemble network model, and weight and aggregate the soft labels of each branch in the student ensemble network as the final soft labels generated by the student ensemble network model.

[0079] Step III.3. Based on the differences between the soft labels generated by the teacher ensemble network model and the student ensemble network model obtained in Step III.2, a second ensemble knowledge distillation is performed to train the weights of each branch in the student ensemble network model. The ensemble distillation function is expressed as follows:

[0080] L KD =L KL (S A ,T A / T)+L MSE (S A ,T A )

[0081] in and These represent the Kullback-Leibler divergence loss and the mean square error loss, respectively. T is the distillation temperature hyperparameter, which controls the soft label output by the teacher subnetwork.

[0082] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A semi-supervised software defect prediction method based on graph representation learning and knowledge distillation, characterized in that: It includes a graph representation method for source code and a method for constructing a semi-supervised software defect prediction model based on knowledge distillation. The method for constructing the semi-supervised software defect prediction model utilizes a complete graph representation of the source code to pre-train a teacher ensemble network model, and extracts knowledge from the teacher ensemble network model and injects it into the student ensemble network model through knowledge distillation technology, including the following steps: Step A. Select a preset number of source codes of various types as samples and extract abstract syntax tree structure and graph structure from the source code. Use static code analysis tools to obtain the static features of each module in the source code as the initial vector representation of the nodes in the graph structure and tree structure, and then proceed to step B. Step B. Divide the abstract syntax tree structure into statement-level trees, input them into the Word2vec model and encode them as word vectors. Use the word vectors as input to a bidirectional recurrent neural network to learn the semantic features of the source code, and then proceed to step C. Step C. Concatenate the semantic features learned in Step B with the traditional static features extracted in Step A to form a new node state vector, thus obtaining the final representation of the corresponding node of the module, and then proceed to Step D; Step D. Embed the final node representation obtained in Step C into the graph structure extracted in Step A as the initial state vector of the node; The construction method of the semi-supervised software defect prediction model based on knowledge distillation includes the following steps: Step I. Pre-train the teacher ensemble network model using a difference-based co-training algorithm with only a small number of class labels. The termination condition for pre-training is that the performance of all sub-classifiers in the teacher ensemble network model no longer improves or the labeled dataset no longer expands. Once the termination condition is met, proceed to Step II. Step II. Match the subclassifiers in the teacher ensemble network with the branches in the student ensemble network one by one to perform the first knowledge distillation to train the internal parameters of the branches in the student ensemble network, and then proceed to Step III; Step III. The soft labels generated by the sub-classifiers in the teacher ensemble network are weighted and aggregated as the final result of the teacher ensemble network model. The final result of the teacher ensemble network and the final result of the student ensemble network are then subjected to a second knowledge distillation to train the weights of each branch in the student ensemble network.

2. The semi-supervised software defect prediction method based on graph representation learning and knowledge distillation according to claim 1, characterized in that: Step A includes the following sub-steps: Step A1. Use the javalang tool to parse the program source code into an abstract syntax tree structure and extract function nodes from it; Step A2. Use the third-party tool understand to construct an adjacency matrix based on the control flow graph of the source code to represent the dependencies between modules and obtain the graph structure of the source code; Step A3. Use static code analysis tools to extract the static features of the modules corresponding to the nodes in the graph structure and abstract syntax tree structure, and initialize the node state vectors.

3. The semi-supervised software defect prediction method based on graph representation learning and knowledge distillation according to claim 1, characterized in that: Step B includes the following sub-steps: Step B1. Traverse the abstract syntax tree structure obtained in step A1, map all user-defined names to fixed names, and transform the AST into a node sequence through pre-sorted traversal; Step B2. Map the sentence-level trees obtained in Step B1 to their word vector sequence representations using the word2vec model so that they can be directly trained by the neural network; Step B3. Input the word vector sequence obtained in step B2 into a bidirectional recurrent neural network for training, and finally obtain a vector representation of the semantic features of the source code.

4. The semi-supervised software defect prediction method based on graph representation learning and knowledge distillation according to claim 1, characterized in that: Step C includes the following sub-steps: Step C1. Concatenate the traditional static features obtained in step A3 with the semantic features of a specified dimension learned through the bidirectional recurrent neural network in step B3 to form a new vector representation; Step C2. Apply SPSS software to standardize the vector representation obtained in step C1. Next, determine the correlation between features, then set the number of principal components and perform principal component analysis according to the principal component expression. Finally, obtain the dimensionality-reduced feature vector as the final representation of the node.

5. The semi-supervised software defect prediction method based on graph representation learning and knowledge distillation according to claim 3, characterized in that: The word2vec model in step B2 is a lightweight neural network that consists of only an input layer, a hidden layer, and an output layer. This method transfers the word vector representation method in natural language processing to the representation learning of source code. Its objective optimization function is as follows: The output layer is normalized by softmax, and u represents the original result of the output layer.

6. The semi-supervised software defect prediction method based on graph representation learning and knowledge distillation according to claim 1, characterized in that, Step I includes the following sub-steps: Step I 1. Divide the graph nodes into two parts: a labeled dataset and an unlabeled dataset; Step 1.

2. Train the teacher ensemble network using the labeled dataset and use the trained teacher ensemble network to label the nodes in the unlabeled dataset with pseudo-labels; Step I 3. Calculate the confidence of nodes with pseudo-labels. Add nodes with confidence scores higher than a preset threshold to the labeled dataset, and use their corresponding pseudo-labels as real labels in the next training. Nodes with confidence scores lower than the preset threshold are kept in the unlabeled dataset. Step I4. Repeat Step I2 and Step I3 until the performance of all subclassifiers in the teacher ensemble network no longer improves or the labeled dataset no longer expands.

7. The semi-supervised software defect prediction method based on graph representation learning and knowledge distillation according to claim 1, characterized in that: In the knowledge distillation framework, it is necessary to ensure that the number of sub-classifiers in the teacher ensemble network is consistent with the number of branches in the student ensemble network. Step II includes the following sub-steps: Step II 1. Match the subclassifiers in the pre-trained teacher ensemble network with the branches in the student ensemble network one by one; Step II.

2. Perform the first distillation using the differences between the soft labels generated by the subclassifiers in the pre-trained teacher ensemble network and the soft labels generated by each branch in the student ensemble network z. The distillation function is as follows: Where K is a hyperparameter representing the number of layers in the branch, ||||2 is the L2 norm, and Θ represents the set of trainable parameters.

8. The semi-supervised software defect prediction method based on graph representation learning and knowledge distillation according to claim 1, characterized in that: The specific sub-steps of step III are as follows: Step III 1. Initialize the weights of each branch in the student ensemble network model using the mean; Step III 2. Interpolate and aggregate the soft labels of each subclass in the teacher ensemble network as the final soft labels generated by the teacher ensemble network model, and weight and aggregate the soft labels of each branch in the student ensemble network as the final soft labels generated by the student ensemble network model. Step III.

3. Based on the differences between the soft labels generated by the teacher ensemble network model and the student ensemble network model obtained in Step III.2, a second ensemble knowledge distillation is performed to train the weights of each branch in the student ensemble network model. The ensemble distillation function is expressed as follows: L KD =L KL (S A ,T A / T)+L MSE (S A ,T A ) in and These represent the Kullback-Leibler divergence loss and the mean square error loss, respectively. T is the distillation temperature hyperparameter, which controls the soft label output by the teacher subnetwork.

Images