Edge device-oriented knowledge distillation method and device

Abstract

The application provides an edge device knowledge distillation method and device. A student model adopts a multi-branch multi-head design, a plurality of teacher models with different structures are selected for knowledge distillation, a dynamic jump-out module is added to each layer of the multi-branch multi-head student model after knowledge distillation, adaptive training and reasoning of the student model are realized, parameters of the trained multi-branch multi-head student model are frozen, multi-stage optimization training of the dynamic jump-out module is performed, and the trained and optimized multi-branch multi-head student model is re-parameterized to obtain a single-branch single-head target student model, thereby enhancing the learning effect of the student model, speeding up the reasoning speed, improving the memory utilization rate of the edge device, and realizing adaptive reasoning of input.

Description

Technical Field

[0001] This application relates to the field of neural network prediction technology, and in particular to a knowledge distillation method and apparatus for edge devices. Background Technology

[0002] With the continuous advancement of industrial technology, large-scale neural networks have achieved satisfactory results in prediction tasks for large edge devices. However, as large edge devices in many real-world industrial scenarios become increasingly intelligent, their sensor data exhibits diverse characteristics. The limited computing power and memory of industrial edge devices make it difficult to deploy complex algorithms on embedded terminals or edge devices. To address this, the industry seeks algorithms with low computational complexity and high prediction accuracy for edge device prediction.

[0003] In existing technologies, knowledge distillation can transfer knowledge from large, deep teacher models to small, shallow student models. Through knowledge distillation, prediction tasks for edge devices can be completed, thereby achieving the goal of low complexity.

[0004] However, the above solutions may result in poor learning outcomes for students under different operating conditions or different degradation processes under the same operating conditions, as well as slower inference speed and low memory utilization. Summary of the Invention

[0005] This application provides a knowledge distillation method and apparatus for edge devices, which addresses the problem that industrial edge devices have limited computing power and memory, making it difficult to deploy complex algorithms on embedded terminals or edge devices.

[0006] In a first aspect, this application provides a knowledge distillation method for edge devices, comprising:

[0007] Based on the configuration information of the models stored in the model repository, multiple selected teacher models with different structures are trained to obtain multiple trained teacher models.

[0008] Obtain the predicted hidden layer and predicted label information of multiple trained teacher models, and perform whitening and noise reduction processing on the hidden layer of each trained teacher model to obtain the processed hidden layer of each trained teacher model.

[0009] A multi-branch, multi-head student model is generated by stacking multiple basic modules. Each basic module includes multiple one-dimensional convolutional branches of different sizes and batch normalization layers.

[0010] Based on the hidden layer processed by each trained teacher model, the predicted label information, and the multi-branch multi-head student model, multi-teacher knowledge distillation is performed to obtain the first-stage student model.

[0011] A dynamic exit module is added to each layer of the first-stage student model. The confidence of each layer is calculated according to the pre-designed confidence function. The parameters of the first-stage student model are frozen to obtain the second-stage student model.

[0012] Using the confidence level of each layer as a label, the dynamic exit module of the second-stage student model is trained for at least one stage to obtain the third-stage student model.

[0013] The third-stage student model is reparameterized to obtain a single-branch, single-head target student model.

[0014] In conjunction with the first aspect, in some embodiments, the step of performing multi-teacher knowledge distillation based on the hidden layers processed by each trained teacher model, the predicted label information, and the multi-branch multi-head student model to obtain the first-stage student model includes:

[0015] Based on the output of the last convolutional layer of the multi-branch multi-head student model, the output of the last convolutional layer is flattened and expanded to different knowledge distillation heads to obtain multiple processed distillation heads.

[0016] The batch normalization layer normalizes the processed multiple distillation heads to obtain the prediction output of the multi-branch multi-head student model.

[0017] The knowledge distillation loss is calculated by distilling the processed multiple distillation heads using the output of the penultimate hidden layer of the multiple trained teacher models.

[0018] Based on the predicted label information and the real labels of the multiple trained teacher models, a mixed weight is set to obtain a mixed label;

[0019] By distilling the predicted output of the multi-branch multi-head student model using the mixed labels, the mean squared error loss is calculated.

[0020] The target loss function is obtained by adding the knowledge distillation loss and the mean square error loss.

[0021] Based on the target loss function, the model parameters of the multi-branch, multi-head student model are optimized using a regularized backpropagation algorithm to obtain the first-stage student model.

[0022] In conjunction with the first aspect, in some embodiments, adding a dynamic exit module to each layer of the first-stage student model to freeze the parameters of the first-stage student model to obtain the second-stage student model includes:

[0023] The dynamic exit module is added between each layer of the first-stage student model to obtain the processed first-stage student model, wherein the dynamic exit module includes an adapter and a shared classifier;

[0024] By limiting the parameters of a pre-obtained traditional confidence function, a new confidence function is obtained.

[0025] Based on the confidence function, the confidence of the first layer of the processed first-stage student model is calculated, and it is determined whether the confidence of the first layer meets the preset exit rule.

[0026] If the confidence level of the first layer satisfies the exit rule, the calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained.

[0027] If the confidence level of the first layer does not meet the exit rule, the confidence level of the next layer is calculated, and it is determined whether the confidence level of the next layer meets the exit rule, until the confidence level of the next layer meets the preset exit rule. The calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained.

[0028] In conjunction with the first aspect, in some embodiments, the reparameterization of the third-stage student model to obtain a single-branch, single-head target student model includes:

[0029] Using neural network degradation theory, the multi-head outputs of the third-stage student model are implicitly integrated, and each convolution kernel and its corresponding batch normalization layer in the third-stage student model are reconstructed and fused through the principle of convolution additivity to obtain the single-branch single-head target student model.

[0030] In conjunction with the first aspect, in some embodiments, the method further includes:

[0031] The target student model is deployed in industrial edge devices using a pre-configured deployment algorithm.

[0032] Secondly, this application provides a knowledge distillation apparatus for edge devices, comprising:

[0033] The first training module is used to train multiple selected teacher models with different structures according to the configuration information of the models stored in the model repository, so as to obtain multiple trained teacher models.

[0034] The information acquisition module is used to acquire the predicted hidden layer and predicted label information of multiple trained teacher models;

[0035] The first processing module is used to perform whitening and noise reduction processing on the hidden layer of each trained teacher model to obtain the processed hidden layer of each trained teacher model.

[0036] The second processing module is used to perform multi-layer stacking processing based on multiple basic modules to generate a multi-branch multi-head student model. Each basic module includes multiple one-dimensional convolutional branches of different sizes and batch normalization layers.

[0037] The distillation module is used to perform multi-teacher knowledge distillation based on the hidden layer processed by each trained teacher model, the predicted label information, and the multi-branch multi-head student model to obtain the first-stage student model.

[0038] The third processing module is used to add a dynamic exit module to each layer of the first-stage student model, calculate the confidence of each layer according to the pre-designed confidence function, freeze the parameters of the first-stage student model, and obtain the second-stage student model.

[0039] The second training module is used to train the dynamic exit module of the second-stage student model for at least one stage, using the confidence level of each layer as a label, to obtain the third-stage student model.

[0040] The reparameterization module is used to reparameterize the student model in the third stage to obtain a single-branch, single-head target student model.

[0041] In conjunction with the second aspect, in some embodiments, the distillation module includes:

[0042] The first processing unit is used to flatten and expand the output of the last convolutional layer of the multi-branch multi-head student model to different knowledge distillation heads, so as to obtain multiple distillation heads after processing.

[0043] The normalization processing unit is used to normalize the processed multiple distillation heads through the batch normalization layer to obtain the prediction output of the multi-branch multi-head student model.

[0044] The first distillation unit is used to distill the processed multiple distillation heads through the output of the penultimate hidden layer of the multiple trained teacher models, and calculate the knowledge distillation loss.

[0045] The second processing unit is used to set the mixing weights and obtain the mixed labels based on the predicted label information and the real labels of the multiple trained teacher models;

[0046] The second distillation unit is used to distill the prediction output of the multi-branch multi-head student model through the mixed labels and calculate the mean squared error loss.

[0047] The third processing unit is used to obtain the target loss function by adding the knowledge distillation loss and the mean squared error loss; the optimization unit optimizes the model parameters of the multi-branch multi-head student model according to the target loss function using a regularized backpropagation algorithm to obtain the first-stage student model.

[0048] In conjunction with the second aspect, in some embodiments, the third processing module includes:

[0049] The first processing unit is used to add the dynamic jump-out module between each layer of the first-stage student model to obtain the processed first-stage student model.

[0050] The parameter limiting unit is used to limit the parameters of the pre-obtained traditional confidence function to obtain the confidence function.

[0051] The second processing unit is used to calculate the confidence level of the first layer of the processed first-stage student model according to the confidence level function, and to determine whether the confidence level of the first layer meets the preset exit rule.

[0052] The third processing unit is used to end the calculation if the confidence level of the first layer meets the exit rule, freeze the parameters of the first-stage student model, and obtain the second-stage student model.

[0053] The fourth processing unit is used to continue calculating the confidence of the next layer if the confidence of the first layer does not meet the exit rule, and to determine whether the confidence of the next layer meets the exit rule, until the confidence of the next layer meets the preset exit rule, then end the calculation, freeze the parameters of the first stage student model, and obtain the second stage student model.

[0054] Thirdly, embodiments of this application may also provide an electronic device, including: a processor and a memory;

[0055] The memory is used to store computer instructions;

[0056] The processor is configured to execute computer instructions stored in the memory, causing the electronic device to perform the knowledge distillation method for edge devices as described in the first aspect.

[0057] Fourthly, embodiments of this application may also provide a storage medium storing a computer program;

[0058] When the computer program is executed, it implements the knowledge distillation method for edge devices as described in the first aspect.

[0059] The knowledge distillation method and apparatus for edge devices provided in this application differ from existing technologies in that, for time series prediction tasks of edge devices with diverse data characteristics, different operating conditions, or different degradation processes under the same operating condition, this scheme uses teacher models with different structures for training and designs a multi-branch, multi-head student model. Through knowledge distillation using multiple teachers with different structures, and by reparameterizing and adaptively training the knowledge-distilled multi-branch, multi-head student model, prediction of edge devices is achieved. This effectively solves the problems of poor learning performance, slow inference speed, and low memory utilization of student networks. Attached Figure Description

[0060] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0061] Figure 1 An application scenario diagram of the knowledge distillation method for edge devices provided in the embodiments of this application;

[0062] Figure 2 A schematic flowchart of an embodiment of the knowledge distillation method for edge devices provided in this application;

[0063] Figure 3 A schematic flowchart of Embodiment 2 of the knowledge distillation method for edge devices provided in this application;

[0064] Figure 4 A flowchart illustrating Embodiment 3 of the knowledge distillation method for edge devices provided in this application;

[0065] Figure 5 A schematic diagram of a knowledge distillation architecture for edge devices provided in an embodiment of this application;

[0066] Figure 6 A schematic diagram of a second knowledge distillation architecture for edge devices provided in this application embodiment;

[0067] Figure 7 This is a schematic diagram illustrating a specific example architecture of the knowledge distillation method for edge devices provided in this application.

[0068] Figure 8 A flowchart illustrating a specific example of the edge device knowledge distillation method provided in this application embodiment;

[0069] Figure 9 This is a schematic diagram of the structure of an embodiment of the knowledge distillation apparatus for edge devices provided in this application.

[0070] Figure 10This is a schematic diagram of the structure of Embodiment 2 of the knowledge distillation apparatus for edge devices provided in this application.

[0071] Figure 11 This is a schematic diagram of the structure of Embodiment 3 of the knowledge distillation apparatus for edge devices provided in this application.

[0072] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0073] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] In this era of rapid advancements in intelligent technology, many large industrial edge devices are becoming increasingly intelligent. Large neural networks have achieved significant results in prediction tasks for these devices. However, due to the limited computing power and memory of industrial edge devices, it is difficult to deploy complex algorithms on embedded terminals or edge devices. Knowledge distillation, a method that transfers knowledge from large, deep teacher models to small, shallow student models, is an ideal approach for industry. However, different neural networks are better suited to specific feature extraction tasks, and the sensor data collected by various large industrial edge devices often has diverse characteristics. If multi-teacher knowledge distillation is used, the student network may perform poorly when the size of the student model differs significantly from that of the teacher model. If the student model uses a multi-branch design, inference speed will slow down and memory utilization will decrease. Furthermore, for time series prediction of edge devices under different operating conditions or different degradation processes under the same operating condition, traditional knowledge distillation methods apply the same computational cost to each sample, failing to provide adaptive inference to the input.

[0076] To address the aforementioned issues, this application provides a knowledge distillation method for edge devices, achieving lightweight and adaptive prediction for time series forecasting tasks on edge devices under different operating conditions or different degradation processes under the same operating condition. Specifically, current time series forecasting tasks for large edge devices are implemented using large neural networks. During the research process, the inventors discovered that when neural network algorithms are particularly complex, it is difficult to deploy them on embedded terminals or edge devices. While knowledge distillation can simplify the teacher model, the simplified student model exhibits poor learning performance and cannot perform adaptive inference based on input. Furthermore, if the student model employs a multi-branch design, it may slow down inference speed and reduce memory utilization. Considering these issues, the inventors investigated whether knowledge distillation can be achieved using multi-teacher models and multi-branch, multi-head student models with different structures. The distilled student model is then subjected to adaptive and reparameterized training to achieve the goal of simplifying the student model without compromising learning performance. Based on this, the technical solution of this application is proposed.

[0077] Figure 1 This diagram illustrates an application scenario for the knowledge distillation method for edge devices provided in this embodiment of the application. For example... Figure 1 As shown, the knowledge distillation method for edge devices provided in this application can be applied to real-world industrial scenarios, which at least include edge devices in actual factories and their data. Edge devices can be a series of devices requiring prediction, such as machinery. The data for edge devices can be industrial time-series data such as edge device remaining life prediction, mining and refining product quality inspection, and engine fault detection. This solution does not limit the types of edge devices and their data.

[0078] This application does not impose any restrictions on the specific form of each device.

[0079] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0080] Figure 2 A schematic flowchart of an embodiment of the knowledge distillation method for edge devices provided in this application is shown below. Figure 2 As shown, the specific steps include:

[0081] S101: Based on the configuration information of the models stored in the model repository, train the selected teacher models with different structures respectively to obtain multiple trained teacher models.

[0082] In this step, multiple teacher models with different structures are selected, and then the configuration information of the models stored in the model repository of industrial edge devices is obtained. Based on this configuration information, the selected teacher models with different structures are trained respectively to obtain multiple trained teacher models.

[0083] In one specific implementation, multiple large-scale models with different structures and good prediction performance are selected as teacher models. Model configuration information is obtained from an existing edge device model repository. This configuration information includes industrial time-series data such as edge device remaining life prediction, mining and refining industrial product quality prediction, and engine fault detection. Based on this configuration information, the selected teacher models with different structures are trained separately. During training, no restrictions are placed on the architecture, number of parameters, or whether these teacher models are integrated, resulting in multiple trained teacher models.

[0084] S102: Obtain the predicted hidden layer and predicted label information of each trained teacher model, and perform whitening and noise reduction processing on the hidden layer of each trained teacher model to obtain the processed hidden layer of each trained teacher model.

[0085] In this step, during the training of multiple teacher models with different structures, the predicted hidden layer and predicted label information of each trained teacher model are calculated and saved. By performing whitening and noise reduction processing on the hidden layer of each trained teacher model, the processed hidden layer of each trained teacher model is obtained.

[0086] In one specific implementation, based on multiple trained teacher models, the predicted hidden layer and predicted label information of each trained teacher model are calculated and saved. Then, a common domain-specific whitening and denoising operation is performed on the hidden layer of each trained teacher model to remove redundant information. The specific whitening and denoising operation method is not limited here, and finally, the processed hidden layer of each trained teacher model is obtained.

[0087] S103: Perform multi-layer stacking processing based on multiple basic modules to generate a multi-branch, multi-head student model, and initialize the model parameters of the multi-branch, multi-head student model.

[0088] In this step, multiple one-dimensional convolutional branches of different sizes are set in each basic module. After each convolutional layer, there is a batch normalization layer. The basic modules are stacked in multiple layers to generate a multi-branch multi-head student model, and the model parameters of the multi-branch multi-head student model are initialized.

[0089] In one specific implementation, multiple one-dimensional convolutional branches of different sizes are defined in each basic module, wherein a one-dimensional convolutional operation is defined, including:

[0090] Assumption If it's the input of channel C, then there are T elements of size T. The convolution kernel can transform the input into Where O is a feature map with resolution T and containing D channels. Finally, the operation of the j-th convolutional kernel can be defined as follows:

[0091]

[0092] Among them, M :,k and F :,k (j) These are the k-th channel inputs of M and F, respectively. In addition, F... :,k (j) This indicates the j-th convolutional kernel among all T convolutional kernels.

[0093] Each convolutional layer is followed by a batch normalization layer, which also provides a regularization effect. Equation (1) is then transformed into the following formula:

[0094]

[0095] Where μ j and δ j It is the channel mean and standard deviation of the input feature map, γ j and β j These are learnable scaling factors and bias terms. Finally, a multi-branch, multi-head student model is generated, and its model parameters are initialized.

[0096] S104: Based on the hidden layer processed by each trained teacher model, the predicted label information, and the multi-branch, multi-head student model, perform multi-teacher knowledge distillation to obtain the first-stage student model.

[0097] In this step, based on the hidden layer and predicted label information of each trained teacher model and the multi-branch multi-head student model, multi-teacher knowledge distillation is performed to obtain the first-stage student model.

[0098] In one specific implementation, the output of the final convolutional layer of the multi-branch, multi-head student model is obtained. This output is flattened and expanded to different distillation heads to obtain multiple processed distillation heads. Then, the processed multiple distillation heads are normalized through a batch normalization layer to obtain the predicted output of the multi-branch, multi-head student model. Next, the processed multiple distillation heads are distilled using the output of the penultimate hidden layer of multiple trained teacher models, and the knowledge distillation loss is calculated. Based on the predicted label information and the true labels of multiple trained teacher models, mixing weights are set to obtain mixed labels. Then, the predicted output of the multi-branch, multi-head student model is distilled using the mixed labels to calculate the mean squared error loss. Finally, the knowledge distillation loss and the mean squared error loss are added to obtain the target loss function. Based on the target loss function, the model parameters of the multi-branch, multi-head student model are optimized using a regularized backpropagation algorithm to obtain the first-stage student model.

[0099] S105: Add a dynamic exit module to each layer of the first-stage student model, calculate the confidence level of each layer according to the pre-designed confidence function, freeze the parameters of the first-stage student model, and obtain the second-stage student model.

[0100] In this step, based on the first-stage student model, a dynamic exit module is added to each layer of the first-stage student model. The first-stage student model is trained through the dynamic exit module. After training is completed, the parameters of the first-stage student model are frozen to obtain the second-stage student model.

[0101] In one specific implementation, based on the first-stage student model, a dynamic exit module is added to each layer of the first-stage student model. This dynamic exit module includes an adapter and a shared classifier. The configuration information of the models stored in the model repository is input into the first-stage student model to train it. First, the parameters of the traditional confidence function are constrained to obtain the confidence function. Then, the adapter calculates the confidence score of the first layer of the processed first-stage student model based on the confidence function and transmits the confidence score to the shared classifier. Based on the confidence score, the shared classifier calculates the exit probability of that layer. The shared classifier then determines whether the calculated exit probability of that layer meets a preset exit rule, which includes:

[0102] First, assume the predetermined dynamic exit threshold for the i-th layer is thresi. Note that the threshold may be different for each layer, and the threshold is obtained by a genetic algorithm that balances the accuracy and speed of the model prediction within a given validation set. Let pi be the probability of exiting the current layer for each layer. Then, as long as pi > thresi, the user can exit the layer.

[0103] If the exit probability of the first layer meets the exit rule, the calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained.

[0104] If the exit probability of the first layer does not meet the exit rule, the exit probability of the next layer is calculated and it is determined whether the exit probability of the next layer meets the exit rule, until the exit probability of the layer meets the preset exit rule, the calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained.

[0105] S106: Using the confidence level of each layer as a label, train the dynamic exit module of the second-stage student model for at least one stage to obtain the third-stage student model.

[0106] In this step, using all the calculated confidence scores as labels, the dynamic jump module of the second-stage student model is trained for at least one stage to obtain the third-stage student model.

[0107] In one specific implementation, the process of training the first-stage student model is repeated using all calculated confidence scores as labels. The specific process is similar to that in S105 and will not be repeated here. By modifying the parameters in the confidence function, the dynamic exit module is trained for at least one stage to obtain the third-stage student model.

[0108] S107: Perform reparameterization on the third-stage student model to obtain a single-branch, single-head target student model.

[0109] In this step, the third-stage student model is a multi-branch, multi-head student model, which is relatively complex. Therefore, the third-stage student model is reparameterized to obtain a single-head, single-branch target student model, thereby simplifying and streamlining the third-stage student model.

[0110] In one specific implementation, the output of the last convolutional layer of the multi-head, multi-branch student model is flattened, and the hidden layers are expanded to form multiple distillation heads. These multiple distillation heads are used to extract hidden features from multiple trained teacher models. Simultaneously, these distillation heads can be used to form the model's prediction head through simple linear projection. After obtaining the third-stage student model, the third-stage student model aggregates information from multiple trained teacher models. Utilizing neural network degradation theory, the multi-head outputs of the third-stage student model are implicitly integrated, and the multi-head projection is represented as:

[0111]

[0112]

[0113] y = ReLU(flatten(O) last W r +b r (3)

[0114] Among them, W r and b r These are the weights and biases of the fully connected layer after parameter rebalancing, respectively, and n is the number of teacher models.

[0115] By leveraging the principle of convolution additivity, each convolutional kernel and its corresponding normalization layer in the third-stage student model are reconstructed and fused to obtain a single-branch, single-head target student model.

[0116] The knowledge distillation method for edge devices provided in this application also includes deploying a single-branch, single-head target student model in industrial edge devices using a pre-configured deployment algorithm.

[0117] This embodiment provides a knowledge distillation method for edge devices. It designs a multi-branch, multi-head student model, performs knowledge distillation on this model by selecting multiple teacher models with different structures, and adds a dynamic exit module to the distilled model. Then, it designs a confidence function, and by modifying the parameters of the confidence function, achieves multi-stage training of the dynamic exit module, making the multi-branch, multi-head student model adaptive. By reparameterizing the adaptive student model, a single-branch, single-head target student model is finally obtained. This method simplifies the multi-branch, multi-head student model into a single-branch, single-head target student model, accelerating inference speed and increasing memory utilization on edge devices.

[0118] Figure 3 A schematic flowchart of Embodiment 2 of the knowledge distillation method for edge devices provided in this application is shown below. Figure 3 As shown, based on the above embodiments, in the specific implementation of this knowledge distillation method for edge devices, step S104, which involves performing multi-teacher knowledge distillation based on the hidden layer of each trained teacher model, the predicted label information, and the multi-branch, multi-head student model to obtain the first-stage student model, further includes the following steps:

[0119] S1041: Based on the output of the last convolutional layer of the multi-branch multi-head student model, flatten and expand the output of the last convolutional layer to different knowledge distillation heads to obtain multiple processed distillation heads.

[0120] In this step, the output of the final convolutional layer of the multi-branch, multi-head student model is obtained. Then, the output of the final convolutional layer is flattened and expanded to different knowledge distillation heads, thus obtaining multiple processed distillation heads.

[0121] In one specific implementation, the output of the convolutional layer of the multi-branch, multi-head student model is represented as:

[0122]

[0123] Where i represents the i-th layer, l represents the l-th layer, BatchNorm is batch normalization, and ReLU is the modified linear unit activation function.

[0124] The output O of the final convolutional layer of the multi-branch, multi-head student model is obtained. last Then the output of the final convolutional layer is flattened and expanded to different knowledge distillation heads.

[0125]

[0126] Where i represents the i-th, KDHead i For the output of the i-th knowledge distillation head in the multi-head student model, O last It is the output of the last convolutional layer of the student model. "flatten" refers to a one-dimensional flattening operation. W i b is the weight matrix of the fully connected layer for the i-th distillation head. i It is the fully connected layer bias term for the i-th distillation head, μ i This refers to the mean of the normalization layer of the i-th distillation head batch, δ. i γ is the variance of the normalization layer of the i-th distillation head batch. i β is the learnable variance of the normalization layer of the i-th distillation head batch. i It is the learnable mean of the normalization layer of the i-th distillation head batch.

[0127] S1042: The batch normalization layer normalizes the processed multiple distillation heads to obtain the prediction output of the multi-branch multi-head student model.

[0128] In this step, the batch normalization layer performs batch normalization on the processed multiple distillation heads to obtain the prediction output of the multi-branch multi-head student model.

[0129] In one specific implementation, the processed multiple distillation heads are batch normalized through a batch normalization layer to obtain the prediction output of the multi-branch multi-head student model. The prediction output formula of the multi-branch multi-head student model is expressed as:

[0130]

[0131] Among them, W i ′ is the linear mapping matrix of the i-th knowledge distillation head, b i ′ is the bias term of the i-th knowledge distillation head. ReLU is a common nonlinear activation function.

[0132] S1043: The knowledge distillation loss is calculated by using the output of the penultimate hidden layer of multiple trained teacher models and multiple distillation heads after distillation processing.

[0133] In this step, multiple distillation heads are obtained after processing. The knowledge distillation loss is calculated by distilling the processed distillation heads through the output of the penultimate hidden layer of multiple trained teacher models.

[0134] In one specific implementation, multiple different knowledge distillation heads (KDHead) are obtained. i Where i represents the i-th. Multiple different knowledge distillation heads (KDHeads) are obtained by distilling the outputs of the penultimate hidden layers of multiple trained teacher models. i Thus, the knowledge distillation loss is calculated, and the knowledge distillation loss is expressed as:

[0135]

[0136] in, KDHead represents the training dataset. i S and KDHead i T dim represents the features of the student and teacher models, respectively. i It is the distilled feature dimension of multiple trained teacher models.

[0137] S1044: Based on the predicted label information and the real labels of multiple trained teacher models, set the mixed weights to obtain the mixed labels.

[0138] In this step, in order to guide the multi-branch, multi-head student model to mimic the predictions of multiple trained teacher models, a mixed weight is set based on the predicted label information and the real labels of multiple trained teacher models to obtain a mixed label.

[0139] In one specific implementation, in order to enable the multi-branch, multi-head student model to better mimic the predictions of multiple trained teacher models, a mixed weight is set based on the predicted label information and the real labels of multiple trained teacher models to obtain a mixed label.

[0140] S1045: By mixing labels, the predicted output of the multi-branch multi-head student model is distilled, and the mean squared error loss is calculated.

[0141] In this step, a mixed label is obtained. The predicted output of the multi-branch multi-head student model is distilled using the mixed label to obtain the mean squared error loss.

[0142] In one specific implementation, the predicted output of the multi-branch, multi-head student model is distilled by mixing labels to obtain the mean squared error loss, which is expressed as:

[0143]

[0144] in, and Let represent the output of the i-th student model and the output of the teacher model, respectively, and let λ represent the weights of the mixed teacher model's prediction and the true label.

[0145] S1046: The target loss function is obtained by adding the knowledge distillation loss and the mean squared error loss.

[0146] In this step, the knowledge distillation loss and the mean squared error loss are added together to form the vehicle target loss function.

[0147] In one specific implementation, the knowledge distillation loss and the mean squared error loss are added together to obtain the target loss function, which is expressed as:

[0148]

[0149] in, This represents the knowledge distillation loss. denoted as mean squared error loss, and μ represents the contribution of distillation loss to the target loss.

[0150] S1047: Based on the objective loss function, the model parameters of the multi-branch, multi-head student model are optimized using a regularized backpropagation algorithm to obtain the first-stage student model.

[0151] In this step, the model parameters of the multi-branch, multi-head student model are optimized using a regularized backpropagation algorithm based on the objective loss function, thereby obtaining the first-stage student model.

[0152] In one specific implementation, the regularized backpropagation algorithm optimizes the model parameters of the multi-branch, multi-head student model based on the target loss function, thereby obtaining the first-stage student model.

[0153] The knowledge distillation method for edge devices provided in this embodiment adopts a multi-branch, multi-head design for the student model and performs knowledge distillation with multi-teacher models of different structures. This solves the problem that neural networks are only suitable for specific feature extraction tasks, while also enhancing the learning effect of the student model.

[0154] Figure 4 A schematic flowchart of Embodiment 3 of the knowledge distillation method for edge devices provided in this application is shown below. Figure 4As shown, based on the above embodiment one, in the specific implementation of this knowledge distillation method for edge devices, step S105 adds a dynamic exit module to each layer of the first-stage student model, calculates the confidence of each layer according to the pre-designed confidence function, freezes the parameters of the first-stage student model, and obtains the second-stage student model. The specific implementation also includes the following steps:

[0155] S1051: Add a dynamic exit module between each layer of the first-stage student model to obtain the processed first-stage student model.

[0156] In this step, the first-stage student model is composed of multiple layers of basic modules stacked together. Dynamic jump-out modules are added between the layers to obtain the processed first-stage student model.

[0157] In one specific implementation, dynamic exit modules are added between the layers of the first-stage student model, which is composed of multiple stacked basic modules. These dynamic exit modules include adapters and shared classifiers. The resulting first-stage student model is then processed.

[0158] S1052: The pre-obtained traditional confidence function is parameterized to obtain the confidence function.

[0159] In this step, the confidence level is calculated by dynamically exiting the module, and the parameters of the pre-obtained traditional confidence level function are limited to obtain the confidence level function.

[0160] In one specific implementation, the pre-acquired traditional confidence function is expressed as follows: Where tanh represents the hyperbolic tangent function, p i The probability is usually close to zero because industrial model prediction is more difficult compared to other prediction tasks. This is because the prediction error varies greatly and tanh is prone to saturation. Therefore, to avoid probability saturation, the parameters of the pre-obtained traditional confidence function are constrained, and the confidence function is set as follows:

[0161]

[0162] Here, β is used to scale the error, α is used to prevent the probability from being too large or too small, and γ is a label smoothing parameter used to smooth the labels, improve the model's predictive performance, and prevent overfitting.

[0163] S1053: Calculate the confidence level of the first layer of the processed first-stage student model based on the confidence function, and determine whether the confidence level of the first layer meets the preset exit rule.

[0164] In this step, the pre-obtained traditional confidence function parameters are constrained to obtain the confidence function. Based on the confidence function, the confidence of the first layer of the processed first-stage student model is calculated. The shared classifier calculates the exit probability of the first layer based on the confidence and determines whether the exit probability of the first layer meets the preset exit rules.

[0165] In one specific implementation, the confidence level of the first layer of the processed first-stage student model is calculated based on the confidence function. The shared classifier calculates the exit probability of the first layer based on the confidence level and determines whether the exit probability of the first layer meets the preset exit rules. The preset exit rules include:

[0166] First, assume the predetermined dynamic exit threshold for the i-th layer is thresi. Note that the threshold may be different for each layer, and the threshold is obtained by a genetic algorithm that balances the accuracy and speed of the model prediction within a given validation set. Let pi be the probability of exiting the current layer for each layer. Then, as long as pi > thresi, the user can exit the layer.

[0167] If the bounce probability of the first layer meets the preset bounce rule, the calculation ends. If the bounce probability of the first layer does not meet the preset bounce rule, the calculation continues to calculate the confidence level and bounce probability of the next layer until the bounce probability meets the preset bounce rule, at which point the calculation ends.

[0168] S1054: If the confidence level of the first layer meets the preset exit rule, the calculation ends and the second-stage student model is obtained.

[0169] In this step, the confidence of the first layer of the processed first-stage student model is calculated based on the confidence function. The shared classifier calculates the exit probability of the first layer based on the confidence. If the exit probability satisfies the preset exit rule, the calculation ends and the second-stage student model is obtained.

[0170] In one specific implementation, the confidence level of the first layer of the processed first-stage student model is calculated based on the confidence function, and the confidence level is transmitted to the shared classifier in the dynamic exit module. The shared classifier calculates the exit probability based on the confidence level. If the exit probability meets the preset exit rule, the calculation ends and the second-stage student model is obtained.

[0171] S1055: If the confidence level of the first layer does not meet the exit rule, continue to calculate the confidence level and exit probability of the next layer, and determine whether the exit probability of the next layer meets the preset exit rule, until the exit probability meets the preset exit rule of the layer, then end the calculation and obtain the second-stage student model.

[0172] In this step, the confidence level of the first layer of the processed first-stage student model is calculated based on the confidence function. If the bounce probability of the first layer does not meet the bounce rule, the confidence level and bounce probability of the second layer are calculated until the calculated bounce probability meets the preset bounce rule of the layer. The calculation ends and the second-stage student model is obtained.

[0173] In one specific implementation, the confidence level of the first layer of the processed first-stage student model is calculated based on the confidence function, and the confidence level is transmitted to the shared classifier in the dynamic exit module. The shared classifier calculates the exit probability based on the confidence level. If the exit probability does not meet the preset exit rule, the confidence level of the second layer is calculated, and the exit probability is obtained based on the confidence level of the second layer. The calculation continues to determine whether the exit probability obtained from the second layer meets the preset exit rule. If it does, the calculation ends. If it does not, the calculation continues to the next layer until the exit probability obtained based on the calculated confidence level meets the preset exit rule. The calculation ends, and the second-stage student model is obtained.

[0174] The knowledge distillation method for edge devices provided in this embodiment adds a dynamic exit module to the first-stage student model, enabling adaptive training of the first-stage student model. By freezing the model parameters of the first-stage student model, the dynamic exit model is optimized and trained in multiple stages, resulting in a more adaptive second-stage student model.

[0175] Figure 5 This is a schematic diagram of a knowledge distillation architecture for edge devices provided in an embodiment of this application, such as... Figure 5 As shown, the knowledge distillation architecture for edge devices mainly includes three modules: a multi-teacher model knowledge distillation module, a dynamic exit module, and a reparameterization module.

[0176] The multi-teacher model knowledge distillation module is used to output the first-stage student model. Specifically, several large models with different structures and good prediction performance are selected as teacher models. The configuration information of the models is obtained from the existing edge device model repository. The model configuration information includes industrial time-series data such as edge device remaining life prediction, mining and refining industrial product quality prediction, and engine fault detection. Based on these industrial time-series data, the selected teacher models with different structures are trained separately. The student model adopts a multi-branch, multi-head design. Based on the trained teacher models, knowledge distillation is performed on the multi-branch, multi-head student model to obtain the target loss, which includes knowledge distillation loss and mean squared error loss. The target loss is optimized through a regularized backpropagation algorithm, and finally, the first-stage student model is output.

[0177] The dynamic exit module, including an adapter and a shared classifier, is used to train the first-stage student model to obtain the third-stage student model. Specifically, after multi-teacher knowledge distillation to obtain the first-stage student model, a dynamic exit module is added to each layer of the first-stage student model. This module derives the confidence function by limiting the pre-obtained parameters of the traditional confidence function, such as... Figure 6 As shown, the adapter adds one-dimensional convolution and one-dimensional flattening to the feature layers of each layer's output from the first-stage student model. The number of channels is scaled using 1×1 convolutions to ensure that the feature maps have the same hidden dimension after flattening. The output of the adapter's one-dimensional flattening is then input into the shared classifier, such as... Figure 5 The shared classifier shown first undergoes further nonlinearization through a fully connected layer. The result of the nonlinearization is linearly mapped to a 1D output, and finally, a 0-1 activation function is used to obtain the exit probability of each layer. The exit probabilities are then used to determine whether a preset exit rule is met. This preset exit rule includes:

[0178] First, assume the predetermined dynamic exit threshold for the i-th layer is thresi. Note that the threshold may be different for each layer, and the threshold is obtained by a genetic algorithm that balances the accuracy and speed of the model prediction within a given validation set. Let pi be the probability of exiting the current layer for each layer. Then, as long as pi > thresi, the user can exit the layer.

[0179] If the exit probability of the first layer meets the preset exit rule, the calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained. If the exit probability of the first layer does not meet the preset exit rule, the calculation continues for the next layer until the exit probability of the obtained layer meets the preset exit rule. At this point, the calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained. Finally, the training process for the second-stage student model is repeated for at least one stage, and the parameters of the dynamic exit module are modified to optimize the training of the dynamic exit module, resulting in the third-stage student model.

[0180] The reparameterization module is used to reparameterize a multi-branch, multi-head student model into a single-branch, single-head student model. The specific implementation steps include:

[0181] Multi-head reparameterization, when the model capacity is appropriate, can indeed improve model performance through multi-model distillation. Therefore, the output of the last convolutional layer of the third-stage student model is flattened, and the hidden layers are expanded to form multiple distillation heads. These multiple distillation heads are used to extract hidden features from multiple trained teacher models. Simultaneously, these distillation heads can be used to form the model's prediction head through simple linear projection. Through multi-head reparameterization, the student model aggregates information from multiple trained teacher models and implicitly integrates their outputs. This effectively transforms multi-head reparameterization into a single head.

[0182] Multi-branch reparameterization, compared to single-branch structures, extracts multi-scale information through convolutional kernels of varying sizes. In a multi-branch, multi-head student model, each module's convolutional layer is followed by a batch normalization layer, which transforms each convolutional kernel and the subsequent batch normalization layer into a new convolution with a bias vector. Let {F} c ,b c Given the new kernel and deviation, formula (1) can be transformed into:

[0183]

[0184] in, Let δ be the weight of the j-th channel of the one-dimensional convolution kernel. j Let γ be the variance of the batch normalized layer of the j-th channel of the one-dimensional convolution kernel. j Let μ be the learnable variance of the batch normalized layer of the j-th channel of the one-dimensional convolutional kernel. j β is the mean of the batch normalized layer of the j-th channel of the one-dimensional convolution kernel. j F is the learnable mean of the batch normalized layer of the j-th channel of the one-dimensional convolutional kernel. c This refers to the weights of the new one-dimensional convolutional kernel after batch normalization of the layer weights and the one-dimensional convolutional kernel. c This is to bias the one-dimensional convolution kernel and the new one-dimensional convolution kernel after batch normalization of layer reparameters.

[0185] Furthermore, convolutional kernels of different sizes are additive only if they have the same number of output channels and reasonable padding. Specifically, the multi-branch reparameterization proposed in this application will be illustrated using a three-layer convolutional layer as an example. Figure 7 As shown, the specific implementation includes:

[0186] exist Figure 7 In this model, the identity branch can be viewed as a 1×1 convolution with an identity matrix. These convolution kernels can then be reconstructed and merged using the principle of convolution additivity to create a new reparameterized kernel. This means that convolution kernels from multiple branches can be merged. Figure 7In this diagram, C1, C2, C3, and C4 are the parameters of the convolution kernel. Finally, multi-branch reparameterization is used to convert the multi-branch function into a single-branch function.

[0187] The knowledge distillation architecture for edge devices provided in this application realizes multi-teacher knowledge distillation of different structures by training teacher models with different structures and then distilling knowledge on multi-branch, multi-head student models. It also adds a dynamic exit module and a reparameterization module to the student model to achieve adaptive training on the input, while also achieving the goal of simplifying the student model without weakening its learning ability.

[0188] The following example, using a three-layered multi-branch, multi-head student model, illustrates the edge device-oriented knowledge distillation method proposed in this application. Figure 8 A flowchart illustrating a specific example of the edge device-oriented knowledge distillation method provided in this application. The specific implementation steps include:

[0189] S201: Obtain the configuration information of the models stored in the model repository, select multiple teacher models with different structures for training, obtain multiple trained teacher models, and perform knowledge distillation on the multi-branch, multi-head student model based on the multiple trained teacher models.

[0190] In this step, the configuration information of the stored models is obtained from the industrial scene model repository. Multiple teacher models with different structures are selected and trained to obtain multiple trained teacher models. Based on the multiple trained teacher models, the predicted hidden layer and predicted label information of the multiple trained teacher models are obtained. Whitening and noise reduction processing is performed on the hidden layer of each trained teacher model to obtain the processed hidden layer of each trained teacher model. Three-layer stacking processing is performed based on multiple basic modules to generate a multi-branch multi-head student model. Based on the processed hidden layer of each trained teacher model, the predicted label information, and the multi-branch multi-head student model, multi-teacher knowledge distillation processing is performed to obtain the first-stage student model.

[0191] In one specific implementation, configuration information of the model stored in the industrial scenario model is obtained. This configuration information is the multi-condition industrial time series. This configuration information is input into multiple teacher models with different structures, and each teacher model with a different structure is trained separately. The student model adopts a three-layer basic module stacked design. Each module includes three one-dimensional convolutional branches of different sizes and a batch normalization layer. Knowledge distillation is performed on the multi-branch multi-head student model based on the multiple trained teacher models to obtain the first-stage student model.

[0192] S202: Add a dynamic exit module between each layer of the student model in the first stage.

[0193] In this step, the configuration information obtained above is input into the first-stage student model. At the same time, a dynamic jump-out module is added between each layer of the first-stage student model to train the first-stage student model. After the training is completed, the parameters of the first-stage student model are frozen to obtain the second-stage student model. The second-stage student model is then trained to obtain the third-stage student model.

[0194] In one specific implementation, the first-stage student model is adaptively trained based on the configuration information obtained above. Simultaneously, a dynamic exit module is added between each layer of the first-stage student model. A confidence function is pre-designed in the dynamic exit module. The outputs of the first convolutional layer and the batch normalization layer of the first-stage student model are input to the dynamic exit module. Based on the outputs of the first convolutional layer and the batch normalization layer of the first-stage student model, the dynamic exit module calculates the confidence of the first layer according to the pre-designed confidence function. Then, it calculates the exit probability of the first layer based on the shared classifier in the dynamic exit module. Finally, it determines whether the calculated exit probability of the first layer satisfies a preset exit rule, wherein the preset exit rule includes:

[0195] First, assume the predetermined dynamic exit threshold for the i-th layer is thresi. Note that the threshold may be different for each layer, and the threshold is obtained by a genetic algorithm that balances the accuracy and speed of the model prediction within a given validation set. Let pi be the probability of exiting the current layer for each layer. Then, as long as pi > thresi, the user can exit the layer.

[0196] If the calculated bounce probability of the first layer meets the preset bounce rule, the calculation ends, and the second-stage student model is obtained. If not, the confidence score of the second layer is calculated, and the bounce probability of the second layer is calculated based on the confidence score. The calculation continues to check whether the calculated bounce probability of the second layer meets the preset bounce rule until a layer is reached where the calculated bounce probability meets the preset bounce rule. At this point, the calculation ends, and the parameters of the first-stage student model are frozen, resulting in the second-stage student model. The dynamic bounce module of the second-stage student model is trained for at least one stage, finally yielding the third-stage student model.

[0197] S203: Reparameterization of multi-branch, multi-head student models.

[0198] In this step, after obtaining the third-stage student model, the third-stage student model is reparameterized to obtain a single-branch, single-head target student model.

[0199] In one specific implementation, after obtaining the third-stage student model, the convolutional branches of each layer of the third-stage student model are reparameterized. Each convolutional kernel and its subsequent batch normalization layer are transformed into a new convolution with a bias vector. Then, the output of the last convolutional layer of the third-stage student model is flattened, and the hidden layers are expanded to form multiple distillation heads. These multiple distillation heads are used to extract the hidden features of multiple trained teacher models. The multi-branch, multi-head student model aggregates the information from multiple trained teacher models and reparameterizes it into an implicit ensemble. This results in a single-branch, single-head target student model.

[0200] This application illustrates a knowledge distillation method for edge devices. The student model employs a three-layer stacked basic module approach. By training multiple teacher models with different selected structures, multiple training sums of teacher models are obtained. Then, knowledge distillation is performed on the multi-branch, multi-head student model, achieving the goal of multi-teacher knowledge distillation with different structures. Dynamic exit modules and reparameterization modules are used to optimize the training of the student model, achieving adaptive training to the input and simplifying the student model.

[0201] Figure 9 This is a schematic diagram of the structure of a knowledge distillation device for edge devices provided in an embodiment of this application, as shown in the following figure. Figure 9 As shown, the knowledge distillation apparatus 300 for edge devices includes:

[0202] The first training module 301 is used to train multiple selected teacher models with different structures according to the configuration information of the models stored in the model repository, so as to obtain multiple trained teacher models.

[0203] The information acquisition module 302 is used to acquire the predicted hidden layer and predicted label information of multiple trained teacher models.

[0204] The first processing module 303 is used to perform whitening and noise reduction processing on the hidden layer of each trained teacher model to obtain the processed hidden layer of each trained teacher model.

[0205] The second processing module 304 is used to perform multi-layer stacking processing based on multiple basic modules to generate a multi-branch multi-head student model, wherein each basic module includes multiple one-dimensional convolutional branches of different sizes and batch normalization layers.

[0206] Distillation module 305 is used to perform multi-teacher knowledge distillation based on the hidden layer of each trained teacher model, the predicted label information, and the multi-branch multi-head student model to obtain the first-stage student model.

[0207] The third processing module 306 is used to add a dynamic exit module to each layer of the first-stage student model, calculate the confidence of each layer according to the pre-designed confidence function, freeze the parameters of the first-stage student model, and obtain the second-stage student model.

[0208] The second training module 307 is used to train the dynamic exit module of the second-stage student model for at least one stage, using the confidence level of each layer as a label, to obtain the third-stage student model.

[0209] The reparameterization module 308 is used to reparameterize the student model in the third stage to obtain a single-branch, single-head target student model.

[0210] Figure 10 A schematic diagram of the structure of the knowledge distillation device for edge devices provided in this application is shown in Embodiment 2. Figure 10 As shown, the distillation module 304 includes:

[0211] The first processing unit 3041 is used to flatten and expand the output of the last convolutional layer of the multi-branch multi-head student model to different knowledge distillation heads, so as to obtain multiple distillation heads after processing.

[0212] The normalization processing unit 3042 is used to normalize the processed multiple distillation heads through the batch normalization layer to obtain the prediction output of the multi-branch multi-head student model.

[0213] The first distillation unit 3043 is used to calculate the knowledge distillation loss by using the output of the penultimate hidden layer of multiple trained teacher models, and the multiple distillation heads after distillation processing.

[0214] The second processing unit 3044 is used to set the mixing weights and obtain the mixed labels based on the predicted label information and the real labels of multiple trained teacher models.

[0215] The second distillation unit 3045 is used to distill the predicted output of the multi-branch multi-head student model by mixing labels and to calculate the mean squared error loss.

[0216] The third processing unit 3046 is used to obtain the target loss function by adding the knowledge distillation loss and the mean square error loss.

[0217] Optimization unit 3047 optimizes the model parameters of the multi-branch, multi-head student model using a regularized backpropagation algorithm based on the objective loss function, thus obtaining the first-stage student model.

[0218] Figure 11 A schematic diagram of the structure of the knowledge distillation device for edge devices provided in this application is shown in Embodiment 3. Figure 11As shown, the third processing module 306 includes:

[0219] The first processing unit 3061 is used to add a dynamic exit module between each layer of the first-stage student model to obtain the processed first-stage student model.

[0220] The parameter limiting unit 3062 is used to limit the parameters of the pre-obtained traditional confidence function to obtain the confidence function.

[0221] The second processing unit 3063 is used to calculate the confidence level of the first layer of the processed first-stage student model according to the confidence level function, and to determine whether the confidence level of the first layer meets the preset exit rule.

[0222] The third processing unit 3064 is used to end the calculation if the confidence level of the first layer meets the exit rule, freeze the parameters of the first-stage student model, and obtain the second-stage student model.

[0223] The fourth processing unit 3065 is used to continue calculating the confidence of the next layer if the confidence of the first layer does not meet the exit rule, and to determine whether the confidence of the next layer meets the exit rule, until the confidence of the layer meets the preset exit rule, then the calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained.

[0224] This application also provides an electronic device 400. Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 400 includes:

[0225] Memory 401 is used to store computer instructions.

[0226] Processor 402 is configured to execute computer instructions stored in memory, causing the electronic device to perform the technical solutions of any of the foregoing method embodiments.

[0227] This application also provides a storage medium storing a computer program; when the computer program is executed, it implements the technical solutions of any of the foregoing method embodiments.

[0228] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0229] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A knowledge distillation method for edge devices, comprising: Based on the configuration information of the models stored in the model repository, multiple selected teacher models with different structures are trained to obtain multiple trained teacher models. Obtain the predicted hidden layer and predicted label information of multiple trained teacher models, and perform whitening and noise reduction processing on the hidden layer of each trained teacher model to obtain the processed hidden layer of each trained teacher model. The model is generated by stacking multiple basic modules to generate a multi-branch, multi-head student model. The model parameters of the multi-branch, multi-head student model are initialized. Each basic module includes multiple one-dimensional convolutional branches of different sizes and batch normalization layers. Based on the hidden layer processed by each trained teacher model, the predicted label information, and the multi-branch multi-head student model, multi-teacher knowledge distillation is performed to obtain the first-stage student model. A dynamic exit module is added to each layer of the first-stage student model. Based on a pre-designed confidence function, the confidence level of each layer is calculated, and the parameters of the first-stage student model are frozen to obtain the second-stage student model. The confidence function is set as follows: ,in, To scale the error, To prevent the probability from being too high or too low, It is a label smoothing parameter used to smooth labels, improve the model's predictive performance, and prevent overfitting; Using the confidence level of each layer as a label, the dynamic exit module of the second-stage student model is trained for at least one stage to obtain the third-stage student model. The third-stage student model is reparameterized to obtain a single-branch, single-head target student model.

2. The method according to claim 1, characterized in that, The process of performing multi-teacher knowledge distillation based on the hidden layers processed by each trained teacher model, the predicted label information, and the multi-branch multi-head student model to obtain the first-stage student model includes: Based on the output of the last convolutional layer of the multi-branch multi-head student model, the output of the last convolutional layer is flattened and expanded to different knowledge distillation heads to obtain multiple processed distillation heads. The batch normalization layer normalizes the processed multiple distillation heads to obtain the prediction output of the multi-branch multi-head student model. The knowledge distillation loss is calculated by distilling the processed multiple distillation heads using the output of the penultimate hidden layer of the multiple trained teacher models. Based on the predicted label information and the real labels of the multiple trained teacher models, a mixed weight is set to obtain a mixed label; By distilling the predicted output of the multi-branch multi-head student model using the mixed labels, the mean squared error loss is calculated. The target loss function is obtained by adding the knowledge distillation loss and the mean square error loss. Based on the target loss function, the model parameters of the multi-branch, multi-head student model are optimized using a regularized backpropagation algorithm to obtain the first-stage student model.

3. The method according to claim 1, characterized in that, The second-stage student model is obtained by adding a dynamic exit module to each layer of the first-stage student model, calculating the confidence level of each layer according to a pre-designed confidence function, freezing the parameters of the first-stage student model, and then: The dynamic exit module is added between each layer of the first-stage student model to obtain the processed first-stage student model, wherein the dynamic exit module includes an adapter and a shared classifier; By limiting the parameters of a pre-obtained traditional confidence function, a new confidence function is obtained. Based on the confidence function, the confidence of the first layer of the processed first-stage student model is calculated, and it is determined whether the confidence of the first layer meets the preset exit rule. If the confidence level of the first layer satisfies the exit rule, the calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained. If the confidence level of the first layer does not meet the exit rule, the confidence level of the next layer is calculated, and it is determined whether the confidence level of the next layer meets the exit rule, until the confidence level of the next layer meets the preset exit rule. The calculation ends, the parameters of the first-stage student model are frozen, and the second-stage student model is obtained.

4. The method according to claim 1, characterized in that, The reparameterization process for the third-stage student model to obtain a single-branch, single-head target student model includes: Using neural network degradation theory, the multi-head outputs of the third-stage student model are implicitly integrated, and each convolution kernel and its corresponding batch normalization layer in the third-stage student model are reconstructed and fused through the principle of convolution additivity to obtain the single-branch single-head target student model.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The single-branch, single-head target student model is deployed in industrial edge devices using a pre-configured deployment algorithm.

6. A knowledge distillation apparatus for edge devices, comprising: The first training module is used to train multiple selected teacher models with different structures according to the configuration information of the models stored in the model repository, so as to obtain multiple trained teacher models. The information acquisition module is used to acquire the predicted hidden layer and predicted label information of multiple trained teacher models; The first processing module is used to perform whitening and noise reduction processing on the hidden layer of each trained teacher model to obtain the processed hidden layer of each trained teacher model. The second processing module is used to perform multi-layer stacking processing based on multiple basic modules to generate a multi-branch multi-head student model and initialize the model parameters of the multi-branch multi-head student model. Each basic module includes multiple one-dimensional convolutional branches of different sizes and batch normalization layers. The distillation module is used to perform multi-teacher knowledge distillation based on the hidden layer processed by each trained teacher model, the predicted label information, and the multi-branch multi-head student model to obtain the first-stage student model. The third processing module is used to add a dynamic exit module to each layer of the first-stage student model, calculate the confidence level of each layer according to a pre-designed confidence function, freeze the parameters of the first-stage student model, and obtain the second-stage student model; the confidence function is set as follows: ,in, To scale the error, To prevent the probability from being too high or too low, It is a label smoothing parameter used to smooth labels, improve the model's predictive performance, and prevent overfitting; The second training module is used to train the dynamic exit module of the second-stage student model for at least one stage, using the confidence level of each layer as a label, to obtain the third-stage student model. The reparameterization module is used to reparameterize the student model in the third stage to obtain a single-branch, single-head target student model.

7. The apparatus according to claim 6, characterized in that, The distillation module includes: The first processing unit is used to flatten and expand the output of the last convolutional layer of the multi-branch multi-head student model to different knowledge distillation heads, so as to obtain multiple distillation heads after processing. The normalization processing unit is used to normalize the processed multiple distillation heads through the batch normalization layer to obtain the prediction output of the multi-branch multi-head student model. The first distillation unit is used to distill the processed multiple distillation heads through the output of the penultimate hidden layer of the multiple trained teacher models, and calculate the knowledge distillation loss. The second processing unit is used to set the mixing weights and obtain the mixed labels based on the predicted label information and the real labels of the multiple trained teacher models; The second distillation unit is used to distill the prediction output of the multi-branch multi-head student model through the mixed labels and calculate the mean squared error loss. The third processing unit is used to obtain the target loss function by adding the knowledge distillation loss and the mean square error loss; The optimization unit optimizes the model parameters of the multi-branch, multi-head student model using a regularized backpropagation algorithm based on the target loss function to obtain the first-stage student model.

8. The apparatus according to claim 6, characterized in that, The third processing module includes: The first processing unit is used to add the dynamic jump-out module between each layer of the first-stage student model to obtain the processed first-stage student model. The parameter limiting unit is used to limit the parameters of the pre-obtained traditional confidence function to obtain the confidence function. The second processing unit is used to calculate the confidence level of the first layer of the processed first-stage student model according to the confidence level function, and to determine whether the confidence level of the first layer meets the preset exit rule. The third processing unit is used to end the calculation if the confidence level of the first layer meets the exit rule, freeze the parameters of the first-stage student model, and obtain the second-stage student model. The fourth processing unit is used to continue calculating the confidence of the next layer if the confidence of the first layer does not meet the exit rule, and to determine whether the confidence of the next layer meets the exit rule, until the confidence of the next layer meets the preset exit rule, then end the calculation, freeze the parameters of the first stage student model, and obtain the second stage student model.

9. An electronic device, characterized in that, Includes: processor, memory; The memory is used to store computer instructions; The processor is configured to execute computer instructions stored in the memory, causing the electronic device to perform the knowledge distillation method for edge devices as described in any one of claims 1 to 5.

10. A storage medium, characterized in that, The storage medium stores a computer program; When the computer program is executed, it implements the knowledge distillation method for edge devices as described in any one of claims 1 to 5.

Images