Knowledge distillation method and device, computer device and readable storage medium

Abstract

Embodiments of the present application provide a knowledge distillation method and device, computer equipment and a readable storage medium. The method comprises: based on a preset edge deployment model and a guide model, constructing an output distribution loss based on the difference in the prediction probability distribution output for the sample data in the sample data set; based on the attention matrix difference output by the guide model and the edge deployment model, constructing an attention distillation loss; based on the feature map difference output by the feature extraction level of the guide model and the edge deployment model, constructing a multi-level feature distillation loss; based on the feature similarity difference output by the guide model and the edge deployment model, constructing a relationship distillation loss; based on the attention distillation loss, the multi-level feature distillation loss and the relationship distillation loss, constructing a comprehensive loss, and based on the multiple output distribution losses and the multiple comprehensive losses corresponding to the multiple guide models, performing knowledge distillation on the preset edge deployment model to obtain a target edge deployment model. In this way, the generalization ability and performance of the edge deployment model can be improved.

Description

Technical Field

[0001] This application relates to the field of knowledge distillation technology, and in particular to a knowledge distillation method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] Knowledge distillation (KD) is a model compression and knowledge transfer technique. Its core idea is to transfer the knowledge contained in a well-trained, complex and powerful "guide model" to a simpler and less computationally intensive "edge deployment model". This allows the edge deployment model to not only fit the real categories, but also learn the fine-grained cognition of the similarity between categories from the guide model.

[0003] In related technologies, a commonly used knowledge distillation scheme is single-teacher knowledge distillation combined with model compression techniques. Specifically, this scheme typically selects a specific large pre-trained model as the guide model. During the distillation phase, the label probability distribution generated by the guide model during training is used as an additional supervision signal, jointly optimized with the hard-label loss of the edge deployment model itself, thereby achieving knowledge transfer. However, this single-guide model knowledge distillation method relies on knowledge from only a single source for transfer, leading to problems of knowledge dimensional uniformity and homogenization. This makes it difficult to meet the requirements of edge devices for model diversity and adaptability, resulting in insufficient generalization and poor model performance in the distilled edge deployment model. Summary of the Invention

[0004] This application proposes a knowledge distillation method, apparatus, computer equipment, and readable storage medium, which can improve the generalization ability and performance of the edge deployment model after knowledge distillation.

[0005] To achieve the above objectives, a first aspect of this application proposes a knowledge distillation method, the method comprising:

[0006] Obtain a sample dataset, and based on the preset edge deployment model and each bootstrap model, construct the corresponding output distribution loss for the difference between the predicted probability distributions of each sample data output in the sample dataset;

[0007] Based on each attention head of each guiding model and each edge deployment model, a corresponding attention distillation loss is constructed for the differences between the attention matrices output by each sample data.

[0008] Based on each feature extraction level of each guiding model and the edge deployment model, a corresponding multi-level feature distillation loss is constructed for the differences between the feature maps output by each sample data.

[0009] Based on each of the guiding models and the edge deployment models, a corresponding relation distillation loss is constructed for the difference between the output feature similarities of any sample data pairs in the sample dataset.

[0010] Based on the attention distillation loss, the multi-level feature distillation loss, and the relation distillation loss, a comprehensive loss for each guided model is constructed. Then, based on the multiple output distribution losses and multiple comprehensive losses corresponding to the multiple guided models, knowledge distillation is performed on the preset edge deployment model to obtain the target edge deployment model. The multiple guided models have different network architectures.

[0011] Accordingly, a second aspect of the embodiments of this application proposes a knowledge distillation apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire a sample dataset and, based on the preset edge deployment model and each bootstrap model, construct a corresponding output distribution loss for the difference between the predicted probability distributions of each sample data output in the sample dataset.

[0013] The first construction module is used to construct a corresponding attention distillation loss based on the differences between the attention matrices output by each sample data, for each attention head of each guiding model and each edge deployment model.

[0014] The second construction module is used to construct a corresponding multi-level feature distillation loss based on the differences between the feature maps output by each sample data, according to each feature extraction level of each guiding model and the edge deployment model.

[0015] The third construction module is used to construct a corresponding relation distillation loss based on the differences in the output feature similarity between any sample data pairs in the sample dataset, based on each guiding model and the edge deployment model.

[0016] The distillation module is used to construct the comprehensive loss of each guided model based on the attention distillation loss, the multi-level feature distillation loss and the relation distillation loss, and to perform knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and multiple comprehensive losses corresponding to the multiple guided models to obtain the target edge deployment model, wherein the multiple guided models have different network architectures.

[0017] In some embodiments, the distillation module is further configured to:

[0018] Obtain the data properties corresponding to the sample data, wherein the data properties include at least one of image properties, text properties, and multimodal properties;

[0019] Obtain the adaptability index of the network architecture of each bootstrap model to the data properties, and determine the fusion weight corresponding to each bootstrap model based on the adaptability index;

[0020] Based on the fusion weights corresponding to each guided model, the output distribution loss and the comprehensive loss are adjusted to obtain the corresponding target output distribution loss and target comprehensive loss;

[0021] Based on the multiple target output distribution losses and multiple target comprehensive losses corresponding to the multiple guidance models, knowledge distillation is performed on the preset edge deployment model to obtain the target edge deployment model.

[0022] In some embodiments, the acquisition module is further configured to:

[0023] Obtain preset adjustment parameters and temperature hyperparameters, wherein the temperature hyperparameters are used to soften the prediction probability distribution of each guided model;

[0024] Obtain the first predicted probability distribution output by each guided model for each sample data in the sample dataset, and the second predicted probability distribution output by the preset edge deployment model for each sample data, and determine the corresponding distribution divergence value based on the first predicted probability distribution and the second predicted probability distribution;

[0025] The first output distribution loss sub-item is determined based on the product of the adjustment parameter, the temperature hyperparameter, and the distribution divergence value.

[0026] Obtain the classification loss corresponding to the preset edge deployment model, and adjust the classification loss based on the adjustment parameters to obtain the second output distribution loss sub-item;

[0027] Based on the first output distribution loss sub-item and the second output distribution loss sub-item, the output distribution loss between each guiding model and the edge deployment model is constructed.

[0028] In some implementations, the first building module is further configured to:

[0029] Based on each first attention head of each guided model, output the first attention matrix corresponding to each sample data, and based on each second attention head of the edge deployment model, output the second attention matrix corresponding to each sample data;

[0030] Perform singular value decomposition on the second attention matrix to obtain the target second attention matrix;

[0031] Based on the difference between the first attention matrix and the corresponding target second attention matrix, a corresponding attention distillation loss is constructed;

[0032] Based on multiple attention distillation sub-losses corresponding to multiple first attention heads, an attention distillation loss is constructed between each guiding model and the edge deployment model.

[0033] In some implementations, the second building module is further configured to:

[0034] Based on each first feature extraction level of each guided model, a first feature map corresponding to each sample data is output, and based on each second feature extraction level of the edge deployment model, a second feature map corresponding to each sample data is output;

[0035] Based on the first feature map, the number of channels and spatial size of the second feature map are adjusted to obtain an updated second feature map;

[0036] The first feature map is subjected to norm normalization to obtain the target first feature map, and the updated second feature map is subjected to norm normalization to obtain the target second feature map.

[0037] The target difference value between the first feature map of the target and the corresponding second feature map of the target is calculated using a distance metric function.

[0038] Obtain the weight coefficient of each feature extraction level, and based on the product of the weight coefficient and the target difference value, obtain the hierarchical feature distillation loss corresponding to each feature extraction level;

[0039] Based on the sum of multiple feature distillation losses corresponding to multiple feature extraction levels, a multi-level feature distillation loss is constructed between each guiding model and the edge deployment model.

[0040] In some implementations, the second building module is further configured to:

[0041] For each of the first feature extraction levels, a global average pooling operation is performed on the corresponding first feature map to obtain a first vector;

[0042] The sum of multiple first vectors corresponding to multiple first feature extraction levels is obtained to obtain the total target vector;

[0043] Based on the ratio of the sum of the first vector and the target vector, the weight coefficient of each feature extraction level is obtained.

[0044] In some implementations, the third building module is further configured to:

[0045] For each guided model, extract the first feature vector of the first sample data and the second feature vector of the second sample data contained in any sample data pair;

[0046] Using the edge deployment model, the third feature vector corresponding to the first sample data and the fourth feature vector corresponding to the second sample data are extracted.

[0047] Obtain the first feature similarity between the first feature vector and the second feature vector, and determine the first ratio based on the ratio of the first feature similarity to the first temperature parameter corresponding to each guided model;

[0048] Obtain the second feature similarity between the third feature vector and the fourth feature vector, and determine the second ratio based on the ratio of the second feature similarity to the second temperature parameter corresponding to the edge deployment model;

[0049] The first ratio is normalized to obtain the target first ratio, and the second ratio is normalized to obtain the target second ratio.

[0050] Based on the difference between the first target ratio and the second target ratio, the relational distillation loss of the arbitrary sample data pair is obtained;

[0051] Based on multiple relational distillation losses of multiple sample data pairs corresponding to the sample dataset, a relational distillation loss between each bootstrap model and the edge deployment model is constructed.

[0052] Accordingly, a third aspect of the present application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the knowledge distillation method of any one of the embodiments of the first aspect of the present application.

[0053] Accordingly, a fourth aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the knowledge distillation method of any one of the embodiments of the first aspect of this application.

[0054] This application embodiment acquires a sample dataset and, based on a preset edge deployment model and each guiding model, constructs a corresponding output distribution loss for the differences between the predicted probability distributions output by each sample data in the sample dataset; based on each attention head of each guiding model and edge deployment model, constructs a corresponding attention distillation loss for the differences between the attention matrices output by each sample data; based on each feature extraction level of each guiding model and edge deployment model, constructs a corresponding multi-level feature distillation loss for the differences between the feature maps output by each sample data; based on each guiding model and edge deployment model, constructs a corresponding relation distillation loss for the differences between the output feature similarities of any sample data in the sample dataset; based on the attention distillation loss, multi-level feature distillation loss, and relation distillation loss, constructs a comprehensive loss for each guiding model; and performs knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and multiple comprehensive losses corresponding to multiple guiding models to obtain the target edge deployment model, wherein the multiple guiding models have distinct network architectures. Therefore, by introducing multiple guiding models with distinct network architectures, the problem of a single guiding model having a limited knowledge perspective can be effectively overcome, avoiding homogeneous dependencies during distillation. Simultaneously, by constructing a comprehensive loss function using a multi-layered knowledge distillation mechanism, richer and more diverse knowledge sources and finer-grained supervision signals can be provided. Specifically, multiple distinct guiding models can effectively solve the problem of knowledge homogeneity. Output distribution loss conveys class probability knowledge, attention distillation loss captures fine-grained attention mechanisms within the model, multi-level feature distillation loss ensures consistency in feature extraction layers, and relational distillation loss encodes global structural information between samples. This fusion of multi-layered, multi-source knowledge enables the edge deployment model to learn more robust and generalized representations, reducing the risk of overfitting and improving adaptability and performance on edge devices. In summary, this application can improve the generalization ability and performance of the knowledge-distilled edge deployment model. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the architecture of the knowledge distillation system provided in the embodiments of this application;

[0056] Figure 2 This is a flowchart of the knowledge distillation method provided in the embodiments of this application;

[0057] Figure 3 This is a schematic diagram of the functional modules of the knowledge distillation apparatus provided in the embodiments of this application;

[0058] Figure 4 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] Knowledge distillation (KD) is a model compression and knowledge transfer technique. Its core idea is to transfer the knowledge contained in a well-trained, complex and powerful "guide model" to a simpler and less computationally intensive "edge deployment model". This allows the edge deployment model to not only fit the real categories, but also learn the fine-grained cognition of the similarity between categories from the guide model.

[0063] In related technologies, a commonly used knowledge distillation scheme is single-teacher knowledge distillation combined with model compression techniques. Specifically, this scheme typically selects a specific large pre-trained model as the guide model. During the distillation phase, the label probability distribution generated by the guide model during training is used as an additional supervision signal, jointly optimized with the hard-label loss of the edge deployment model itself, thereby achieving knowledge transfer. However, this single-guide model knowledge distillation method relies on knowledge from only a single source for transfer, leading to problems of knowledge dimensional uniformity and homogenization. This makes it difficult to meet the requirements of edge devices for model diversity and adaptability, resulting in insufficient generalization and poor model performance in the distilled edge deployment model.

[0064] Based on this, embodiments of this application provide a knowledge distillation method, apparatus, computer device, and readable storage medium, which can improve the generalization ability of the edge deployment model after knowledge distillation and improve the performance of the edge deployment model.

[0065] The knowledge distillation method, apparatus, computer equipment, and readable storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the knowledge distillation system in the embodiments of this application is described.

[0066] Please refer to Figure 1 In some embodiments, this application provides a knowledge distillation system, including a terminal 11 and a server 12.

[0067] For example, terminal 11 can be a smartphone, tablet, smart camera, edge computing device, etc. Terminal 11 can be responsible for deploying a lightweight pre-defined edge deployment model and performing local real-time inference tasks, such as object detection and classification. Terminal 11 can also collect local input data (e.g., images, text) and perform basic preprocessing operations, such as normalization and cropping, to ensure that the data is adapted to the input requirements of the pre-defined edge deployment model. Furthermore, terminal 11 can also record inference latency and accuracy metrics and feed this performance data back to server 12 via the network so that poorly performing models can be adjusted and optimized in a timely manner.

[0068] Furthermore, the server-side component 12 can be a high-performance computer cluster, cloud server, or other hardware facility with powerful computing capabilities, responsible for running multiple guided models, executing knowledge distillation algorithms, and optimizing models. The server-side component 12 can run multiple heterogeneous guided models (e.g., CNN, Transformer architectures) and adjust the weights of each guided model in real time through a dynamic attention fusion mechanism, providing complementary knowledge sources. In addition, the server-side component 12 can execute complex and time-consuming knowledge distillation algorithms, using structured knowledge transfer strategies to generate or update lightweight pre-defined edge deployment models, and then distribute the optimized lightweight models to the terminal 11 via the network for deployment and use.

[0069] The knowledge distillation method in this application can be illustrated by the following examples.

[0070] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0071] In some implementations, knowledge distillation methods can be applied to scenarios where efficient, high-performance AI models (pre-defined edge deployment models) are deployed in resource-constrained environments. Specific application scenarios include, but are not limited to, smart mobile devices, Internet of Things (IoT) edge devices, autonomous vehicles, and some industrial automation scenarios.

[0072] In this embodiment, the description will focus on the knowledge distillation device, which can be integrated into a computer device. See also Figure 2 , Figure 2 This is a flowchart illustrating the steps of a knowledge distillation method provided in this application embodiment. Taking the knowledge distillation device specifically integrated into a terminal or server as an example, the specific process when the processor on the terminal or server executes the program instructions corresponding to the knowledge distillation method is as follows:

[0073] Step 101: Obtain the sample dataset, and based on the preset edge deployment model and each pilot model, construct the corresponding output distribution loss for the difference between the predicted probability distributions of each sample data output in the sample dataset.

[0074] In some implementations, in order to improve the accuracy and generalization ability of the lightweight model (preset edge deployment model) in edge deployment scenarios, sample datasets can be obtained, and based on the preset edge deployment model and multiple heterogeneous guiding models, an output distribution loss can be constructed for the difference in the predicted probability distribution of each sample data output. This enables the student model to efficiently learn the soft label knowledge of the teacher model, thereby reducing the accuracy loss caused by model compression and enhancing dynamic adaptability.

[0075] The sample dataset can be a collection of data containing the input samples and their annotation information required for training, such as a set of labeled images in an image classification task.

[0076] Among them, the preset edge deployment model can be a lightweight model designed for edge device resource constraints, that is, a student model, such as MobileNet or SqueezeNet.

[0077] The guiding model can be a teacher model. Because guiding models are often designed to be large and complex to achieve high accuracy and performance, they have high computational and storage requirements; for example, they may be deployed in the cloud or enterprise-level data centers. Specifically, in image classification tasks, the guiding model can be a large convolutional neural network such as ResNet, DenseNet, or Inception, while in natural language processing tasks, it may be a large-scale pre-trained model such as BERT or Transformer. The specific guiding model is determined based on the specific task.

[0078] The sample data can be a single input instance in the sample dataset and its corresponding label, such as an image to be classified.

[0079] The predicted probability distribution can be the set of probability values ​​for each category obtained after the model predicts the input sample data. Specifically, if there is a classification problem with n categories, the predicted probability distribution is a vector of length n, where each element represents the probability that the category is correctly predicted.

[0080] Among them, the output distribution loss can be used as a metric to measure the difference in the predicted distribution between the student model and the teacher model.

[0081] For example, for image classification tasks, sample datasets can be obtained from publicly available image datasets such as ImageNet and CIFAR-10. For natural language processing tasks, they can be obtained from datasets such as the Wikipedia corpus and the IMDB movie review dataset; for specific needs, data can also be automatically scraped from the Internet using web crawlers as sample datasets. The method for obtaining sample datasets can be determined according to the actual situation.

[0082] In some implementations, to calculate the output distribution loss, a sample dataset is first acquired, and each sample in the dataset is predicted using a pre-defined edge deployment model and each bootstrap model, resulting in their respective predicted probability distributions. Then, the output distribution loss is constructed by comparing the differences between these predicted probability distributions, using KL divergence as a metric. For example, in an image classification task, for sample image 'a' in the corresponding dataset, the bootstrap model can predict its probability of belonging to three sample categories as [0.7, 0.2, 0.1], while the pre-defined edge deployment model predicts probabilities as [0.6, 0.3, 0.1]. The difference between these two probability distributions can then be quantified by calculating the KL divergence, thus obtaining the output distribution loss. This loss characterizes the gap in knowledge transfer between the pre-defined edge deployment model and the bootstrap model. This facilitates subsequent optimization of the lightweight edge deployment model, enabling it to learn from the knowledge of larger and more complex bootstrap models, thereby improving model performance.

[0083] The above methods can effectively realize the transfer of multi-dimensional knowledge from the teacher model (guided model) to the student model (pre-defined edge deployment model), reduce the accuracy loss during edge deployment, and improve the generalization ability and dynamic adaptability of the lightweight pre-defined edge deployment model in resource-constrained environments.

[0084] In some implementations, to achieve high-precision knowledge transfer and resource constraint adaptation of the preset edge deployment model for edge devices, the softening degree of the predicted distribution of the guided model can be optimized by introducing a temperature hyperparameter, and the weight of the output distribution loss sub-item can be dynamically adjusted by adjusting parameters to achieve synergistic optimization of knowledge fusion efficiency and model compression effect in multi-teacher collaborative distillation. For example, step 101, "based on the preset edge deployment model and each guided model, constructing the corresponding output distribution loss for the difference between the predicted probability distributions output by each sample data in the sample dataset," may include:

[0085] (101.1) Obtain the preset adjustment parameters and temperature hyperparameters, wherein the temperature hyperparameters are used to soften the prediction probability distribution of each guided model;

[0086] (101.2) Obtain the first predicted probability distribution output by each guided model for each sample data in the sample dataset, and the second predicted probability distribution output by the preset edge deployment model for each sample data, and determine the corresponding distribution divergence value based on the first predicted probability distribution and the second predicted probability distribution.

[0087] (101.3) Determine the first output distribution loss sub-item based on the product of the adjustment parameter, temperature hyperparameter and distribution divergence value;

[0088] (101.4) Obtain the classification loss corresponding to the preset edge deployment model, and adjust the classification loss based on the adjustment parameters to obtain the second output distribution loss sub-item;

[0089] (101.5) Based on the first output distribution loss sub-item and the second output distribution loss sub-item, construct the output distribution loss between each bootstrap model and the edge deployment model.

[0090] The adjustment parameter can be a scalar coefficient used to control the weight ratio between the output distribution loss item and the classification loss. It can be a value determined through grid search or Bayesian optimization, and can be used as follows: express.

[0091] Temperature hyperparameters can be used to adjust the predicted probability distribution of each bootstrap model, making its output probability distribution smoother and facilitating the learning of richer information by the student model. Temperature hyperparameters can be used... It indicates that it can be set by analyzing experimental data, for example, to 2.0 or 3.0.

[0092] The first predicted probability distribution can be the predicted label distribution of the guided model's output of the sample data, which can be used as follows: express.

[0093] The second predicted probability distribution can be the predicted label distribution of the pre-defined edge deployment model on the same sample data as the guiding model, which can be used... express.

[0094] The distribution divergence value can be the KL divergence, which measures the difference between the first and second predicted probability distributions. .

[0095] The first output distribution loss term can be the temperature hyperparameter. Adjusting parameters The product term with the distribution divergence value can be expressed as: .

[0096] The classification loss can be the original cross-entropy loss of the pre-defined edge deployment model for the sample data, which can be used as follows: express.

[0097] The second output distribution loss term can be the product of the adjustment parameter and the classification loss, which can be used as follows: express.

[0098] Specifically, the first step is to obtain the temperature hyperparameter. , It can be incorporated into the Softmax function, and the specific process is as follows:

[0099] ;

[0100] in, This represents the data output by the model's Softmax layer for the i-th category. This indicates that the output data of all categories is summed, where j is the summation index.

[0101] In some implementations, this application can determine the temperature hyperparameter through an adaptive mechanism. Specifically, the temperature hyperparameter T can be dynamically adjusted by utilizing the uncertainty of the model regarding sample data during training. For example, the temperature value at the current moment can be determined by calculating the entropy (representing uncertainty) of the predicted probability distribution of each sample data; or, for example, it can be obtained through experimental tuning on the validation set via cross-validation. For instance, during training, a set of candidate T values ​​(e.g., T=[0.5,1.0,2.0,5.0]) is set, and the output distribution loss and model accuracy corresponding to different T values ​​are evaluated on the validation set through multiple iterations. Finally, the value that minimizes the validation loss is selected as the temperature hyperparameter. In this way, by setting the temperature hyperparameter, it can be ensured that when the model is relatively certain about some sample data, the value of the temperature hyperparameter is low, making the probability distribution sharper (emphasizing high-confidence categories); while when the model is relatively uncertain about some sample data, the value of the temperature hyperparameter is high, making the probability distribution smoother (preserving low-probability information), thereby helping the pre-deployed edge model to better learn the knowledge of the guiding model.

[0102] For example, for each sample data First, obtain the original first predicted probability distribution output by the guided model. Then calculate its entropy. :

[0103] ;

[0104] Where n represents the number of categories, This represents the original first predicted probability distribution of the guided model for the i-th sample data belonging to the j-th category.

[0105] Then, the average entropy of the predicted probability distribution of all sample data can be calculated. :

[0106] ;

[0107] Where N represents the size of the sample dataset.

[0108] Furthermore, based on the aforementioned average entropy, the temperature hyperparameter T at the current moment can be dynamically calculated:

[0109] ;

[0110] in, and These are preset hyperparameters used to control the range and sensitivity of temperature values. This is a target entropy value, which can be set empirically or experimentally, representing the desired level of uncertainty. The temperature hyperparameter can be dynamically determined using the above formula. near At that time, the temperature hyperparameter approaches To remain relatively stable, when keep away At that time, the temperature hyperparameter increases or decreases accordingly to adjust the smoothness of the probability distribution.

[0111] Furthermore, for each sample in the sample dataset, the first predicted probability distribution of each guided model output pair can be obtained separately. The second predicted probability distribution output by the preset edge deployment model Among them, the first prediction probability distribution For each bootstrap model, the original first predicted probability distribution is obtained by passing each sample data in the input sample dataset through the model's forward propagation process. Then, using the formula The final first predicted probability distribution is calculated; similarly, for the preset edge deployment model, each sample data in the input sample dataset is used, and after the model's forward propagation process, the original second predicted probability distribution is obtained. Then use the formula The final second predicted probability distribution is calculated.

[0112] In some implementations, the corresponding distribution divergence value, i.e., the KL divergence, can be calculated based on the first and second predicted probability distributions to measure the difference between the two probability distributions. The specific calculation formula is as follows:

[0113] ;

[0114] Where n represents the number of categories; This represents the first predicted probability distribution for the i-th category. Let represent the second predicted probability distribution for the i-th category.

[0115] Furthermore, based on preset adjustment parameters Temperature over-parameter The first output distribution sub-loss term can be calculated by multiplying the above distribution divergence values: In this way, the degree of knowledge distillation can be flexibly controlled while softening the probability distribution.

[0116] Furthermore, the classification loss corresponding to the preset edge deployment model can be obtained. This loss, also known as cross-entropy loss, is used to measure the difference between the model's prediction and the true label. The calculation formula is as follows:

[0117] ;

[0118] in, Represents the true label of the sample data. This represents the predicted probability of the pre-defined edge deployment model for the i-th category. Then, based on the adjusted parameters... Adjusting the classification loss yields the second output distribution loss component: .

[0119] Finally, based on the first and second output distribution loss items calculated above, the final output distribution loss between each bootstrapping model and the edge deployment model is constructed. :

[0120] ;

[0121] By minimizing the output distribution loss, the parameters of the preset edge deployment model can be effectively optimized, so that it can be as close as possible to the performance of the bootstrap model while maintaining low computational and storage requirements, thus achieving efficient knowledge transfer.

[0122] By employing the above methods, we can extract high-order semantic knowledge from complex guidance models while ensuring the accuracy of the pre-defined edge deployment model in learning real labels. This achieves a balance between accuracy and efficiency in scenarios such as model compression and edge deployment, and has broad application prospects.

[0123] Step 102: Based on each attention head of each bootstrap model and edge deployment model, construct the corresponding attention distillation loss for the differences between the attention matrices output for each sample data.

[0124] In some implementations, in order to achieve efficient knowledge transfer of the attention mechanism in multi-teacher collaborative distillation, the difference loss term of the attention matrix between the heterogeneous teacher model and the edge deployment model can be constructed to dynamically optimize the structured knowledge transfer path and improve the generalization ability and deployment adaptability of the lightweight model in edge scenarios.

[0125] The attention head can be an independent feature extraction unit that is computed in parallel in a multi-head attention mechanism, such as each independent Query-Key-Value mapping module in the Transformer architecture.

[0126] The attention matrix can be the input sequence position correlation weight matrix calculated using the attention head.

[0127] Among them, attention distillation loss can be a loss term that measures the difference between the attention matrices of the teacher model and the student model.

[0128] In some implementations, the attention matrix corresponds to each attention head of the guiding model. It can be generated using the following multi-head attention formula:

[0129] ;

[0130] in, To guide the query weight matrix of the model, The key weight matrix is ​​used to guide the model. This is the dimension scaling factor.

[0131] Furthermore, the attention matrix of the pre-defined edge deployment model is... It is also based on multi-head attention generation, but additionally projected through low-rank singular value decomposition (SVD). Compressing the matrix dimension reduces computational complexity.

[0132] ;

[0133] in, The attention matrix corresponding to each attention head of the guiding model; The attention matrix for the pre-defined edge deployment model.

[0134] Then, the difference between the attention matrix of the bootstrap model and the preset edge deployment model can be calculated using the Frobenius norm:

[0135] ;

[0136] Finally, the loss for all attention heads (M in total) is calculated. The summations are then performed to obtain the final attention distillation loss:

[0137] ;

[0138] In some implementations, attention maps can be viewed as graph-structured data, and the topological relationships of the attention map can be modeled using GNNs to improve the ability to capture long-range dependencies. Specifically, the attention matrices of each guiding model and the pre-defined edge deployment model can be used... and Viewed as an adjacency matrix of a graph, nodes represent feature dimensions, and edge weights represent attention intensity. Then, a Graph Convolutional Network (GCN) or a Graph Attention Network (GAT) is used to extract the topological features of the attention graph to model its topological relationships; that is, for all neighbors k of node i, their features are computed. After weight matrix The linear transformation result is given by the formula:

[0139] ;

[0140] in, This represents the result of updating the features of node i through graph convolution; For activation function, Let i be the set of neighbors of node i. For learnable weights, The information of neighboring nodes is input and used to participate in the feature update of the current node i; The set associated with parameter k The number of elements in the array.

[0141] Furthermore, this can be achieved by comparing the first image embedding of the guided model. The second graph embedding of the preset edge deployment model The differences between them are used to construct the topology consistency loss:

[0142] ;

[0143] in, Indicates the use of the guided model After multiple layers The calculated node embeddings; For using the preset edge deployment model Similarly, through multiple layers The calculated node embeddings.

[0144] In some implementations, the aforementioned topological consistency loss can be used to replace the attention distillation loss, so as to capture global structural information of the attention graph (such as connectivity and clustering) while measuring the difference in attention values, thereby improving the ability to transfer complex semantic structures.

[0145] In some implementations, a second attention matrix corresponding to the model can be deployed at a preset edge. In this approach, a dynamic sparsity constraint is introduced, retaining only the first attention matrix of the guiding model. Highly relevant attention heads are used to improve the efficiency and accuracy of knowledge transfer; this is achieved through the first attention matrix of the guiding model. Highly relevant attention heads can be determined based on data type (text, image, time series, etc.) and task characteristics (classification, detection, generation, etc.). Different data types have different sensitivities to the relevance of attention heads, and different tasks have significantly different requirements for attention heads. For example, for image data, the Structural Similarity Index Measure (SSIM) and the local-to-global ratio can be combined to dynamically allocate sparsity (e.g., retaining 20% ​​of the global heads and 30% of the local heads out of 16 heads); for time series, key time nodes can be aligned using Dynamic Time Warping (DTW) to retain the corresponding attention heads.

[0146] By using the above methods, we can integrate the multi-scale attention feature representations of each heterogeneous guiding model, and use dynamic attention distillation loss to enhance the ability of the preset edge deployment model to pay attention to key features, thereby improving the efficiency of structured knowledge transfer and reducing the risk of accuracy degradation of the preset edge deployment model, thus providing technical support for robust inference of edge models in subsequent complex scenarios.

[0147] In some implementations, to achieve structured knowledge transfer of the attention mechanism in multi-teacher collaborative distillation, a difference loss term in the attention matrix between the heterogeneous teacher model and the edge deployment model can be constructed, and singular value decomposition can be introduced to optimize the expressive power of the student model's attention matrix. This improves the lightweight model's focus on key features and generalization performance, thereby reducing the risk of accuracy loss in edge scenarios. For example, step 102 may include:

[0148] (102.1) Based on each first attention head of each guided model, output the first attention matrix corresponding to each sample data, and based on each second attention head of the edge deployment model, output the second attention matrix corresponding to each sample data;

[0149] (102.2) Perform singular value decomposition on the second attention matrix to obtain the target second attention matrix;

[0150] (102.3) Based on the difference between the first attention matrix and the corresponding target second attention matrix, construct the corresponding attention distillation loss;

[0151] (102.4) Based on multiple attention distillation losses corresponding to multiple first attention heads, construct the attention distillation loss between each guiding model and the edge deployment model.

[0152] The first attention head can be an independent computational unit of the multi-head attention mechanism in the guided model, such as each independent Query-Key-Value mapping module in the Transformer architecture.

[0153] The first attention matrix can be the normalized similarity matrix output by the guiding model through the first attention head.

[0154] The second attention head can be an independent computational unit of the multi-head attention mechanism in the edge deployment model, such as the compressed Query-Key-Value mapping module in the lightweight Transformer structure.

[0155] The second attention matrix can be the original attention weight matrix output by the edge deployment model through the second attention head.

[0156] The target second attention matrix can be a low-rank approximation matrix reconstructed from the second attention matrix after singular value decomposition.

[0157] The attention distillation loss can be a Frobenius norm loss term that measures the difference between the first attention matrix and the target second attention matrix.

[0158] In some implementations, the first attention matrix corresponds to each attention head of the guiding model. It can be generated using the following multi-head attention formula:

[0159] ;

[0160] in, To guide the query weight matrix of the model, The key weight matrix is ​​used to guide the model. This is the dimension scaling factor.

[0161] Furthermore, the second attention matrix of the pre-defined edge deployment model Also based on multi-head attention generation, the specific formula is as follows:

[0162] ;

[0163] Furthermore, due to the need for low-rank SVD projection ( Aligning the attention matrix dimensions with the guiding model reduces computational complexity, thus yielding the target second attention matrix:

[0164] ;

[0165] in, To pre-deploy the query weight matrix of the edge deployment model, The key weight matrix for the pre-defined edge deployment model, This is the dimension scaling factor.

[0166] Then, the difference between the attention matrix of the guided model and the preset edge deployment model can be calculated using the Frobenius norm, which yields the attention distillation loss between a single first attention head and a second attention head:

[0167] ;

[0168] Finally, the losses of all (M in total) first attention heads (or all second attention heads, with the same number of attention heads) are summed to obtain the final attention distillation loss:

[0169] ;

[0170] By using the above methods, the multi-scale attention feature representations of heterogeneous guidance models can be integrated, and the redundant dimensions of the attention matrix of the preset edge deployment model can be compressed using singular value decomposition. This forces the preset edge deployment model to learn the attention distribution of the guidance model, effectively reducing the computational overhead of the preset edge deployment model.

[0171] Step 103: Based on each feature extraction level of each bootstrap model and edge deployment model, construct the corresponding multi-level feature distillation loss for the differences between the feature maps output by each sample data.

[0172] In some implementations, in order to achieve multi-level transfer of structured knowledge in multi-teacher collaborative distillation, a feature map difference loss term can be constructed between the heterogeneous teacher model and the edge deployment model at the feature extraction level to enhance the learning ability of the student model to spatial-semantic joint features, thereby improving the accuracy and dynamic adaptability of the lightweight model in edge scenarios.

[0173] The feature extraction layer can be a computational module in a neural network used to extract local or global features from the input data, such as different convolutional layers in a convolutional neural network or a multi-head attention layer in a Transformer architecture.

[0174] The feature map can be a multidimensional tensor representation generated by the feature extraction layer after processing the input sample data, such as a local spatial feature matrix extracted by the sliding operation of the convolution kernel, or a semantic association graph generated by the attention mechanism.

[0175] Among them, the multi-level feature distillation loss can be a loss term calculated based on the difference between the output feature maps of the guided model and the preset edge deployment model at multiple feature extraction levels.

[0176] In some implementations, knowledge transfer can be achieved by constructing a multi-level feature distillation loss by comparing the output feature maps of the guided model and the preset edge deployment model at multiple feature extraction levels. Specifically, the feature maps of the guided model and the preset edge deployment model can be compared first. Specifically, the feature maps of the guided model at layer l... Features of the k-th layer of the student model The number of channels is adjusted by 1×1 convolution, and then the spatial size is aligned by adaptive pooling to generate matching features. This process can be represented by the following formula:

[0177] ;

[0178] Furthermore, one can adopt The norm measures the feature difference between the guided model and the preset edge deployment model at each feature extraction level, and the multiple feature differences corresponding to all feature extraction levels are added together to obtain the multi-level feature distillation loss.

[0179] By using the above methods, the complementary expressions of heterogeneous teacher models at different feature extraction levels can be integrated, and the student model's ability to jointly model edge features and semantic features can be enhanced by using multi-level feature distillation loss. This reduces information loss during the transfer of structured knowledge and improves the model's generalization performance, thereby providing technical support for high-precision reasoning and efficient resource deployment of edge devices in complex scenarios.

[0180] In some implementations, to achieve multi-level transfer of structured knowledge in multi-teacher collaborative distillation, a feature map difference loss term can be constructed between the heterogeneous teacher model and the edge deployment model at the feature extraction level. This strengthens the student model's ability to learn spatial-semantic joint features, thereby improving the accuracy and dynamic adaptability of the lightweight model in edge scenarios. Step 103 may include:

[0181] (103.1) Based on each first feature extraction level of each guided model, output the first feature map corresponding to each sample data, and based on each second feature extraction level of the edge deployment model, output the second feature map corresponding to each sample data;

[0182] (103.2) Based on the first feature map, the number of channels and spatial size of the second feature map are adjusted to obtain an updated second feature map;

[0183] (103.3) Perform norm normalization on the first feature map to obtain the target first feature map, and perform norm normalization on the updated second feature map to obtain the target second feature map;

[0184] (103.4) Calculate the target difference value between the first feature map of the target and the corresponding second feature map of the target using the distance metric function;

[0185] (103.5) Obtain the weight coefficients of each feature extraction level, and obtain the hierarchical feature distillation loss corresponding to each feature extraction level based on the product of the weight coefficients and the target difference value;

[0186] (103.6) Based on the sum of the loss of multiple feature distillation sub-losses corresponding to multiple feature extraction levels, construct a multi-level feature distillation loss between each guiding model and the edge deployment model.

[0187] The first feature extraction layer can be a computational module in the guided model used to extract local or global features of the input data, such as a shallow convolutional layer in a convolutional neural network (CNN).

[0188] The first feature map can be a multidimensional tensor representation generated by the l-th first feature extraction layer of the guided model after processing the input sample data, and it can be obtained through forward propagation of the guided model. For example, it can be a local spatial feature matrix extracted by the sliding operation of the convolution kernel or a semantic association map generated by the attention mechanism. Each first feature extraction layer in the guided model can output a corresponding first feature map.

[0189] The second feature extraction layer can be a feature extraction module corresponding to the function in the edge deployment model, such as a compressed version of the convolutional layer or a simplified version of the Transformer module in a lightweight CNN structure.

[0190] The second feature map can be the original feature representation generated by processing sample data consistent with the guiding model after the k-th second feature extraction layer of the preset edge deployment model. It can be obtained through forward propagation of the preset edge deployment model, for example, by outputting a low-dimensional feature matrix through lightweight convolution operations. Each second feature extraction layer in the preset edge deployment model can output a corresponding second feature map.

[0191] The number of channels can be the number of feature maps in the depth dimension; for example, the feature map output by a convolutional layer has 64 channels.

[0192] The spatial dimensions can be the dimensions of the feature map in the height and width dimensions, for example, the spatial dimensions of the feature map are H×W.

[0193] The updated second feature map can be a second feature map adjusted through channel clipping or bilinear interpolation. For example, the number of channels in the second feature map can be adjusted to match that of the first feature map using a 1×1 convolution operation. Figure 1Thus, the updated second feature map is obtained.

[0194] The target first feature map can be the result of norm normalization of the first feature map.

[0195] The target second feature map can be the result of norm normalization of the updated second feature map.

[0196] The distance metric function can be a function used to calculate the differences between feature maps, such as Euclidean distance or cosine similarity.

[0197] The target difference value can be the distance measurement result between the first feature map of the target and the second feature map of the target.

[0198] The weight coefficient can be a scalar value used to adjust the importance of different feature extraction levels. It can be obtained through experimental adjustment or set by the operator.

[0199] The hierarchical feature distillation loss can be the product of the weight coefficient and the target difference value.

[0200] In some implementations, for each feature extraction level of the guided model, such as the l-th feature extraction level, a first feature map corresponding to the input sample data (e.g., sample data a) can be extracted. And for each feature extraction level of the predefined edge deployment model, such as the k-th feature extraction level, extract the second feature map of sample data (e.g., sample data a) that is consistent with the input of the guiding model. .

[0201] Furthermore, to make the second feature map match the first feature map in dimensionality, the number of channels and spatial size of the second feature map need to be adjusted. This can be achieved through a 1×1 convolution operation (…). Adjust the number of channels in the second feature map to match that of the first feature map. Figure 1 The spatial dimensions of the second feature map are adjusted to match those of the first feature map through adaptive pooling. Figure 1 Therefore, the updated second feature map can be obtained. This process can be represented as:

[0202] ;

[0203] Furthermore, in order to eliminate scale differences between feature maps, the first feature map is normalized to obtain the target first feature map. And perform norm normalization on the updated second feature map to obtain the target second feature map. .

[0204] In some implementations, the difference between the first feature map and the second feature map of the target can be calculated using a distance metric function (such as cosine similarity, Euclidean distance, etc.). For example, the target difference value can be calculated using Euclidean distance. .

[0205] Furthermore, weight coefficients for each feature level can be extracted. The product of the difference with the target value is used to calculate the hierarchical feature distillation loss corresponding to each feature extraction level. The weight coefficients are obtained by extracting the global representation of the features at each level of the teacher model through global average pooling and then normalizing it. The specific formula is as follows:

[0206] ;

[0207] Therefore, the weight coefficients can be multiplied by the target difference value to calculate the hierarchical feature distillation loss corresponding to the l-th feature extraction level: .

[0208] Furthermore, the multi-level feature distillation losses corresponding to multiple first feature extraction layers (one level feature distillation loss for each first feature extraction layer) can be summed to construct a multi-level feature distillation loss between the guided model and the edge deployment model. :

[0209] ;

[0210] By using the above methods, the complementary expressions of heterogeneous guided models at different feature extraction levels can be integrated, and the multi-level feature distillation loss can be used to enhance the ability of the preset edge deployment model to jointly model edge features and semantic features, thereby reducing information loss in the process of structured knowledge transfer and improving the generalization performance of the preset edge deployment model.

[0211] In some implementations, to ensure that feature extraction layers that contribute more to the overall representation capability of the model have higher weights in the loss function, enabling the pre-defined edge deployment model to learn the key features guiding the model more effectively, layer feature vectors can be obtained through global average pooling, and their weight coefficients can be calculated to optimize the multi-level feature distillation loss function and improve the model's generalization ability. For example, "obtaining the weight coefficients of each feature extraction layer" in (103.5) may include:

[0212] (103.5.1) For each first feature extraction level, perform global average pooling on the corresponding first feature map to obtain the first vector;

[0213] (103.5.2) Obtain the sum of multiple first vectors corresponding to multiple first feature extraction levels to obtain the total target vector;

[0214] (103.5.3) Based on the ratio of the sum of the first vector and the target vector, the weight coefficients of each feature extraction level are obtained.

[0215] The first vector can be a feature vector obtained by performing a global average pooling operation on the first feature map.

[0216] The sum of the target vectors can be the result of a linear superposition of multiple first vectors.

[0217] In some implementations, for each first feature extraction level of the guided model, a global average pooling operation can be performed on the corresponding first feature map to convert the first feature map into a single image. The first vector is obtained by averaging the feature values ​​of all corresponding spatial location levels. .

[0218] Furthermore, the sum of multiple first vectors corresponding to the multiple first feature extraction layers contained in the guided model can be obtained to obtain the total target vector. .

[0219] Among them, the weighting coefficient The global representation of features at each level of the teacher model can be extracted through Global Average Pooling (GAP) and then normalized. Specifically, it can be calculated based on the ratio of the sum of the first vector and the target vector, as shown in the following formula:

[0220] ;

[0221] Where L represents the total number of feature extraction layers in the guided model.

[0222] By using the above methods, the contribution of each first feature extraction level to the overall model performance can be quantified, so as to dynamically adjust the weight allocation of multi-level feature distillation loss and ensure that the preset edge deployment model can learn the key features of the guiding model more effectively, thereby improving the performance and generalization ability of the preset edge deployment model.

[0223] Step 104: Based on each bootstrap model and edge deployment model, construct the corresponding relational distillation loss for the difference in feature similarity between the outputs of any sample data pairs in the sample dataset.

[0224] In some implementations, in order to realize the transfer of higher-order semantic relation knowledge in multi-teacher collaborative distillation, a feature similarity difference loss term for the output of sample data pairs can be constructed by constructing heterogeneous teacher models and edge deployment models to enhance the modeling ability of student models for logical associations between categories, thereby improving the dynamic adaptability and generalization performance of the lightweight preset edge deployment model in edge scenarios.

[0225] The sample data pair can be any two input instances in the sample dataset; or, the sample data pair can be two input instances with semantic relationship, such as image pairs belonging to the same category or different categories in an image classification task, depending on the actual situation.

[0226] Feature similarity can be a similarity metric calculated by the model after extracting features from sample data pairs. For example, it can be calculated using cosine similarity or Euclidean distance.

[0227] Among them, relational distillation loss can be a loss term that measures the difference in similarity between the teacher model and the student model for the output features of the sample data.

[0228] In some implementations, the inner product operation can be used to compute the two feature vectors of the guided model for any two sample data pairs. and Similarly, the similarity between them is calculated through the inner product operation, using two feature vectors of the preset edge deployment model for any pair of sample data (consistent with the sample data pairs used by the guiding model). and The similarity between them.

[0229] Furthermore, the sigmoid function can be used to normalize the similarity separately. After normalization, the relationship distillation loss is obtained by calculating the cross-entropy loss between the normalized feature similarity of the guided model and the normalized feature similarity of the preset edge guided model.

[0230] By using the above methods, we can integrate the guiding model's ability to model high-order semantic relationships between sample data pairs, quantify the similarity between the guiding model and the preset edge guiding model in the feature space, and guide the preset edge guiding model to learn the logical associations between categories by minimizing the relation distillation loss, so that it can better approximate the feature representation of the guiding model.

[0231] In some implementations, to achieve the transfer of higher-order semantic relation knowledge in multi-teacher collaborative distillation, a difference loss term for the similarity between sample data and output features can be constructed between the guiding model and the edge deployment model. This strengthens the ability of the preset edge deployment model to model logical relationships between categories, thereby improving the dynamic adaptability and generalization performance of the lightweight model in edge scenarios. Step 104 may include:

[0232] (104.1) Through each guided model, extract the first feature vector of the first sample data contained in any sample data pair, and the second feature vector of the second sample data contained therein;

[0233] (104.2) Using the edge deployment model, extract the third feature vector corresponding to the first sample data and the fourth feature vector corresponding to the second sample data;

[0234] (104.3) Obtain the first feature similarity between the first feature vector and the second feature vector, and determine the first ratio based on the ratio of the first feature similarity to the first temperature parameter corresponding to each guided model;

[0235] (104.4) Obtain the second feature similarity between the third feature vector and the fourth feature vector, and determine the second ratio based on the ratio of the second feature similarity to the second temperature parameter corresponding to the edge deployment model;

[0236] (104.5) Normalize the first ratio to obtain the target first ratio, and normalize the second ratio to obtain the target second ratio;

[0237] (104.6) Based on the difference between the first target ratio and the second target ratio, the relational distillation loss of any sample data pair is obtained;

[0238] (104.7) Based on multiple relational distillation losses of multiple sample data pairs corresponding to the sample dataset, construct the relational distillation loss between each bootstrap model and the edge deployment model.

[0239] The first sample data can be the first input instance in a sample data pair, such as the image to be identified in an image classification task.

[0240] The first feature vector can be a feature representation extracted by the guiding model from the first sample data, such as a high-dimensional vector output by a fully connected layer.

[0241] The second sample data can be the second input instance in the sample data pair, such as an image that belongs to the same class or a different class as the first sample data. Both the first sample data and the second sample data are in the sample dataset.

[0242] The second feature vector can be a feature representation extracted by the guiding model from the second sample data, such as a high-dimensional vector output by a fully connected layer.

[0243] The third feature vector can be a feature representation extracted from the first sample data by the edge deployment model, such as a vector output by a lightweight convolutional layer.

[0244] The fourth feature vector can be a feature representation extracted from the second sample data by the edge deployment model, such as a vector output by a lightweight convolutional layer.

[0245] The first feature similarity can be the similarity measure of the sample data to the output feature vector by the guiding model.

[0246] The first temperature parameter can be a scalar value used in the guided model to adjust the smoothness of the Softmax output probability distribution.

[0247] The first ratio can be the ratio of the first feature similarity to the first temperature parameter.

[0248] The second feature similarity can be a similarity measure of the sample data to the output feature vector of the edge deployment model.

[0249] The second temperature parameter can be a scalar value used in the edge deployment model to adjust the smoothness of the Softmax output probability distribution.

[0250] The second ratio can be the ratio of the second feature similarity to the second temperature parameter.

[0251] The target first ratio can be the result of normalizing the first ratio.

[0252] The target second ratio can be the result of normalizing the second ratio.

[0253] Among them, the relational distillation loss can be a measure of the difference between the target first ratio and the target second ratio, which can be calculated using the mean square error formula.

[0254] For example, taking the first sample data as sample i and the second sample data as sample j, by inputting sample i and sample j into the guided model respectively, the first feature vector corresponding to sample i can be obtained. and the second feature vector corresponding to sample j Simultaneously, samples i and j are input into the edge deployment model to obtain the third feature vector corresponding to sample i. and the fourth feature vector corresponding to sample j .

[0255] Furthermore, this can be achieved by calculating the inner product between the first and second eigenvectors. The first feature similarity is obtained; then, the first feature similarity is divided by the first temperature parameter. The first ratio is obtained. Therefore, the distribution of the first feature similarity can be adjusted by the first temperature parameter, making the first feature similarity smoother or sharper, thereby improving the model's generalization ability.

[0256] And this can be achieved by calculating the inner product between the third and fourth eigenvectors. The second feature similarity is obtained; then, the second feature similarity is divided by the second temperature parameter. The second ratio is obtained. Therefore, the distribution of the second feature similarity can be adjusted by using the second temperature parameter, making the second feature similarity smoother or sharper, thereby improving the model's generalization ability.

[0257] Furthermore, the sigmoid function can be used. The first ratio is normalized to obtain the target first ratio. and the use of the sigmoid function The second ratio is normalized to obtain the target second ratio. In this way, the first ratio and the second ratio can be mapped to a range of 0 to 1, making it easier to perform calculations on the same scale.

[0258] In some implementations, the cross-entropy loss between the first target ratio and the second target ratio can be calculated to obtain the relational distillation loss. :

[0259] ;

[0260] Furthermore, the relational distillation loss is obtained by summing the relational distillation losses for all sample data pairs (each sample data pair corresponds to a relational distillation loss). :

[0261] ;

[0262] By using the above methods, the relational distillation loss can be used to enhance the learning of logical relationships between categories by the pre-defined edge deployment model, thereby guiding the learning of the pre-defined edge deployment model and enabling it to better approximate the feature representation of the guiding model.

[0263] Step 105: Based on attention distillation loss, multi-level feature distillation loss and relation distillation loss, construct the comprehensive loss of each guided model, and perform knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and multiple comprehensive losses corresponding to multiple guided models to obtain the target edge deployment model. Among them, the multiple guided models have different network architectures.

[0264] In some implementations, in order to achieve synergistic optimization of knowledge transfer efficiency and model compression effect in multi-guided model collaborative distillation, a comprehensive loss function can be constructed by fusing attention distillation loss, multi-level feature distillation loss and relation distillation loss, and the preset edge deployment model can be jointly optimized based on the output distribution loss of the heterogeneous teacher model, so as to improve the accuracy and resource adaptability of the preset edge deployment model in edge scenarios.

[0265] The comprehensive loss can be the total loss term that is the weighted sum of attention distillation loss, multi-level feature distillation loss and relation distillation loss. The weights of each loss can be adjusted according to the actual situation.

[0266] The target edge deployment model can be a lightweight model that has undergone collaborative distillation and comprehensive loss optimization of multiple guiding models, such as MobileNet or SqueezeNet models obtained by updating parameters through backpropagation.

[0267] For example, for each guided model, such as the ResNet-50 convolutional neural network, the corresponding attention distillation loss can be used. Multi-stage characteristic distillation losses With relation to distillation loss Weighted fusion is performed to obtain the comprehensive loss of the guided model. Specifically, the formula for calculating the comprehensive loss is as follows:

[0268] ;

[0269] in, It can be the task loss of the pre-defined edge deployment model, used to measure its performance when performing the main tasks (such as classification and regression), and is directly related to the difference between the output of the pre-defined edge deployment model and the true label. For example, it can be the classification loss. , , , It can be set according to the actual situation.

[0270] In some implementations, adaptive adjustment based on gradient descent can be used. , , , The value of is determined so that the preset edge deployment model can gradually approach the optimal solution. Specifically, for The value at t+1 during the training phase, that is... , , , The value of any weight in the training phase t+1 can be adjusted using the following formula:

[0271] ;

[0272] in, This represents the value of the i-th weight in the (t+1)-th iteration; This represents the value of the i-th weight in the t-th iteration; This represents the learning rate, which determines the magnitude of parameter updates; the larger the learning rate, the larger the update magnitude. Indicates the overall loss; This reflects the loss function on the weights Regarding the trend of change in direction, when the gradient is large, the exponential term is small, and the parameter update amplitude is relatively small, avoiding instability that may be caused by excessive jumps; when the gradient is small, the exponential term is large, allowing the parameters to be adjusted more significantly, accelerating the convergence process.

[0273] In some implementations, multiple bootstrapping models employ distinct network architectures, each designed to capture different types of information. For instance, convolutional neural networks excel at processing local features and spatial structure information in images, while transformer architectures are better at capturing global dependencies. These different architectures often imply different prior assumptions and inductive biases. By using bootstrapping models with diverse architectures, it is possible to ensure that the pre-defined edge deployment model learns rich knowledge from multiple perspectives. Leveraging this differentiated prior knowledge, the pre-defined edge deployment model performs better when faced with new tasks or unseen data.

[0274] For example, the guiding model may include convolutional neural networks (such as ResNet-50), visual transformers (such as ViT-Base), self-attention mechanism-enhanced networks (such as Swing Transformer), and so on.

[0275] Furthermore, after calculating the comprehensive loss of a bootstrap model, this comprehensive loss can be compared with the comprehensive losses of other bootstrap models. The weighted average is calculated to obtain the target comprehensive loss. For example, the first weight corresponds to the first weight in the guided model 1. and comprehensive losses The second weight corresponds to the guiding model 2. and comprehensive losses The third weight corresponding to the guiding model 3 and comprehensive losses The fourth weight corresponds to the guiding model 4. and comprehensive losses So, the overall loss of the target It can be calculated using the following formula:

[0276] ;

[0277] At the same time, the output distribution loss of this guided model is compared with the output distribution loss of other guided models. A weighted average is calculated to obtain the target output distribution loss. The weighting method is similar to that used for calculating the target comprehensive loss, and will not be elaborated here. Finally, based on the target comprehensive loss and the target output distribution loss, the preset edge deployment model is distilled to adjust the model parameters, resulting in the target edge deployment model. Furthermore, the weighting values ​​of the losses corresponding to each model can be set according to actual conditions, such as by technical personnel, randomly, or based on the number of training iterations of the guiding model. For example, the guiding model with the most training iterations can be assigned better weights, etc. This application does not specifically limit this aspect.

[0278] In some implementations, the distillation process can be modeled as a differential equation:

[0279] ;

[0280] in, This represents the parameter vector of the preset edge deployment model; t represents the time variable, that is, the time elapsed or the number of iterations during the training process. This represents the rate of change of the preset edge deployment model over time, i.e., the speed at which the parameters are updated; The total loss function (the sum of the target comprehensive loss and the target output distribution loss, or the target comprehensive loss function, whichever is determined based on the actual situation) is related to... The gradient represents the rate of change of the loss function in the direction of the parameters; This represents a balancing factor, used to control the degree of influence of the differences between the pilot model and the preset edge deployment model; This represents the parameter vector of the k-th guided model.

[0281] This application can be widely used in edge computing scenarios with limited resources and requiring efficient intelligent inference, such as real-time image classification and enhanced photography on smartphones, real-time behavior recognition and anomaly detection in smart security cameras, equipment status monitoring and predictive maintenance in industrial IoT devices, lightweight environmental perception modules in autonomous vehicle systems, and real-time pathological image analysis in portable medical devices. Through multi-teacher knowledge distillation, the accuracy and generalization ability of edge deployment models can be significantly improved, while meeting the deployment requirements of low latency and low power consumption.

[0282] By calculating the gradient of the total loss function with respect to the parameters of the preset edge deployment model and updating the parameters in the opposite direction of the gradient, the preset edge deployment model is ensured to gradually approach the optimal solution, thus improving its performance on the target task. Furthermore, by introducing the difference between the parameters of the bootstrap model and the parameters of the preset edge deployment model and smoothing them using exponential moving average (EMA), the preset edge deployment model can gradually absorb the advantages of each bootstrap model while maintaining its own characteristics, thereby improving its generalization ability and robustness.

[0283] This application embodiment acquires a sample dataset and, based on a preset edge deployment model and each guiding model, constructs a corresponding output distribution loss for the differences between the predicted probability distributions output by each sample data in the sample dataset; based on each attention head of each guiding model and edge deployment model, constructs a corresponding attention distillation loss for the differences between the attention matrices output by each sample data; based on each feature extraction level of each guiding model and edge deployment model, constructs a corresponding multi-level feature distillation loss for the differences between the feature maps output by each sample data; based on each guiding model and edge deployment model, constructs a corresponding relation distillation loss for the differences between the output feature similarities of any sample data in the sample dataset; based on the attention distillation loss, multi-level feature distillation loss, and relation distillation loss, constructs a comprehensive loss for each guiding model; and performs knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and multiple comprehensive losses corresponding to multiple guiding models to obtain the target edge deployment model, wherein the multiple guiding models have distinct network architectures. Therefore, by introducing multiple guiding models with distinct network architectures, the problem of a single guiding model having a limited knowledge perspective can be effectively overcome, avoiding homogeneous dependencies during distillation. Simultaneously, by constructing a comprehensive loss function using a multi-layered knowledge distillation mechanism, richer and more diverse knowledge sources and finer-grained supervision signals can be provided. Specifically, multiple distinct guiding models can effectively solve the problem of knowledge homogeneity. Output distribution loss conveys class probability knowledge, attention distillation loss captures fine-grained attention mechanisms within the model, multi-level feature distillation loss ensures consistency in feature extraction layers, and relational distillation loss encodes global structural information between samples. This fusion of multi-layered, multi-source knowledge enables the edge deployment model to learn more robust and generalized representations, reducing the risk of overfitting and improving adaptability and performance on edge devices. In summary, this application can improve the generalization ability and performance of the knowledge-distilled edge deployment model.

[0284] In some implementations, to achieve dynamic adaptation and optimization of heterogeneous models to multimodal data in multi-teacher collaborative distillation, the matching degree between data properties and network architecture can be quantified and loss weights dynamically adjusted to improve the generalization ability and deployment efficiency of lightweight models in complex data scenarios. Step 105, "based on multiple output distribution losses and multiple comprehensive losses corresponding to multiple guiding models, performs knowledge distillation on the preset edge deployment model to obtain the target edge deployment model," may include:

[0285] (105.1) Obtain the data properties corresponding to the sample data, wherein the data properties include at least one of image properties, text properties and multimodal properties;

[0286] (105.2) Obtain the adaptability index of the network architecture of each bootstrap model to the data properties, and determine the fusion weights corresponding to each bootstrap model based on the adaptability index;

[0287] (105.3) Adjust each output distribution loss and each comprehensive loss according to the fusion weights corresponding to each guided model to obtain the corresponding target output distribution loss and target comprehensive loss;

[0288] (105.4) Based on the multiple target output distribution loss and multiple target comprehensive loss corresponding to multiple guidance models, knowledge distillation is performed on the preset edge deployment model to obtain the target edge deployment model.

[0289] Among them, data properties can be the type characteristics of sample data, such as image properties (e.g., pixel distribution), text properties (e.g., word frequency statistics), or multimodal properties (e.g., the correlation between images and text).

[0290] Among them, image properties can be a quantitative description of visual features in the sample data, such as edge texture features or color histogram statistics extracted by convolutional layers.

[0291] Among them, text properties can be a quantitative description of the language features in the sample data, such as word vectors or TF-IDF features extracted by word embedding models.

[0292] Among them, multimodal properties can be a quantitative description of cross-modal association characteristics in sample data, such as the image-text similarity matrix calculated through a cross-modal attention mechanism.

[0293] The adaptability index can be a metric that measures the degree of matching between the network architecture of the guided model and the specific properties of the data, such as a scalar value calculated by the model's validation accuracy or feature consistency score on the corresponding data.

[0294] Among them, the fusion weight can be a dynamic weight coefficient based on the adaptability index normalization.

[0295] The target output distribution loss can be the output distribution loss term adjusted by the fusion weights.

[0296] The target comprehensive loss can be the comprehensive loss term adjusted by the fusion weights.

[0297] For example, data properties corresponding to sample data can be obtained, including at least one of image properties, text properties, and multimodal properties. In specific implementations, these data properties can be obtained by preprocessing and feature extraction of the sample data. For example, for image data, its size, color distribution, texture features, etc., can be obtained; for text data, word frequency, sentence length, grammatical structure, etc., can be statistically analyzed; for multimodal data, the correlation and complementarity between different modalities need to be comprehensively considered.

[0298] In some implementations, a suitability index for the network architecture to the data properties can be obtained for each bootstrapping model. The suitability index reflects the bootstrapping model's ability and effectiveness in processing a specific type of data. The specific calculation formula is as follows:

[0299] ;

[0300] in, This represents the parameter vector of the i-th guided model. Represents the sample dataset, It is an evaluation function used to measure the degree of fit between the bootstrap model and the properties of the data. The evaluation function can be designed according to the actual situation, for example, by cross-validation or by specific metrics (such as accuracy, F1 score, etc.).

[0301] Furthermore, based on the obtained adaptability index, the fusion weight corresponding to each guiding model can be determined. The fusion weight represents the relative importance of each guiding model in the knowledge distillation process. The specific calculation formula is as follows:

[0302] ;

[0303] Where N represents the total number of guided models; This represents the fusion weight of the i-th guiding model. This approach ensures that the guiding model, better suited to the properties of the data, plays a greater role in knowledge transfer.

[0304] In some implementations, an adaptive adjustment mechanism can be introduced to dynamically update the fusion weights. For example, an adaptive adjustment strategy based on gradient descent can monitor the learning progress and performance of the student model in real time, and then adjust the weights of each guiding model to achieve the optimal knowledge transfer effect.

[0305] For example, if the guiding model 1 corresponds to the first weight and comprehensive losses The second weight corresponds to the guiding model 2. and comprehensive losses The third weight corresponding to the guiding model 3 and comprehensive losses The fourth weight corresponds to the guiding model 4. and comprehensive losses So, the overall loss of the target It can be calculated using the following formula:

[0306] ;

[0307] In some implementations, the target output distribution loss can also be obtained by weighted fusion of the output distribution losses corresponding to multiple guiding models through the above distribution. The specific implementation can be referred to the calculation of the target comprehensive loss above, which will not be elaborated here.

[0308] Furthermore, based on the multiple target output distribution losses and multiple target comprehensive losses corresponding to multiple guidance models, knowledge distillation is performed on the preset edge deployment model to obtain the target edge deployment model.

[0309] It is understood that the number and type of guidance models can be selected according to the actual situation, and this application embodiment does not make specific limitations in this regard.

[0310] By adopting the above methods, the priority of knowledge transfer of the guidance model under different data properties can be dynamically adapted to improve the structured knowledge learning ability of the preset edge deployment model for multimodal data, thereby providing technical support for high-precision inference and efficient resource deployment of subsequent edge devices in heterogeneous data scenarios.

[0311] Please see Figure 3 This application also provides a knowledge distillation apparatus that can implement the above-described knowledge distillation method. The knowledge distillation apparatus includes:

[0312] The acquisition module 31 is used to acquire the sample dataset and, based on the preset edge deployment model and each bootstrap model, construct the corresponding output distribution loss for the difference between the predicted probability distributions of each sample data output in the sample dataset.

[0313] The first building module 32 is used to construct the corresponding attention distillation loss based on the differences between the attention matrices output for each sample data, for each attention head of each bootstrap model and edge deployment model.

[0314] The second building module 33 is used to construct a corresponding multi-level feature distillation loss based on the differences between the feature maps output by each sample data, for each feature extraction level of each bootstrap model and edge deployment model.

[0315] The third building module 34 is used to construct the corresponding relation distillation loss based on the differences between the output feature similarities of any sample data in the sample dataset, based on each bootstrap model and edge deployment model.

[0316] Distillation module 35 is used to construct the comprehensive loss of each guided model based on attention distillation loss, multi-level feature distillation loss and relation distillation loss, and to perform knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and multiple comprehensive losses corresponding to multiple guided models to obtain the target edge deployment model. Among them, the multiple guided models have different network architectures.

[0317] The specific implementation of this knowledge distillation apparatus is basically the same as the specific embodiment of the knowledge distillation method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this application, the knowledge distillation apparatus may also be equipped with other functional modules to implement the knowledge distillation method described above.

[0318] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described knowledge distillation method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0319] Please see Figure 4 , Figure 4 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0320] The processor 41 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0321] The memory 42 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 42 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 42 and is called and executed by the processor 41 using the knowledge distillation method of the embodiments of this application.

[0322] Input / output interface 43 is used to implement information input and output;

[0323] The communication interface 44 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0324] Bus 45 transmits information between various components of the device (e.g., processor 41, memory 42, input / output interface 43, and communication interface 44);

[0325] The processor 41, memory 42, input / output interface 43 and communication interface 44 are connected to each other within the device via bus 45.

[0326] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described knowledge distillation method.

[0327] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0328] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0329] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0330] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0331] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0332] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0333] It should be understood that in this application, "at least one" and "several" refer to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0334] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0335] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0336] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0337] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0338] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A knowledge distillation method, characterized in that, The method includes: Obtain a sample dataset, and based on the preset edge deployment model and each bootstrap model, construct the corresponding output distribution loss for the difference between the predicted probability distributions of each sample data output in the sample dataset; Based on each attention head of each guiding model and each edge deployment model, a corresponding attention distillation loss is constructed for the differences between the attention matrices output by each sample data. Based on each feature extraction level of each guiding model and the edge deployment model, a corresponding multi-level feature distillation loss is constructed for the differences between the feature maps output by each sample data. Based on each of the guiding models and the edge deployment models, a corresponding relation distillation loss is constructed for the difference between the output feature similarities of any sample data pairs in the sample dataset. Based on the attention distillation loss, the multi-level feature distillation loss, and the relation distillation loss, a comprehensive loss for each guided model is constructed, and the data properties corresponding to the sample data are obtained, wherein the data properties include at least one of image properties, text properties, and multimodal properties; the adaptability index of the network architecture of each guided model to the data properties is obtained, and the fusion weights corresponding to each guided model are determined based on the adaptability index; according to the fusion weights corresponding to each guided model, each output distribution loss and each comprehensive loss are adjusted to obtain the corresponding target output distribution loss and target comprehensive loss; based on the multiple target output distribution losses and multiple target comprehensive losses corresponding to multiple guided models, knowledge distillation is performed on the preset edge deployment model to obtain the target edge deployment model, wherein the multiple guided models have distinct network architectures.

2. The knowledge distillation method according to claim 1, characterized in that, The method, based on the preset edge deployment model and each bootstrap model, constructs a corresponding output distribution loss for the difference between the predicted probability distributions output by each sample data in the sample dataset, including: Obtain preset adjustment parameters and temperature hyperparameters, wherein the temperature hyperparameters are used to soften the prediction probability distribution of each guided model; Obtain the first predicted probability distribution output by each guided model for each sample data in the sample dataset, and the second predicted probability distribution output by the preset edge deployment model for each sample data, and determine the corresponding distribution divergence value based on the first predicted probability distribution and the second predicted probability distribution; The first output distribution loss sub-item is determined based on the product of the adjustment parameter, the temperature hyperparameter, and the distribution divergence value. Obtain the classification loss corresponding to the preset edge deployment model, and adjust the classification loss based on the adjustment parameters to obtain the second output distribution loss sub-item; Based on the first output distribution loss sub-item and the second output distribution loss sub-item, the output distribution loss between each guiding model and the edge deployment model is constructed.

3. The knowledge distillation method according to claim 1, characterized in that, For each attention head based on each guiding model and the edge deployment model, a corresponding attention distillation loss is constructed for the difference between the attention matrices output for each sample data, including: Based on each first attention head of each guided model, output the first attention matrix corresponding to each sample data, and based on each second attention head of the edge deployment model, output the second attention matrix corresponding to each sample data; Perform singular value decomposition on the second attention matrix to obtain the target second attention matrix; Based on the difference between the first attention matrix and the corresponding target second attention matrix, a corresponding attention distillation loss is constructed; Based on multiple attention distillation sub-losses corresponding to multiple first attention heads, an attention distillation loss is constructed between each guiding model and the edge deployment model.

4. The knowledge distillation method according to claim 1, characterized in that, Based on each feature extraction layer of each guiding model and the edge deployment model, and considering the differences between the feature maps output by each sample data, a corresponding multi-level feature distillation loss is constructed, including: Based on each first feature extraction level of each guided model, a first feature map corresponding to each sample data is output, and based on each second feature extraction level of the edge deployment model, a second feature map corresponding to each sample data is output; Based on the first feature map, the number of channels and spatial size of the second feature map are adjusted to obtain an updated second feature map; The first feature map is subjected to norm normalization to obtain the target first feature map, and the updated second feature map is subjected to norm normalization to obtain the target second feature map. The target difference value between the first feature map of the target and the corresponding second feature map of the target is calculated using a distance metric function. Obtain the weight coefficient of each feature extraction level, and based on the product of the weight coefficient and the target difference value, obtain the hierarchical feature distillation loss corresponding to each feature extraction level; Based on the sum of multiple feature distillation losses corresponding to multiple feature extraction levels, a multi-level feature distillation loss is constructed between each guiding model and the edge deployment model.

5. The knowledge distillation method according to claim 4, characterized in that, The step of obtaining the weight coefficients for each feature extraction level includes: For each of the first feature extraction levels, a global average pooling operation is performed on the corresponding first feature map to obtain a first vector; The sum of multiple first vectors corresponding to multiple first feature extraction levels is obtained to obtain the total target vector; Based on the ratio of the sum of the first vector and the target vector, the weight coefficient of each feature extraction level is obtained.

6. The knowledge distillation method according to claim 1, characterized in that, Based on each guiding model and the edge deployment model, a corresponding relation distillation loss is constructed for the difference in output feature similarity between any sample data pairs in the sample dataset, including: For each guided model, extract the first feature vector of the first sample data and the second feature vector of the second sample data contained in any sample data pair; Using the edge deployment model, the third feature vector corresponding to the first sample data and the fourth feature vector corresponding to the second sample data are extracted. Obtain the first feature similarity between the first feature vector and the second feature vector, and determine the first ratio based on the ratio of the first feature similarity to the first temperature parameter corresponding to each guided model; Obtain the second feature similarity between the third feature vector and the fourth feature vector, and determine the second ratio based on the ratio of the second feature similarity to the second temperature parameter corresponding to the edge deployment model; The first ratio is normalized to obtain the target first ratio, and the second ratio is normalized to obtain the target second ratio. Based on the difference between the first target ratio and the second target ratio, the relational distillation loss of the arbitrary sample data pair is obtained; Based on multiple relational distillation losses of multiple sample data pairs corresponding to the sample dataset, a relational distillation loss between each bootstrap model and the edge deployment model is constructed.

7. A knowledge distillation apparatus, characterized in that, The device includes: The acquisition module is used to acquire a sample dataset and, based on the preset edge deployment model and each bootstrap model, construct a corresponding output distribution loss for the difference between the predicted probability distributions of each sample data output in the sample dataset. The first construction module is used to construct a corresponding attention distillation loss based on the differences between the attention matrices output by each sample data, for each attention head of each guiding model and each edge deployment model. The second construction module is used to construct a corresponding multi-level feature distillation loss based on the differences between the feature maps output by each sample data, according to each feature extraction level of each guiding model and the edge deployment model. The third construction module is used to construct a corresponding relation distillation loss based on the differences in the output feature similarity between any sample data pairs in the sample dataset, based on each guiding model and the edge deployment model. The distillation module is used to construct the comprehensive loss of each guided model based on the attention distillation loss, the multi-level feature distillation loss, and the relation distillation loss, and to obtain the data properties corresponding to the sample data, wherein the data properties include at least one of image properties, text properties, and multimodal properties; to obtain the adaptability index of the network architecture of each guided model to the data properties, and to determine the fusion weights corresponding to each guided model based on the adaptability index; to adjust each output distribution loss and each comprehensive loss according to the fusion weights corresponding to each guided model to obtain the corresponding target output distribution loss and target comprehensive loss; and to perform knowledge distillation on the preset edge deployment model according to the multiple target output distribution losses and multiple target comprehensive losses corresponding to multiple guided models to obtain the target edge deployment model, wherein the multiple guided models have different network architectures.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the knowledge distillation method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the knowledge distillation method according to any one of claims 1 to 6.

Images