Knowledge distillation method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention provides a knowledge distillation method and device, computer equipment and a readable storage medium. Comprising the following steps: based on a preset edge deployment model and a guide model, constructing output distribution loss for a prediction probability distribution difference output by sample data in a sample data set; constructing attention distillation loss based on attention matrix differences output by the guidance model and the edge deployment model; based on the guide model and the edge deployment model, feature map differences of hierarchical output are extracted to construct multi-level feature distillation loss; on the basis of feature similarity differences output by the guide model and the edge deployment model, constructing relational distillation loss; and based on the attention distillation loss, the multi-level characteristic distillation loss and the relation distillation loss, constructing a comprehensive loss, and according to a plurality of output distribution losses and a plurality of comprehensive losses corresponding to the plurality of guide models, performing knowledge distillation on a preset edge deployment model to obtain a target edge deployment model. Therefore, the generalization ability and performance of the edge deployment model can be improved.

Description

Technical Field

[0001] The present application relates to the field of knowledge distillation technology, and in particular to a knowledge distillation method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] Knowledge Distillation (KD) is a model compression and knowledge transfer technology. Its core idea is to transfer the knowledge contained in a trained, complex, and powerful "guide model" to a "edge deployment model" with a simpler structure and less computational effort. This allows the edge deployment model to fit the real categories while also learning the guide model's fine-grained understanding of the similarities between categories.

[0003] In related technologies, the commonly used knowledge distillation solution is single-teacher knowledge distillation combined with model compression technology. Specifically, this solution usually selects a specific large pre-trained model as the guide model. During the distillation phase, the label probability distribution generated by the guide model during the training phase is used as an additional supervisory signal and jointly optimized with the hard label loss of the edge deployment model itself to achieve knowledge transfer. However, this knowledge distillation method of a single guide model relies solely on knowledge transfer from a single source, resulting in the singleness and homogeneity of the knowledge dimension. This makes it difficult to meet the requirements of edge devices for model diversity and adaptability, resulting in insufficient generalization and poor model performance in the distilled edge deployment model. Summary of the Invention

[0004] This application proposes a knowledge distillation method, apparatus, computer equipment and readable storage medium, which can improve the generalization ability of the edge deployment model after knowledge distillation and improve the performance of the edge deployment model.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a knowledge distillation method, the method comprising: Obtain a sample data set, and construct a corresponding output distribution loss based on the difference between the predicted probability distributions output by each sample data in the sample data set based on the preset edge deployment model and each guided model; Based on each attention head of each guide model and each attention head of the edge deployment model, constructing a corresponding attention distillation loss for the difference between the attention matrices output by each sample data; Based on each feature extraction level of each guided model and the edge deployment model, constructing a corresponding multi-level feature distillation loss for the difference between the feature maps output by each sample data; Based on each of the guide models and the edge deployment model, constructing a corresponding relational distillation loss for the difference between the feature similarities output by any pair of sample data in the sample data set; Based on the attention distillation loss, the multi-level feature distillation loss and the relationship distillation loss, a comprehensive loss for each guided model is constructed, and according to the multiple output distribution losses and multiple comprehensive losses corresponding to the multiple guided models, knowledge distillation is performed on the preset edge deployment model to obtain a target edge deployment model, wherein the multiple guided models have different network architectures.

[0006] Accordingly, a second aspect of the embodiments of the present application provides a knowledge distillation device, comprising: An acquisition module is configured to acquire a sample data set and construct a corresponding output distribution loss based on the difference between the predicted probability distributions output by each sample data in the sample data set based on a preset edge deployment model and each guided model; A first construction module is configured to construct a corresponding attention distillation loss based on the difference between the attention matrices output by each guide model and each attention head of the edge deployment model for each sample data; A second construction module is configured to construct a corresponding multi-level feature distillation loss based on the difference between the feature maps output by each sample data and each feature extraction level of each guide model and the edge deployment model; A third construction module is configured to construct a corresponding relational distillation loss based on the difference between the feature similarities output by any pair of sample data in the sample data set based on each of the guide models and the edge deployment model; A distillation module is used to construct a comprehensive loss for each guided model based on the attention distillation loss, the multi-level feature distillation loss, and the relationship distillation loss, and perform knowledge distillation on the preset edge deployment model according to multiple output distribution losses and multiple comprehensive losses corresponding to multiple guided models to obtain a target edge deployment model, wherein the multiple guided models have different network architectures.

[0007] In some embodiments, the distillation module is further used to: Acquiring data properties corresponding to the sample data, wherein the data properties include at least one of image properties, text properties, and multimodal properties; Obtaining an adaptability index of the network architecture of each guiding model to the data properties, and determining a fusion weight corresponding to each guiding model based on the adaptability index; Adjusting each output distribution loss and each comprehensive loss according to the fusion weight corresponding to each guided model to obtain the corresponding target output distribution loss and target comprehensive loss; According to the multiple target output distribution losses and the multiple target comprehensive losses corresponding to the multiple guidance models, knowledge distillation is performed on the preset edge deployment model to obtain a target edge deployment model.

[0008] In some embodiments, the acquisition module is further configured to: Obtaining preset adjustment parameters and temperature hyperparameters, wherein the temperature hyperparameters are used to soften the predicted probability distribution of each guided model; Obtaining a first predicted probability distribution output by each guidance model for each sample data in the sample data set, and a second predicted probability distribution output by a preset edge deployment model for each sample data, and determining corresponding distribution divergence values ​​based on the first predicted probability distribution and the second predicted probability distribution; determining a first output distribution loss sub-item based on a product of the adjustment parameter, the temperature hyperparameter, and the distribution divergence value; Obtaining a classification loss corresponding to the preset edge deployment model, and adjusting the classification loss based on the adjustment parameter to obtain a second output distribution loss sub-item; Based on the first output distribution loss sub-item and the second output distribution loss sub-item, an output distribution loss between each guided model and the edge deployment model is constructed.

[0009] In some embodiments, the first building block is further configured to: Based on each first attention head of each guide model, output a first attention matrix corresponding to each sample data, and based on each second attention head of the edge deployment model, output a second attention matrix corresponding to each sample data; Performing a singular value decomposition operation on the second attention matrix to obtain a target second attention matrix; Based on the difference between the first attention matrix and the corresponding target second attention matrix, construct a corresponding attention distillation sub-loss; Based on multiple attention distillation sub-losses corresponding to multiple first attention heads, an attention distillation loss is constructed between each guided model and the edge deployment model.

[0010] In some embodiments, the second building block is further configured to: Outputting a first feature map corresponding to each sample data based on each first feature extraction level of each guide model, and outputting a second feature map corresponding to each sample data based on each second feature extraction level of the edge deployment model; Based on the first feature map, adjusting the number of channels and the spatial size of the second feature map to obtain an updated second feature map; performing norm normalization processing on the first feature map to obtain a target first feature map, and performing norm normalization processing on the updated second feature map to obtain a target second feature map; Calculating a target difference value between the target first feature map and the corresponding target second feature map by using a distance metric function; Obtaining a weight coefficient for each feature extraction level, and obtaining a level feature distillation sub-loss corresponding to each feature extraction level based on a product of the weight coefficient and the target difference value; Based on the sum of multiple-level feature distillation sub-losses corresponding to multiple feature extraction levels, a multi-level feature distillation loss is constructed between each guided model and the edge deployment model.

[0011] In some embodiments, the second building block is further configured to: For each first feature extraction level, performing a global average pooling operation on the corresponding first feature map to obtain a first vector; Obtaining the sum of multiple first vectors corresponding to multiple first feature extraction levels to obtain a target vector sum; Based on the ratio of the first vector to the sum of the target vectors, a corresponding weight coefficient of each feature extraction level is obtained.

[0012] In some embodiments, the third building block is further configured to: Extracting a first eigenvector of the first sample data and a second eigenvector of the second sample data included in any sample data pair through each guided model; Extracting, by using the edge deployment model, a third eigenvector corresponding to the first sample data and a fourth eigenvector corresponding to the second sample data; Obtaining a first feature similarity between the first feature vector and the second feature vector, and determining a first ratio based on the first feature similarity and a ratio of a first temperature parameter corresponding to each of the guided models; Obtaining a second feature similarity between the third eigenvector and the fourth eigenvector, and determining a second ratio based on the second feature similarity and a ratio of a second temperature parameter corresponding to the edge deployment model; Normalizing the first ratio to obtain a target first ratio, and normalizing the second ratio to obtain a target second ratio; Obtaining a relational distillation sub-loss for the arbitrary sample data pair based on a difference between the target first ratio and the target second ratio; Based on multiple relational distillation sub-losses of multiple sample data pairs corresponding to the sample data set, a relational distillation loss between each guided model and the edge deployment model is constructed.

[0013] Correspondingly, the third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the knowledge distillation method of any one of the embodiments of the first aspect of the present application.

[0014] Correspondingly, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the knowledge distillation method of any one of the embodiments of the first aspect of the present application.

[0015] The embodiment of the present application obtains a sample data set, and based on the preset edge deployment model and each guiding model, constructs a corresponding output distribution loss for the difference between the predicted probability distributions output by each sample data in the sample data set; based on each attention head of each guiding model and the edge deployment model, constructs a corresponding attention distillation loss for the difference between the attention matrices output by each sample data; based on each feature extraction level of each guiding model and the edge deployment model, constructs a corresponding multi-level feature distillation loss for the difference between the feature maps output by each sample data; based on each guiding model and the edge deployment model, constructs a corresponding relational distillation loss for the difference between the feature similarities output by any pair of sample data in the sample data set; based on the attention distillation loss, the multi-level feature distillation loss and the relational distillation loss, constructs a comprehensive loss for each guiding model, and performs knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and the multiple comprehensive losses corresponding to the multiple guiding models to obtain a target edge deployment model, wherein the multiple guiding models have different network architectures. In this way, by introducing multiple guidance models with different network architectures, it is possible to effectively overcome the problem of a single knowledge perspective of a single guidance model and avoid the homogeneity dependence generated during the distillation process; at the same time, combining a multi-level knowledge distillation mechanism to construct a comprehensive loss function can provide richer and more diverse sources of knowledge and finer-grained supervision signals. Specifically, multiple different guidance models can effectively solve the problem of knowledge homogeneity, transfer category probability knowledge through output distribution loss, capture the fine-grained attention mechanism within the model, multi-level feature distillation loss ensures the consistency of the feature extraction layer, and relational distillation loss encodes the global structural information between samples. This fusion of multi-level and multi-source knowledge enables the edge deployment model to learn more robust and generalized representations, reduces the risk of overfitting, and improves adaptability and performance on edge devices. In summary, the present application can improve the generalization ability of the edge deployment model after knowledge distillation and improve the performance of the edge deployment model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the architecture of the knowledge distillation system provided in an embodiment of the present application; Figure 2 This is a flow chart of the knowledge distillation method provided in an embodiment of the present application; Figure 3 This is a schematic diagram of the functional modules of the knowledge distillation device provided in an embodiment of the present application; Figure 4 This is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0018] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0020] Knowledge Distillation (KD) is a model compression and knowledge transfer technology. Its core idea is to transfer the knowledge contained in a trained, complex, and powerful "guide model" to a "edge deployment model" with a simpler structure and less computational effort. This allows the edge deployment model to fit the real categories while also learning the guide model's fine-grained understanding of the similarities between categories.

[0021] In related technologies, the commonly used knowledge distillation solution is single-teacher knowledge distillation combined with model compression technology. Specifically, this solution usually selects a specific large pre-trained model as the guide model. During the distillation phase, the label probability distribution generated by the guide model during the training phase is used as an additional supervisory signal and jointly optimized with the hard label loss of the edge deployment model itself to achieve knowledge transfer. However, this knowledge distillation method of a single guide model relies solely on knowledge transfer from a single source, resulting in the singleness and homogeneity of the knowledge dimension. This makes it difficult to meet the requirements of edge devices for model diversity and adaptability, resulting in insufficient generalization and poor model performance in the distilled edge deployment model.

[0022] Based on this, the embodiments of the present application provide a knowledge distillation method, apparatus, computer equipment and readable storage medium, which can improve the generalization ability of the edge deployment model after knowledge distillation and improve the performance of the edge deployment model.

[0023] The knowledge distillation method, apparatus, computer equipment, and readable storage medium provided in the embodiments of the present application are specifically illustrated through the following examples. First, the knowledge distillation system in the embodiments of the present application is described.

[0024] Please refer to Figure 1 In some implementations, an embodiment of the present application provides a knowledge distillation system, including a terminal 11 and a server 12 .

[0025] Exemplarily, terminal 11 can be a smartphone, tablet, smart camera, edge computing device, etc. Terminal 11 can be responsible for deploying a lightweight pre-defined edge deployment model and performing local real-time inference tasks, such as object detection and classification. Terminal 11 can also collect local input data (e.g., images, text) and perform basic preprocessing operations, such as normalization and cropping, to ensure that the data is suitable for the input requirements of the pre-defined edge deployment model. Furthermore, terminal 11 can also record inference latency and accuracy metrics and feed this performance data back to server 12 via the network, so that underperforming models can be adjusted and optimized in a timely manner.

[0026] Furthermore, server 12 can be a high-performance computer cluster, cloud server, or other hardware facility with powerful computing capabilities, responsible for running multiple guidance models, executing knowledge distillation algorithms, and optimizing models. Server 12 can run multiple heterogeneous guidance models (e.g., CNN, Transformer architectures) and adjust the weights of each guidance model in real time through a dynamic attention fusion mechanism, providing complementary knowledge sources. Furthermore, server 12 can also execute complex and time-consuming knowledge distillation algorithms, utilize structured knowledge transfer strategies to generate or update lightweight, pre-configured edge deployment models, and distribute the optimized lightweight models to terminal 11 via the network for deployment and use.

[0027] The knowledge distillation method in the embodiments of the present application can be illustrated by the following examples.

[0028] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0029] In some implementations, the knowledge distillation method can be applied to scenarios where efficient, high-performance AI models (pre-defined edge deployment models) are deployed in resource-constrained environments. Specific application scenarios include, but are not limited to, smart mobile devices, Internet of Things (IoT) edge devices, self-driving cars, and some industrial automation scenarios.

[0030] In the embodiment of the present application, the description will be made from the perspective of a knowledge distillation device, which can be integrated into a computer device. Figure 2 , Figure 2 This is a flowchart of the steps of the knowledge distillation method provided in an embodiment of the present application. In this embodiment of the present application, the knowledge distillation device is specifically integrated into a terminal or server as an example. When the processor on the terminal or server executes the program instructions corresponding to the knowledge distillation method, the specific process is as follows: Step 101: Obtain a sample data set, and construct a corresponding output distribution loss based on the difference between the predicted probability distributions output by each sample data in the sample data set based on a preset edge deployment model and each boot model.

[0031] In some embodiments, in order to improve the accuracy and generalization ability of the lightweight model (preset edge deployment model) in the edge deployment scenario, a sample data set can be obtained, and based on the preset edge deployment model and multiple heterogeneous guidance models, an output distribution loss can be constructed for the difference in the predicted probability distribution output of each sample data, so that the student model can efficiently learn the soft label knowledge of the teacher model, thereby reducing the accuracy loss caused by model compression and enhancing dynamic adaptability.

[0032] The sample dataset may be a data set containing input samples required for training and their annotation information, such as a set of labeled images in an image classification task.

[0033] Among them, the preset edge deployment model can be a lightweight model designed for edge device resource constraints, that is, a student model, such as MobileNet or SqueezeNet.

[0034] The guidance model can be a teacher model. Because guidance models are often designed to be large and complex to achieve high accuracy and performance, they have high computing and storage requirements. For example, they can be deployed in the cloud or in enterprise-level data centers. Specifically, in image classification tasks, the guidance model can be a large convolutional neural network such as ResNet, DenseNet, or Inception. In natural language processing tasks, it can be a large-scale pre-trained model such as BERT or Transformer. The specific guidance model is determined based on the specific task.

[0035] The sample data can be a single input instance and its corresponding label in the sample data set, such as an image to be classified.

[0036] The predicted probability distribution is the set of probabilities of belonging to each category after the model predicts the input sample data. Specifically, if there is a classification problem with n categories, the predicted probability distribution is a vector of length n, where each element represents the probability of the correct prediction for that category.

[0037] Among them, output distribution loss can be a metric to measure the difference in prediction distribution between the student model and the teacher model.

[0038] For example, for image classification tasks, sample datasets can be obtained from public image datasets such as ImageNet and CIFAR-10. For natural language processing tasks, datasets such as the Wikipedia corpus and the IMDB movie review dataset can be obtained. For specific needs, web crawlers can also be written to automatically capture data from the internet as sample datasets. The method for obtaining sample datasets can be determined based on actual circumstances.

[0039] In some embodiments, in order to calculate the output distribution loss, a sample data set can be first obtained, and each sample data in the sample data set can be predicted using a preset edge deployment model and each guide model to obtain their respective predicted probability distributions. Then, the output distribution loss is constructed by comparing the differences between these predicted probability distributions, and the output distribution loss can use KL divergence as a measurement standard. For example, in an image classification task, for a sample image a in the sample data set corresponding to the task, the guide model can predict that the probability of it belonging to three sample categories is [0.7, 0.2, 0.1], while the predicted probability of the preset edge deployment model is [0.6, 0.3, 0.1]. Afterwards, the difference between the two can be quantified by calculating the KL divergence between the two probability distributions, thereby obtaining the output distribution loss, which is used to characterize the gap in knowledge transfer between the preset edge deployment model and the guide model. In this way, it can be convenient to help optimize the lightweight edge deployment model in the future, so that it can learn the knowledge of a larger and more complex guide model and improve model performance.

[0040] Through the above methods, the migration of multi-dimensional knowledge from the teacher model (guidance model) to the student model (pre-set edge deployment model) can be effectively realized, reducing the accuracy loss during edge deployment, so as to improve the generalization ability and dynamic adaptability of the lightweight pre-set edge deployment model in resource-constrained environments.

[0041] In some embodiments, in order to achieve high-precision knowledge transfer and resource constraint adaptation of the preset edge deployment model of the edge device, the softening degree of the prediction distribution of the guidance model can be optimized by introducing a temperature hyperparameter, and the weight of the output distribution loss sub-item can be dynamically adjusted in combination with the adjustment parameter to achieve the coordinated optimization of the knowledge fusion efficiency and model compression effect in multi-teacher collaborative distillation. For example, in step 101, "based on the preset edge deployment model and each guidance model, for each sample data in the sample data set, the difference between the predicted probability distribution is constructed to construct the corresponding output distribution loss" can include: (101.1) obtaining preset tuning parameters and temperature hyperparameters, wherein the temperature hyperparameter is used to soften the predicted probability distribution of each bootstrap model; (101.2) Obtain a first predicted probability distribution output by each guidance model for each sample data in the sample data set, and a second predicted probability distribution output by the preset edge deployment model for each sample data, and determine corresponding distribution divergence values ​​based on the first predicted probability distribution and the second predicted probability distribution; (101.3) determining a first output distribution loss sub-term based on a product of a tuning parameter, a temperature hyperparameter, and a distribution divergence value; (101.4) Obtaining a classification loss corresponding to the preset edge deployment model, and adjusting the classification loss based on the adjustment parameter to obtain a second output distribution loss sub-item; (101.5) Construct an output distribution loss between each guided model and the edge deployment model based on the first output distribution loss sub-term and the second output distribution loss sub-term.

[0042] Among them, the adjustment parameter can be a scalar coefficient used to control the weight ratio between the output distribution loss sub-item and the classification loss, which can be a value determined by grid search or Bayesian optimization, and can be used express.

[0043] Among them, the temperature hyperparameter can be used to adjust the predicted probability distribution of each guided model, making its output probability distribution smoother and facilitating the student model to learn more information. It means that it can be set by analyzing experimental data, for example, to 2.0 or 3.0.

[0044] The first predicted probability distribution can be the predicted label distribution output by the guidance model for the sample data, which can be used express.

[0045] The second predicted probability distribution can be the predicted label distribution output by the preset edge deployment model for the same sample data as the guidance model, which can be expressed as express.

[0046] The distribution divergence value can be the KL divergence that measures the difference between the first predicted probability distribution and the second predicted probability distribution. .

[0047] Among them, the first output distribution loss sub-item can be the temperature hyperparameter , adjust parameters The product term with the distribution divergence value can be expressed as .

[0048] Among them, the classification loss can be the original cross entropy loss of the preset edge deployment model for the sample data, which can be used express.

[0049] Among them, the second output distribution loss sub-item can be the product term of the adjustment parameter and the classification loss, which can be expressed as express.

[0050] Specifically, we can first obtain the temperature hyperparameters , It can be introduced into the Softmax function. The specific process is as follows: ; in, Represents the data output by the Softmax layer of the model for the i-th category, Indicates traversal and summation of the output data of all categories, where j is the summation index.

[0051] In some embodiments, the present application can determine the temperature hyperparameter through an adaptive mechanism. Specifically, the temperature hyperparameter T can be dynamically adjusted by utilizing the uncertainty of the model for sample data during the training process. For example, the temperature value at the current moment can be determined by calculating the entropy (representing uncertainty) of the predicted probability distribution of each sample data; or, for example, it can be obtained through cross-validation and experimental tuning on the validation set. For example, during the training process, a set of candidate T values ​​(such as T=[0.5, 1.0, 2.0, 5.0]) is set, and the output distribution loss and model accuracy corresponding to different T values ​​are evaluated on the validation set through multiple iterations, and finally the value that minimizes the validation loss is selected as the temperature hyperparameter. In this way, by setting the temperature hyperparameter, it is ensured that when the model is relatively certain about certain sample data, the value of the temperature hyperparameter is low, making the probability distribution sharper (emphasizing high confidence categories); and when the model is relatively uncertain about certain sample data, the value of the temperature hyperparameter is high, making the probability distribution smoother (retaining small probability information), thereby helping the preset edge deployment model to better learn the knowledge that guides the model.

[0052] For example, for each sample data , first obtain the original first predicted probability distribution output by the bootstrap model , and then calculate its entropy : ; Where n represents the number of categories, Represents the original first predicted probability distribution of the bootstrap model for the i-th sample data belonging to the j-th category.

[0053] After that, the average entropy of the predicted probability distribution of all sample data can be calculated : ; Where N represents the size of the sample dataset.

[0054] Furthermore, based on the above average entropy, the temperature hyperparameter T at the current moment can be dynamically calculated: ; in, and It is a preset hyperparameter used to control the range and sensitivity of temperature values; Is a target entropy value that can be set based on experience or experiment to represent the desired level of uncertainty. The temperature hyperparameter is dynamically determined by the above formula, which can be near When , the temperature hyperparameter approaches , remain relatively stable, when keep away , the temperature hyperparameter increases or decreases accordingly to adjust the smoothness of the probability distribution.

[0055] Furthermore, for each sample data in the sample data set, the first predicted probability distribution of each guided model output pair can be obtained respectively. and the second predicted probability distribution output by the preset edge deployment model Among them, the first prediction probability distribution For each boot model, each sample data in the input sample data set is passed through the forward propagation process of the model to obtain the original first prediction probability distribution , then, using the formula The final first prediction probability distribution is calculated; similarly, for the preset edge deployment model, each sample data in the sample data set is also input, and after the forward propagation process of the model, the original second prediction probability distribution is obtained. , then use the formula The final second predicted probability distribution is calculated.

[0056] In some embodiments, the corresponding distribution divergence value, that is, the KL divergence, can be calculated based on the first predicted probability distribution and the second predicted probability distribution to measure the difference between the two probability distributions. The specific calculation formula is: ; Where n represents the number of categories; represents the first predicted probability distribution output for the i-th category, represents the second predicted probability distribution output for the i-th category.

[0057] Furthermore, based on the preset adjustment parameters , temperature hyperparameter And the product of the above distribution divergence values, the first output distribution loss term can be calculated: In this way, the degree of knowledge distillation can be flexibly controlled while softening the probability distribution.

[0058] Furthermore, the classification loss corresponding to the preset edge deployment model can be obtained , this loss is also called cross entropy loss, which is used to measure the difference between the model prediction result and the true label. The calculation formula is as follows: ; in, represents the true label of the sample data, Represents the predicted probability of the preset edge deployment model for the i-th category. Then, based on the adjustment parameters Adjust the classification loss to obtain the second output distribution loss sub-item: .

[0059] Finally, based on the first output distribution loss sub-item and the second output distribution loss sub-item calculated above, the final output distribution loss between each guided model and the edge deployment model is constructed. : ; By minimizing the output distribution loss, the parameters of the preset edge deployment model can be effectively optimized so that it can be as close to the performance of the guided model as possible while maintaining low computing and storage requirements, thereby achieving efficient knowledge transfer.

[0060] Through the above methods, we can not only extract high-order semantic knowledge from complex guidance models, but also ensure the accuracy of learning real labels by the preset edge deployment model, thereby achieving a balance between accuracy and efficiency in scenarios such as model compression and edge deployment, and has broad application prospects.

[0061] In step 102, based on the difference between the attention matrices output by each guide model and each attention head of the edge deployment model for each sample data, a corresponding attention distillation loss is constructed.

[0062] In some embodiments, in order to achieve efficient knowledge transfer of the attention mechanism in multi-teacher collaborative distillation, the attention matrix difference loss term of the heterogeneous teacher model and the edge deployment model can be constructed to dynamically optimize the structured knowledge transfer path and improve the generalization ability and deployment adaptability of the lightweight model in edge scenarios.

[0063] Among them, the attention head can be an independent feature extraction unit calculated in parallel in the multi-head attention mechanism, such as each independent Query-Key-Value mapping module in the Transformer architecture.

[0064] The attention matrix can be an input sequence position correlation weight matrix calculated by the attention head.

[0065] Among them, the attention distillation loss can be a loss term that measures the difference between the attention matrices of the teacher model and the student model.

[0066] In some embodiments, the attention matrix corresponding to each attention head of the guidance model is It can be generated by the following multi-head attention formula: ; in, is the query weight matrix that guides the model, is the key weight matrix for the bootstrap model, is the dimension scaling factor.

[0067] Furthermore, the attention matrix of the preset edge deployment model Also based on multi-head attention generation, but additionally through low-rank singular value decomposition (SVD) projection ( ) Compress the matrix dimension and reduce the computational complexity: ; in, The attention matrix corresponding to each attention head of the guided model; The attention matrix for the preset edge deployment model.

[0068] Afterwards, the difference between the attention matrix of the guided model and the preset edge deployment model can be calculated by the Frobenius norm: ; Finally, the loss of all attention heads (a total of M) Accumulate and get the final attention distillation loss: ; In some implementations, the attention graph can be viewed as graph structure data, and the topological relationship of the attention graph can be modeled by GNN to improve the ability to capture long-distance dependencies. Specifically, the attention matrix of each guiding model and the preset edge deployment model can be and It is regarded as an adjacency matrix of a graph, where nodes represent feature dimensions and edge weights represent attention strength. Then, a graph convolutional network (GCN) or a graph attention network (GAT) is used to extract the topological features of the attention graph to model the topological relationship of the attention graph, that is, for all neighbors k of node i, its features are calculated. Through the weight matrix The linear transformation result is: ; in, Represents the result after the feature of node i is updated by graph convolution; is the activation function, is the neighbor set of node i, are learnable weights, It is the information input of neighbor nodes and participates in the feature update of the current node i; is the set associated with parameter k The number of elements in .

[0069] Furthermore, the first graph embedding of the guided model can be compared and the second graph embedding of the preset edge deployment model The difference between them is used to construct the topological consistency loss: ; in, Indicates the use of the guided model , through multiple layers Calculated node embeddings; For users using the preset edge deployment model , also through multiple layers Computed node embeddings.

[0070] In some embodiments, the above-mentioned topological consistency loss can be used to replace the attention distillation loss to measure the difference in attention values ​​while also capturing the global structural information of the attention graph (such as connectivity and clustering), thereby improving the ability to transfer complex semantic structures.

[0071] In some embodiments, a second attention matrix corresponding to the model can be deployed at a preset edge In the , dynamic sparsity constraint is introduced, and only the first attention matrix of the guiding model is retained Highly relevant attention heads, thereby improving the efficiency and accuracy of knowledge transfer; and the first attention matrix of the guiding model Highly relevant attention heads can be determined based on the data type (text, image, time series, etc.) and task characteristics (classification, detection, generation, etc.). Different data types have different sensitivities to the relevance of attention heads, and different tasks have significantly different requirements for attention heads. For example, for image data, the Structural Similarity Index Measure (SSIM) and the local-global ratio can be combined to dynamically allocate sparsity (for example, retaining 20% ​​of the global heads and 30% of the local heads out of 16 heads). For time series, dynamic time warping (DTW) can be used to align key timing nodes and retain the corresponding attention heads.

[0072] Through the above method, the multi-scale attention feature expressions of each heterogeneous guiding model can be integrated, and the dynamic attention distillation loss can be used to enhance the preset edge deployment model's ability to focus on key features, so as to improve the efficiency of structured knowledge transfer and reduce the risk of accuracy degradation of the preset edge deployment model, thereby providing technical support for the robust reasoning of edge models in subsequent complex scenarios.

[0073] In some embodiments, in order to achieve structured knowledge transfer of the attention mechanism in multi-teacher collaborative distillation, a difference loss term of the attention matrix between the heterogeneous teacher model and the edge deployment model can be constructed, and singular value decomposition can be introduced to optimize the expression ability of the student model attention matrix, thereby improving the lightweight model's attention to key features and generalization performance, thereby reducing the risk of accuracy loss in edge scenarios. For example, step 102 may include: (102.1) Based on each first attention head of each guiding model, output a first attention matrix corresponding to each sample data, and based on each second attention head of the edge deployment model, output a second attention matrix corresponding to each sample data; (102.2) Performing a singular value decomposition operation on the second attention matrix to obtain a target second attention matrix; (102.3) Based on the difference between the first attention matrix and the corresponding target second attention matrix, construct the corresponding attention distillation sub-loss; (102.4) Based on multiple attention distillation sub-losses corresponding to multiple first attention heads, construct an attention distillation loss between each guided model and the edge deployment model.

[0074] Among them, the first attention head can be an independent computational unit that guides the multi-head attention mechanism in the model, such as each independent Query-Key-Value mapping module in the Transformer architecture.

[0075] The first attention matrix may be a normalized similarity matrix output by the guidance model through the first attention head.

[0076] Among them, the second attention head can be an independent computing unit of the multi-head attention mechanism in the edge deployment model, such as the compressed version of the Query-Key-Value mapping module in the lightweight Transformer structure.

[0077] The second attention matrix can be the original attention weight matrix output by the edge deployment model through the second attention head.

[0078] Among them, the target second attention matrix can be a low-rank approximation matrix reconstructed after performing singular value decomposition on the second attention matrix.

[0079] Among them, the attention distillation sub-loss can be a Frobenius norm loss term that measures the difference between the first attention matrix and the target second attention matrix.

[0080] In some embodiments, the first attention matrix corresponding to each attention head of the guidance model It can be generated by the following multi-head attention formula: ; in, is the query weight matrix that guides the model, is the key weight matrix for the bootstrap model, is the dimension scaling factor.

[0081] Furthermore, the second attention matrix of the preset edge deployment model Also based on multi-head attention generation, the specific formula is as follows: ; Furthermore, due to the need to project through low-rank SVD ( ) aligns the attention matrix dimension between the guidance model and reduces the computational complexity. Therefore, the target second attention matrix can be obtained: ; in, is the query weight matrix of the preset edge deployment model, is the key weight matrix of the preset edge deployment model, is the dimension scaling factor.

[0082] Afterwards, the difference between the attention matrix of the guided model and the preset edge deployment model can be calculated by the Frobenius norm, that is, the attention distillation loss between the single first attention head and the second attention head can be obtained: ; Finally, the losses of all (a total of M) first attention heads (or all second attention heads, the number of attention heads is the same) are accumulated to obtain the final attention distillation loss: ; Through the above method, the multi-scale attention feature expressions of heterogeneous guidance models can be integrated, and the redundant dimensions of the attention matrix of the preset edge deployment model can be compressed using singular value decomposition to force the preset edge deployment model to learn the attention distribution of the guidance model, effectively reducing the computational overhead of the preset edge deployment model.

[0083] In step 103 , based on each feature extraction level of each guide model and edge deployment model, a corresponding multi-level feature distillation loss is constructed for the difference between the feature maps output by each sample data.

[0084] In some embodiments, in order to achieve multi-level migration of structured knowledge in multi-teacher collaborative distillation, a feature map difference loss term can be constructed between the heterogeneous teacher model and the edge deployment model at the feature extraction level to enhance the student model's ability to learn spatial-semantic joint features, thereby improving the accuracy and dynamic adaptability of the lightweight model in edge scenarios.

[0085] Among them, the feature extraction level can be a computing module in a neural network used to extract local or global features of input data, such as different convolutional layers in a convolutional neural network or multi-head attention layers in a Transformer architecture.

[0086] Among them, the feature map can be a multi-dimensional tensor representation generated by the feature extraction layer after processing the input sample data, such as the local spatial feature matrix extracted by the convolution kernel sliding operation, or the semantic association map generated by the attention mechanism.

[0087] Among them, the multi-level feature distillation loss can be a loss term calculated based on the difference in feature maps output by the guidance model and the preset edge deployment model at multiple feature extraction levels.

[0088] In some embodiments, a multi-level feature distillation loss can be constructed by comparing the output feature maps of the guidance model and the preset edge deployment model at multiple feature extraction levels to achieve knowledge transfer. Specifically, the feature maps between the guidance model and the preset edge deployment model can be compared. Specifically, the feature maps of the first layer of the guidance model are compared. With the k-th layer features of the student model The number of channels is adjusted through 1×1 convolution, and then the spatial size is aligned through adaptive pooling to generate matching features. This process is expressed in the following formula: ; Furthermore, we can use The norm measures the feature difference between the guided model and the preset edge deployment model in each feature extraction level, and adds up the multiple feature differences corresponding to all feature extraction levels to obtain the multi-level feature distillation loss.

[0089] Through the above methods, the complementary expressions of heterogeneous teacher models at different feature extraction levels can be integrated, and the multi-level feature distillation loss can be used to enhance the student model's ability to jointly model edge features and semantic features, so as to reduce information loss in the process of structured knowledge transfer and improve the model generalization performance, thereby providing technical support for subsequent high-precision reasoning and efficient resource deployment of edge devices in complex scenarios.

[0090] In some embodiments, in order to achieve multi-level migration of structured knowledge in multi-teacher collaborative distillation, a feature map difference loss term can be constructed between the heterogeneous teacher model and the edge deployment model at the feature extraction level to enhance the student model's ability to learn spatial-semantic joint features, thereby improving the accuracy and dynamic adaptability of the lightweight model in edge scenarios. Step 103 may include: (103.1) Outputting a first feature map corresponding to each sample data based on each first feature extraction level of each guided model, and outputting a second feature map corresponding to each sample data based on each second feature extraction level of the edge deployment model; (103.2) Based on the first feature map, adjust the number of channels and spatial size of the second feature map to obtain an updated second feature map; (103.3) performing norm normalization on the first feature map to obtain a target first feature map, and performing norm normalization on the updated second feature map to obtain a target second feature map; (103.4) calculating a target difference value between the target first feature map and the corresponding target second feature map using a distance metric function; (103.5) Obtain the weight coefficient for each feature extraction level, and obtain the level-wise feature distillation loss corresponding to each feature extraction level based on the product of the weight coefficient and the target difference value; (103.6) Construct a multi-level feature distillation loss between each guided model and the edge deployment model based on the sum of multiple levels of feature distillation sub-losses corresponding to multiple feature extraction levels.

[0091] The first feature extraction level may be a computational module in a guided model for extracting local or global features of input data, such as a shallow convolutional layer in a convolutional neural network (CNN).

[0092] The first feature map can be a multidimensional tensor representation generated by the first feature extraction layer of the guidance model after processing the input sample data. It can be calculated through the forward propagation of the guidance model. For example, it can be a local spatial feature matrix extracted by a sliding convolution kernel operation or a semantic association map generated by an attention mechanism. Each first feature extraction layer included in the guidance model can output a corresponding first feature map.

[0093] Among them, the second feature extraction level can be a feature extraction module of corresponding functions in the edge deployment model, such as a compressed convolutional layer or a simplified Transformer module in a lightweight CNN structure.

[0094] The second feature map may be a raw feature representation generated by the kth second feature extraction level of the preset edge deployment model after processing sample data consistent with the guidance model. The second feature map may be obtained by forward propagation of the preset edge deployment model, such as a low-dimensional feature matrix output by a lightweight convolution operation. Each second feature extraction level included in the preset edge deployment model may output a corresponding second feature map.

[0095] The number of channels can be the number of feature maps in the depth dimension. For example, the number of channels of the feature map output by the convolutional layer is 64.

[0096] The spatial size may be the size of the feature map in terms of height and width, for example, the spatial size of the feature map is H×W.

[0097] The updated second feature map may be a second feature map adjusted by channel cropping or bilinear interpolation. For example, the number of channels of the second feature map may be adjusted to the same as the number of channels of the first feature map by a 1×1 convolution operation. Figure 1 The updated second feature map is obtained.

[0098] The target first feature map may be a result of performing norm normalization processing on the first feature map.

[0099] The target second feature map may be a result of performing norm normalization processing on the updated second feature map.

[0100] The distance metric function may be a function for calculating the difference between feature maps, such as Euclidean distance or cosine similarity.

[0101] The target difference value may be a distance measurement result between the target first feature map and the target second feature map.

[0102] The weight coefficient may be a scalar value used to adjust the importance of different feature extraction levels, which may be obtained through experimental adjustment or set by an operator.

[0103] Among them, the hierarchical feature distillation sub-loss can be the product term of the weight coefficient and the target difference value.

[0104] In some embodiments, for each feature extraction level of the guidance model, for example, the first feature extraction level, the first feature map corresponding to the input sample data (for example, sample data a) may be extracted. , and for each feature extraction level of the preset edge deployment model, such as the kth feature extraction level, extract the second feature map of the sample data (such as sample data a) that is consistent with the input of the guidance model .

[0105] Furthermore, in order to make the second feature map match the first feature map in terms of dimension, it is necessary to adjust the number of channels and spatial size of the second feature map. Specifically, this can be done through a 1×1 convolution operation ( ) Adjust the number of channels of the second feature map to the same as the first feature Figure 1 The spatial size of the second feature map is adjusted to the same as the first feature map through adaptive pooling operation. Figure 1 So, we can get the updated second feature map . This process can be expressed as: ; Furthermore, in order to eliminate the scale difference between feature maps, the first feature map is normalized to obtain the target first feature map , and normalize the updated second feature map to obtain the target second feature map .

[0106] In some embodiments, the difference between the target first feature map and the target second feature map can be calculated using a distance metric function (such as cosine similarity, Euclidean distance, etc.). For example, the target difference value can be calculated using Euclidean distance. .

[0107] Furthermore, the weight coefficient of each feature extraction level can be The product of the difference between the two values ​​is used to calculate the hierarchical feature distillation loss corresponding to each feature extraction level. The weight coefficient can be obtained by extracting the global representation of each level of the teacher model through the global average pooling operation and performing normalization. The specific formula is as follows: ; In this way, the weight coefficient can be multiplied by the target difference value to calculate the level feature distillation loss corresponding to the lth feature extraction level: .

[0108] Furthermore, the multiple-level feature distillation sub-loss corresponding to the multiple first feature extraction levels (one level feature distillation sub-loss corresponding to each first feature extraction level) can be summed to construct a multi-level feature distillation loss between the guidance model and the edge deployment model. : ; Through the above method, the complementary expressions of heterogeneous guidance models at different feature extraction levels can be integrated, and the multi-level feature distillation loss can be used to enhance the joint modeling ability of edge features and semantic features of the preset edge deployment model, so as to reduce the information loss in the process of structured knowledge transfer and improve the generalization performance of the preset edge deployment model.

[0109] In some implementations, to ensure that feature extraction layers that contribute more to the overall representation capability of the model have a higher weight in the loss function, so that the pre-set edge deployment model can more effectively learn the key features that guide the model, global average pooling can be used to obtain layer feature vectors and calculate their weight coefficients to optimize the multi-level feature distillation loss function and improve the generalization capability of the model. For example, "obtaining the weight coefficient for each feature extraction layer" in (103.5) may include: (103.5.1) For each first feature extraction level, perform a global average pooling operation on the corresponding first feature map to obtain a first vector; (103.5.2) Obtain the sum of multiple first vectors corresponding to multiple first feature extraction levels to obtain the target vector sum; (103.5.3) Based on the ratio of the first vector to the sum of the target vectors, obtain the corresponding weight coefficient for each feature extraction level.

[0110] The first vector may be a feature vector obtained by performing a global average pooling operation on the first feature map.

[0111] The target vector sum may be a linear superposition result of multiple first vectors.

[0112] In some embodiments, for each first feature extraction level of the guidance model, a global average pooling operation may be performed on the corresponding first feature map, and the first feature map The corresponding eigenvalues ​​of all spatial position levels are averaged to obtain the corresponding first vector .

[0113] Furthermore, the sum of multiple first vectors corresponding to multiple first feature extraction levels included in the guidance model can be obtained to obtain the sum of the target vectors .

[0114] Among them, the weight coefficient The global representation of the features of each level of the teacher model can be extracted through the global average pooling (GAP) operation and normalized. It can be calculated based on the ratio of the first vector to the sum of the target vectors. The specific formula is as follows: ; Where L represents the total number of feature extraction levels in the guided model.

[0115] Through the above method, the contribution of each first feature extraction level to the overall model performance can be quantified to dynamically adjust the weight distribution of multi-level feature distillation loss, ensuring that the preset edge deployment model can more effectively learn the key features of the guidance model, thereby improving the performance and generalization ability of the preset edge deployment model.

[0116] In step 104 , based on each guide model and edge deployment model, a corresponding relational distillation loss is constructed for the difference between the feature similarities outputted by any pair of sample data in the sample dataset.

[0117] In some embodiments, in order to achieve the transfer of high-order semantic relationship knowledge in multi-teacher collaborative distillation, the feature similarity difference loss term of the output of the sample data of the heterogeneous teacher model and the edge deployment model can be constructed to enhance the student model's ability to model the logical associations between categories, thereby improving the dynamic adaptability and generalization performance of the lightweight preset edge deployment model in edge scenarios.

[0118] Among them, the sample data pair can be any two input instances in the sample data set; or, the sample data pair can also be two input instances with semantic association, such as image pairs belonging to the same category or different categories in an image classification task, and the specific selection is based on the actual situation.

[0119] The feature similarity may be a similarity measure calculated by the model after extracting features from the sample data pair. For example, it may be calculated using cosine similarity or Euclidean distance.

[0120] Among them, the relation distillation loss can be a loss term that measures the difference in similarity between the output features of the teacher model and the student model for sample data.

[0121] In some embodiments, the inner product operation can be used to calculate the two feature vectors of the sample data pair composed of any two sample data. and Similarly, through the inner product operation, the two feature vectors of the preset edge deployment model for any sample data pair (consistent with the sample data pair used by the guidance model) are calculated. and The similarity between them.

[0122] Furthermore, the sigmoid function can be used to normalize the similarities respectively, and after normalization, the relationship distillation loss is obtained by calculating the cross entropy loss between the normalized feature similarity of the guided model and the normalized feature similarity of the preset edge guided model.

[0123] Through the above method, the high-order semantic relationship modeling ability of the guided model for sample data pairs can be integrated, and the similarity between the guided model and the preset edge guided model in the feature space can be quantified, so as to guide the preset edge guided model to learn the logical association between categories by minimizing the relation distillation loss, so that it can better approximate the feature representation of the guided model.

[0124] In some embodiments, in order to achieve the transfer of high-order semantic relationship knowledge in multi-teacher collaborative distillation, a difference loss term can be constructed between the output feature similarity of the guided model and the edge deployment model for sample data, thereby strengthening the modeling ability of the preset edge deployment model for logical associations between categories, thereby improving the dynamic adaptability and generalization performance of the lightweight model in edge scenarios. Step 104 may include: (104.1) Extracting a first eigenvector of the first sample data included in any sample data pair and a second eigenvector of the second sample data included in any sample data pair through each bootstrap model; (104.2) Extracting, by using the edge deployment model, a third eigenvector corresponding to the first sample data and a fourth eigenvector corresponding to the second sample data; (104.3) Obtaining a first feature similarity between the first eigenvector and the second eigenvector, and determining a first ratio based on the first feature similarity and a ratio of a first temperature parameter corresponding to each guided model; (104.4) Obtaining a second feature similarity between the third eigenvector and the fourth eigenvector, and determining a second ratio based on the second feature similarity and a ratio of a second temperature parameter corresponding to the edge deployment model; (104.5) normalizing the first ratio to obtain a target first ratio, and normalizing the second ratio to obtain a target second ratio; (104.6) Based on the difference between the target first ratio and the target second ratio, the relational distillation loss for any sample data pair is obtained; (104.7) Based on multiple relational distillation sub-losses for multiple sample data pairs corresponding to the sample dataset, a relational distillation loss is constructed between each guided model and the edge deployment model.

[0125] The first sample data may be the first input instance in a sample data pair, such as a picture to be identified in an image classification task.

[0126] The first feature vector may be a feature representation that guides the model to extract the first sample data, such as a high-dimensional vector output by a fully connected layer.

[0127] The second sample data may be the second input instance in the sample data pair, such as an image belonging to the same class or a different class as the first sample data, and both the first sample data and the second sample data are in the sample data set.

[0128] The second feature vector may be a feature representation that guides the model to extract the second sample data, such as a high-dimensional vector output by a fully connected layer.

[0129] The third feature vector may be a feature representation extracted by the edge deployment model from the first sample data, such as a vector output by a lightweight convolutional layer.

[0130] The fourth feature vector may be a feature representation extracted by the edge deployment model from the second sample data, such as a vector output by a lightweight convolutional layer.

[0131] The first feature similarity may be a similarity measure of the output feature vector of the guided model to the sample data.

[0132] The first temperature parameter may be a scalar value used in the guidance model to regulate the smoothness of the Softmax output probability distribution.

[0133] The first ratio may be a ratio of the first feature similarity to the first temperature parameter.

[0134] The second feature similarity may be a similarity measure of the output feature vector of the edge deployment model for the sample data.

[0135] The second temperature parameter may be a scalar value used to regulate the smoothness of the Softmax output probability distribution in the edge deployment model.

[0136] The second ratio may be a ratio of the second feature similarity to the second temperature parameter.

[0137] The target first ratio may be a result of normalizing the first ratio.

[0138] The target second ratio may be a result of normalizing the second ratio.

[0139] The relational distillation sub-loss may be a difference measure between the target first ratio and the target second ratio, which may be calculated using a mean square error formula.

[0140] For example, taking the first sample data as sample i and the second sample data as sample j as an example, by inputting sample i and sample j into the guided model respectively, the first eigenvector corresponding to sample i can be obtained. , and the second eigenvector corresponding to sample j At the same time, sample i and sample j are input into the edge deployment model respectively to obtain the third eigenvector corresponding to sample i , and the fourth eigenvector corresponding to sample j .

[0141] Furthermore, we can calculate the inner product between the first eigenvector and the second eigenvector , get the first feature similarity; then, divide the first feature similarity by the first temperature parameter , get the first ratio In this way, the distribution of the first feature similarity can be adjusted by the first temperature parameter, so that the first feature similarity is smoother or sharper, thereby improving the generalization ability of the model.

[0142] And we can calculate the inner product between the third eigenvector and the fourth eigenvector , get the second feature similarity; then, divide the second feature similarity by the second temperature parameter , and obtain the second ratio In this way, the distribution of the second feature similarity can be adjusted by the second temperature parameter, so that the second feature similarity is smoother or sharper, thereby improving the generalization ability of the model.

[0143] Furthermore, the sigmoid function can be used Normalize the first ratio to obtain the target first ratio , and using the sigmoid function Normalize the second ratio to obtain the target second ratio In this way, the first ratio and the second ratio can be mapped to between 0 and 1, which facilitates calculation on the same scale.

[0144] In some embodiments, the cross entropy loss between the target first ratio and the target second ratio can be calculated to obtain the relation distillation loss: : ; Furthermore, the relational distillation loss of all sample data pairs (each sample data pair corresponds to a relational distillation loss) is summed to obtain the relational distillation loss : ; Through the above method, the relation distillation loss can be used to strengthen the preset edge deployment model's learning of the logical associations between categories, thereby guiding the learning of the preset edge deployment model so that it can better approximate the feature representation of the guidance model.

[0145] In step 105, based on the attention distillation loss, the multi-level feature distillation loss, and the relationship distillation loss, a comprehensive loss is constructed for each guided model, and knowledge distillation is performed on the preset edge deployment model according to the multiple output distribution losses and the multiple comprehensive losses corresponding to the multiple guided models to obtain the target edge deployment model, wherein the multiple guided models have different network architectures.

[0146] In some embodiments, in order to achieve the coordinated optimization of knowledge transfer efficiency and model compression effect in multi-guided model collaborative distillation, a comprehensive loss function can be constructed by integrating attention distillation loss, multi-level feature distillation loss and relationship distillation loss, and the preset edge deployment model can be jointly optimized based on the output distribution loss of the heterogeneous teacher model to improve the accuracy and resource adaptability of the preset edge deployment model in edge scenarios.

[0147] Among them, the comprehensive loss can be the total loss term that is the weighted sum of attention distillation loss, multi-level feature distillation loss and relationship distillation loss. The weight corresponding to each loss can be adjusted according to actual conditions.

[0148] The target edge deployment model can be a lightweight model that has undergone collaborative distillation of multiple guided models and comprehensive loss optimization, such as a MobileNet or SqueezeNet model obtained by updating parameters through backpropagation.

[0149] For example, for each guided model, such as the convolutional neural network ResNet-50, the corresponding attention distillation loss , multi-level feature distillation loss With relation distillation loss Perform weighted fusion to obtain the comprehensive loss of the guided model. Specifically, the formula for calculating the comprehensive loss can be as follows: ; in, It can be the task loss of the preset edge deployment model, which is used to measure its performance in performing the main task (such as classification, regression), and is directly related to the difference between the output of the preset edge deployment model and the true label. For example, it can be a classification loss; 、 、 、 It can be set according to actual conditions.

[0150] In some embodiments, adaptive adjustment can be based on gradient descent 、 、 、 The value of is set so that the preset edge deployment model can gradually approach the optimal solution. Specifically, for The value at training stage t+1, that is, 、 、 、 The value of any weight in the training phase t+1 can be adjusted by the following formula: ; in, represents the value of the i-th weight in the t+1-th iteration; represents the value of the i-th weight in the t-th iteration; Represents the learning rate, which determines the amplitude of parameter update. The larger the learning rate, the larger the update amplitude. represents comprehensive loss; Reflects the loss function in weights The trend of change in direction: when the gradient is large, the exponential term is small, and the parameter update amplitude is relatively small, avoiding the instability caused by excessive jumps; when the gradient is small, the exponential term is large, allowing the parameters to be adjusted more significantly, accelerating the convergence process.

[0151] In some embodiments, multiple guidance models have different network architectures, and different network architecture designs tend to capture different types of information. For example, convolutional neural networks excel at processing local features and spatial structure information in images, while transformer architectures are better at capturing global dependencies. Different architectures often contain different prior assumptions and inductive biases. By using guidance models with multiple different architectures, it is possible to ensure that the preset edge deployment model can learn rich knowledge from multiple perspectives. By leveraging this differentiated prior knowledge, the preset edge deployment model can perform better when faced with new tasks or unseen data.

[0152] Exemplarily, the guidance model may include a convolutional neural network (such as ResNet-50), a visual transformer (such as ViT-Base), a self-attention mechanism enhanced network (such as Swin Transformer), and the like.

[0153] Furthermore, after calculating the comprehensive loss of a guided model, the comprehensive loss of the guided model can be compared with the comprehensive loss of other guided models. Perform weighted average calculation to obtain the target comprehensive loss. For example, guide model 1 corresponds to the first weight and comprehensive loss , guide model 2 corresponds to the second weight and comprehensive loss , guide model 3 corresponds to the third weight and comprehensive loss , guide model 4 corresponds to the fourth weight and comprehensive loss , then the target comprehensive loss It can be calculated by the following formula: ; At the same time, the output distribution loss of this guided model is combined with the output distribution loss of other guided models A weighted average calculation is performed to obtain the target output distribution loss. The weighting method is similar to the calculation method of the target comprehensive loss and will not be repeated here. Finally, based on the target comprehensive loss and the target output distribution loss, the preset edge deployment model is distilled to adjust the model parameters to obtain the target edge deployment model. Furthermore, the weighted value of the loss corresponding to each model can be set according to actual conditions, such as by a technician, or randomly set, or set by the number of training times of the guide model. For example, the guide model with the most training times can be assigned a better weighted weight, etc. The embodiments of the present application do not specifically limit this.

[0154] In some embodiments, the distillation process can be modeled as a differential equation: ; in, represents the parameter vector of the preset edge deployment model; t represents the time variable, that is, the time or number of iterations during the training process; Indicates the rate of change of the preset edge deployment model over time, that is, the speed of parameter update; Represents the total loss function (the sum of the target comprehensive loss and the target output distribution loss, or the target comprehensive loss function, which is determined according to the actual situation) about The gradient of , which represents the rate of change of the loss function in the direction of the parameter; represents a balancing factor, which is used to control the impact of the difference between the bootstrap model and the preset edge deployment model; represents the parameter vector of the k-th bootstrap model.

[0155] This application can be widely used in edge computing scenarios where resources are limited and efficient intelligent reasoning is required, such as real-time image classification and enhanced photography on smartphones, real-time behavior recognition and anomaly detection in smart security cameras, equipment status monitoring and predictive maintenance in industrial IoT devices, lightweight environmental perception modules in autonomous driving vehicle systems, and instant pathological image analysis on portable medical devices. Through multi-teacher knowledge distillation, the accuracy and generalization ability of edge deployment models can be significantly improved, while meeting the deployment requirements of low latency and low power consumption.

[0156] By calculating the gradient of the total loss function with respect to the parameters of the pre-set edge deployment model and updating the parameters in the opposite direction of the gradient, we ensure that the pre-set edge deployment model can gradually approach the optimal solution, improving its performance on the target task. Furthermore, by introducing the difference between the guidance model parameters and the pre-set edge deployment model parameters and smoothing them using an exponential moving average (EMA), the pre-set edge deployment model can gradually absorb the advantages of each guidance model while maintaining its own characteristics, improving its generalization and robustness.

[0157] The embodiment of the present application obtains a sample data set, and based on the preset edge deployment model and each guiding model, constructs a corresponding output distribution loss for the difference between the predicted probability distributions output by each sample data in the sample data set; based on each attention head of each guiding model and the edge deployment model, constructs a corresponding attention distillation loss for the difference between the attention matrices output by each sample data; based on each feature extraction level of each guiding model and the edge deployment model, constructs a corresponding multi-level feature distillation loss for the difference between the feature maps output by each sample data; based on each guiding model and the edge deployment model, constructs a corresponding relational distillation loss for the difference between the feature similarities output by any pair of sample data in the sample data set; based on the attention distillation loss, the multi-level feature distillation loss and the relational distillation loss, constructs a comprehensive loss for each guiding model, and performs knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and the multiple comprehensive losses corresponding to the multiple guiding models to obtain a target edge deployment model, wherein the multiple guiding models have different network architectures. In this way, by introducing multiple guidance models with different network architectures, it is possible to effectively overcome the problem of a single knowledge perspective of a single guidance model and avoid the homogeneity dependence generated during the distillation process; at the same time, combining a multi-level knowledge distillation mechanism to construct a comprehensive loss function can provide richer and more diverse sources of knowledge and finer-grained supervision signals. Specifically, multiple different guidance models can effectively solve the problem of knowledge homogeneity, transfer category probability knowledge through output distribution loss, capture the fine-grained attention mechanism within the model, multi-level feature distillation loss ensures the consistency of the feature extraction layer, and relational distillation loss encodes the global structural information between samples. This fusion of multi-level and multi-source knowledge enables the edge deployment model to learn more robust and generalized representations, reduces the risk of overfitting, and improves adaptability and performance on edge devices. In summary, the present application can improve the generalization ability of the edge deployment model after knowledge distillation and improve the performance of the edge deployment model.

[0158] In some embodiments, in order to achieve dynamic adaptation and optimization of heterogeneous models to multimodal data in multi-teacher collaborative distillation, the generalization ability and deployment efficiency of lightweight models in complex data scenarios can be improved by quantifying the matching degree between data properties and network architecture and dynamically adjusting loss weights. In step 105, "performing knowledge distillation on the preset edge deployment model based on multiple output distribution losses and multiple comprehensive losses corresponding to multiple guidance models to obtain a target edge deployment model" may include: (105.1) Obtaining data properties corresponding to the sample data, wherein the data properties include at least one of image properties, text properties, and multimodal properties; (105.2) Obtain the adaptability index of the network architecture of each guided model to the data properties, and determine the fusion weight corresponding to each guided model based on the adaptability index; (105.3) According to the fusion weight corresponding to each guided model, each output distribution loss and each comprehensive loss are adjusted to obtain the corresponding target output distribution loss and target comprehensive loss; (105.4) Based on the multiple target output distribution losses and multiple target comprehensive losses corresponding to the multiple guidance models, the preset edge deployment model is knowledge distilled to obtain the target edge deployment model.

[0159] The data properties may be type characteristics of the sample data, such as image properties (such as pixel distribution), text properties (such as word frequency statistics), or multimodal properties (such as the correlation between image and text).

[0160] The image properties can be quantitative descriptions of visual features in the sample data, such as edge texture features or color histogram statistics extracted by a convolutional layer.

[0161] The text properties can be a quantitative description of the language features in the sample data, such as word vectors or TF-IDF features extracted by a word embedding model.

[0162] Among them, the multimodal property can be a quantitative description of the cross-modal correlation characteristics in the sample data, such as the image-text similarity matrix calculated by the cross-modal attention mechanism.

[0163] Among them, the adaptability index can be an indicator to measure the matching degree of the guiding model network architecture to the specific data properties, such as a scalar value calculated by the verification accuracy or feature consistency score of the model on the corresponding data.

[0164] The fusion weight may be a dynamic weight coefficient normalized based on the adaptability index.

[0165] Among them, the target output distribution loss can be the output distribution loss term after fusion weight adjustment.

[0166] Among them, the target comprehensive loss can be the comprehensive loss term after the fusion weight is adjusted.

[0167] For example, data properties corresponding to the sample data can be obtained, where the data properties include at least one of image properties, text properties, and multimodal properties. In specific implementations, these data properties can be obtained by preprocessing and feature extraction of the sample data. For example, for image data, its size, color distribution, and texture features can be obtained; for text data, word frequency, sentence length, grammatical structure, etc. can be counted; and for multimodal data, the correlation and complementarity between different modalities need to be comprehensively considered.

[0168] In some implementations, for each guidance model, an adaptability index of its network architecture to the data properties can be obtained. The adaptability index reflects the ability and effectiveness of the guidance model in processing a specific type of data. The specific calculation formula is as follows: ; in, represents the parameter vector of the i-th bootstrap model, represents a sample dataset, is an evaluation function that measures the degree of fit between the bootstrap model and the data properties. The evaluation function can be designed based on the actual situation, for example, by calculating it through cross-validation or specific indicators such as accuracy or F1 value.

[0169] Furthermore, based on the obtained adaptability index, the fusion weight corresponding to each guidance model can be further determined. The fusion weight represents the relative importance of each guidance model in the knowledge distillation process. The specific calculation formula is as follows: ; Where N represents the total number of guided models; represents the fusion weight of the i-th guided model. In this way, it can ensure that the guided model that better matches the nature of the data plays a greater role in the knowledge transfer process.

[0170] In some implementations, an adaptive adjustment mechanism can be introduced to dynamically update the fusion weights. For example, a gradient descent-based adaptive adjustment strategy can monitor the learning progress and performance of the student model in real time and adjust the weights of each guiding model to achieve optimal knowledge transfer.

[0171] For example, if the bootstrap model 1 corresponds to the first weight and comprehensive loss , guide model 2 corresponds to the second weight and comprehensive loss , guide model 3 corresponds to the third weight and comprehensive loss , guide model 4 corresponds to the fourth weight and comprehensive loss , then the target comprehensive loss It can be calculated by the following formula: ; In some embodiments, the target output distribution loss can also be obtained by weighted fusion of the output distribution losses corresponding to multiple guidance models through the above distribution. The specific implementation can refer to the calculation of the above target comprehensive loss, which will not be repeated here.

[0172] Furthermore, based on the multiple target output distribution losses and multiple target comprehensive losses corresponding to the multiple guidance models, the preset edge deployment model is subjected to knowledge distillation to obtain the target edge deployment model.

[0173] It is understandable that the number and type of guidance models can be selected according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0174] Through the above methods, the knowledge migration priority of the guided model under different data properties can be dynamically adapted to enhance the structured knowledge learning ability of the preset edge deployment model for multimodal data, thereby providing technical support for high-precision reasoning and efficient resource deployment of subsequent edge devices in heterogeneous data scenarios.

[0175] See also Figure 3 The present application also provides a knowledge distillation device that can implement the above-mentioned knowledge distillation method. The knowledge distillation device includes: An acquisition module 31 is configured to acquire a sample data set and construct a corresponding output distribution loss based on the difference between the predicted probability distributions outputted by each sample data in the sample data set, based on the preset edge deployment model and each guided model; A first construction module 32 is used to construct a corresponding attention distillation loss based on the difference between the attention matrices output by each guide model and each attention head of the edge deployment model for each sample data; A second construction module 33 is configured to construct a corresponding multi-level feature distillation loss based on the difference between the feature maps output by each sample data, based on each feature extraction level of each guide model and the edge deployment model; A third construction module 34 is configured to construct a corresponding relational distillation loss based on the difference between the feature similarities output by any pair of sample data in the sample dataset, based on each guide model and edge deployment model; The distillation module 35 is used to construct a comprehensive loss for each guidance model based on the attention distillation loss, the multi-level feature distillation loss, and the relationship distillation loss, and perform knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and the multiple comprehensive losses corresponding to the multiple guidance models to obtain the target edge deployment model, wherein the multiple guidance models have different network architectures.

[0176] The specific implementation of the knowledge distillation device is basically the same as the specific embodiment of the knowledge distillation method described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiment of this application, the knowledge distillation device can also be equipped with other functional modules to implement the knowledge distillation method in the above embodiment.

[0177] The present application also provides a computer device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described knowledge distillation method. The computer device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0178] See also Figure 4 , Figure 4 The hardware structure of a computer device according to another embodiment is shown. The computer device includes: The processor 41 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application. The memory 42 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 42 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 42 and is called by the processor 41 to execute the knowledge distillation method of the embodiments of this application. Input / output interface 43, used for information input and output; Communication interface 44, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); bus 45 , which transmits information between the various components of the device (e.g., processor 41 , memory 42 , input / output interface 43 , and communication interface 44 ); The processor 41 , the memory 42 , the input / output interface 43 and the communication interface 44 are connected to each other in communication within the device via a bus 45 .

[0179] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned knowledge distillation method.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0182] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0183] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0184] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0185] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0186] It should be understood that in this application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0188] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0191] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A knowledge distillation method, characterized in that: The method comprises: Obtain a sample data set, and construct a corresponding output distribution loss based on the difference between the predicted probability distributions output by each sample data in the sample data set based on the preset edge deployment model and each guided model; Based on each attention head of each guide model and each attention head of the edge deployment model, constructing a corresponding attention distillation loss for the difference between the attention matrices output by each sample data; Based on each feature extraction level of each guided model and the edge deployment model, constructing a corresponding multi-level feature distillation loss for the difference between the feature maps output by each sample data; Based on each of the guide models and the edge deployment model, constructing a corresponding relational distillation loss for the difference between the feature similarities output by any pair of sample data in the sample data set; Based on the attention distillation loss, the multi-level feature distillation loss and the relationship distillation loss, a comprehensive loss for each guided model is constructed, and according to the multiple output distribution losses and multiple comprehensive losses corresponding to the multiple guided models, knowledge distillation is performed on the preset edge deployment model to obtain a target edge deployment model, wherein the multiple guided models have different network architectures.

2. The knowledge distillation method according to claim 1, characterized in that: The method of performing knowledge distillation on the preset edge deployment model according to the multiple output distribution losses and the multiple comprehensive losses corresponding to the multiple guidance models to obtain a target edge deployment model includes: Acquiring data properties corresponding to the sample data, wherein the data properties include at least one of image properties, text properties, and multimodal properties; Obtaining an adaptability index of the network architecture of each guiding model to the data properties, and determining a fusion weight corresponding to each guiding model based on the adaptability index; Adjusting each output distribution loss and each comprehensive loss according to the fusion weight corresponding to each guided model to obtain the corresponding target output distribution loss and target comprehensive loss; According to the multiple target output distribution losses and the multiple target comprehensive losses corresponding to the multiple guidance models, knowledge distillation is performed on the preset edge deployment model to obtain a target edge deployment model.

3. The knowledge distillation method according to claim 1, characterized in that: The method of constructing a corresponding output distribution loss based on the difference between the predicted probability distribution output of each sample data in the sample data set based on the preset edge deployment model and each guide model includes: Obtaining preset adjustment parameters and temperature hyperparameters, wherein the temperature hyperparameters are used to soften the predicted probability distribution of each guided model; Obtaining a first predicted probability distribution output by each guidance model for each sample data in the sample data set, and a second predicted probability distribution output by a preset edge deployment model for each sample data, and determining corresponding distribution divergence values ​​based on the first predicted probability distribution and the second predicted probability distribution; determining a first output distribution loss sub-item based on a product of the adjustment parameter, the temperature hyperparameter, and the distribution divergence value; Obtaining a classification loss corresponding to the preset edge deployment model, and adjusting the classification loss based on the adjustment parameter to obtain a second output distribution loss sub-item; Based on the first output distribution loss sub-item and the second output distribution loss sub-item, an output distribution loss between each guided model and the edge deployment model is constructed.

4. The knowledge distillation method according to claim 1, characterized in that: The step of constructing a corresponding attention distillation loss based on the difference between the attention matrices output by each sample data based on each attention head of each guide model and the edge deployment model includes: Based on each first attention head of each guide model, output a first attention matrix corresponding to each sample data, and based on each second attention head of the edge deployment model, output a second attention matrix corresponding to each sample data; Performing a singular value decomposition operation on the second attention matrix to obtain a target second attention matrix; Based on the difference between the first attention matrix and the corresponding target second attention matrix, construct a corresponding attention distillation sub-loss; Based on multiple attention distillation sub-losses corresponding to multiple first attention heads, an attention distillation loss is constructed between each guided model and the edge deployment model.

5. The knowledge distillation method according to claim 1, characterized in that: The constructing of a corresponding multi-level feature distillation loss based on the difference between the feature maps output by each sample data at each feature extraction level of each guide model and the edge deployment model includes: Outputting a first feature map corresponding to each sample data based on each first feature extraction level of each guide model, and outputting a second feature map corresponding to each sample data based on each second feature extraction level of the edge deployment model; Based on the first feature map, adjusting the number of channels and the spatial size of the second feature map to obtain an updated second feature map; performing norm normalization processing on the first feature map to obtain a target first feature map, and performing norm normalization processing on the updated second feature map to obtain a target second feature map; Calculating a target difference value between the target first feature map and the corresponding target second feature map by using a distance metric function; Obtaining a weight coefficient for each feature extraction level, and obtaining a level feature distillation sub-loss corresponding to each feature extraction level based on a product of the weight coefficient and the target difference value; Based on the sum of multiple-level feature distillation sub-losses corresponding to multiple feature extraction levels, a multi-level feature distillation loss is constructed between each guided model and the edge deployment model.

6. The knowledge distillation method according to claim 5, characterized in that: The obtaining of the weight coefficient of each feature extraction level includes: For each first feature extraction level, performing a global average pooling operation on the corresponding first feature map to obtain a first vector; Obtaining the sum of multiple first vectors corresponding to multiple first feature extraction levels to obtain a target vector sum; Based on the ratio of the first vector to the sum of the target vectors, a corresponding weight coefficient of each feature extraction level is obtained.

7. The knowledge distillation method according to claim 1, characterized in that: The constructing of a corresponding relational distillation loss based on each of the guide models and the edge deployment model for the difference between the feature similarities output by any pair of sample data in the sample data set includes: Extracting a first eigenvector of the first sample data and a second eigenvector of the second sample data included in any sample data pair through each guided model; Extracting, by using the edge deployment model, a third eigenvector corresponding to the first sample data and a fourth eigenvector corresponding to the second sample data; Obtaining a first feature similarity between the first feature vector and the second feature vector, and determining a first ratio based on the first feature similarity and a ratio of a first temperature parameter corresponding to each of the guided models; Obtaining a second feature similarity between the third eigenvector and the fourth eigenvector, and determining a second ratio based on the second feature similarity and a ratio of a second temperature parameter corresponding to the edge deployment model; Normalizing the first ratio to obtain a target first ratio, and normalizing the second ratio to obtain a target second ratio; Obtaining a relational distillation sub-loss for the arbitrary sample data pair based on a difference between the target first ratio and the target second ratio; Based on multiple relational distillation sub-losses of multiple sample data pairs corresponding to the sample data set, a relational distillation loss between each guided model and the edge deployment model is constructed.

8. A knowledge distillation device, characterized in that: The device comprises: An acquisition module is configured to acquire a sample data set and construct a corresponding output distribution loss based on the difference between the predicted probability distributions output by each sample data in the sample data set based on a preset edge deployment model and each guided model; A first construction module is configured to construct a corresponding attention distillation loss based on the difference between the attention matrices output by each guide model and each attention head of the edge deployment model for each sample data; A second construction module is configured to construct a corresponding multi-level feature distillation loss based on the difference between the feature maps output by each sample data and each feature extraction level of each guide model and the edge deployment model; A third construction module is configured to construct a corresponding relational distillation loss based on the difference between the feature similarities output by any pair of sample data in the sample data set based on each of the guide models and the edge deployment model; A distillation module is used to construct a comprehensive loss for each guided model based on the attention distillation loss, the multi-level feature distillation loss, and the relationship distillation loss, and perform knowledge distillation on the preset edge deployment model according to multiple output distribution losses and multiple comprehensive losses corresponding to multiple guided models to obtain a target edge deployment model, wherein the multiple guided models have different network architectures.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the knowledge distillation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the knowledge distillation method according to any one of claims 1 to 7 is implemented.

Images