Multi-model collaborative knowledge distillation optimization method based on edge calculation

Abstract

The invention discloses a multi-model collaborative knowledge distillation optimization method based on edge computing. The method comprises the following steps that a plurality of edge devices construct local models respectively and train the local models through collected local data, and the edge devices upload the trained models to a server; real-time performance evaluation is carried out on local models on the edge devices, whether the performance evaluation result meets preset conditions or not is judged, if not, the local models are updated, and the model updating comprises the following steps that 1, a multi-model collaborative distillation strategy is adopted, and knowledge exchange is carried out on the local models on the different edge devices; 2) carrying out data heterogeneity and imbalance processing; 3) carrying out resource adaptive scheduling; (4) the server carries out local model aggregation and issues model updating content in the iteration process to the edge device; if yes, the local model does not need to be updated in the iteration. According to the invention, the performance of multiple models on the edge device can be improved, the communication overhead is reduced, and the data privacy is ensured.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to a multi-model collaborative knowledge distillation optimization method based on edge computing. Background Art

[0002] In the field of edge computing, especially in the application of mobile devices and IoT terminals, data processing and model training face the challenge of resource constraints. In order to enable these devices to efficiently run complex deep learning models, knowledge distillation technology has emerged. It improves the operating efficiency of models in resource-constrained environments through model compression and acceleration. However, most of the existing knowledge distillation technologies focus on the optimization of a single model, ignoring the need for collaborative optimization of multiple models and multiple tasks in edge computing environments. In addition, the heterogeneity and imbalance of data between edge devices further increase the complexity of knowledge distillation. These problems limit the further improvement of model performance.

[0003] Previous knowledge distillation methods were mainly performed between a single teacher-student model pair, where the student model was trained by imitating the behavior of the teacher model. These methods are effective in single model compression, but in multi-model environments, especially in edge computing scenarios, traditional methods are difficult to apply directly due to device diversity and inconsistent data distribution. In addition, differences in communication and computing capabilities between edge devices also complicate the implementation of effective knowledge distillation strategies on these devices. Previous model aggregation algorithms often lead to poor model performance and require more communication rounds to converge when processing non-independent and identically distributed data due to a lack of consideration of data diversity.

[0004] In order to solve the above problems, researchers have tried to introduce the concepts of multi-task learning and knowledge distillation to achieve collaborative optimization among multiple models. These methods allow different models to share knowledge by exchanging knowledge between edge devices, thereby improving performance. However, when dealing with data heterogeneity and imbalance, these methods often adopt simple data sampling or model averaging strategies, which cannot fully consider the differences in data distribution and may cause the model to prefer certain data categories, thereby affecting the generalization ability of the model. In addition, existing multi-model collaborative optimization methods are also insufficient in resource scheduling. Since the computing power and network conditions of edge devices are dynamically changing, fixed-frequency knowledge distillation and resource allocation strategies may not be able to adapt to these changes, resulting in resource waste or inefficient model training. Some methods may still perform knowledge distillation when the network conditions are poor, which not only increases communication overhead, but also may affect model performance due to incomplete data transmission. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a multi-model collaborative knowledge distillation optimization method based on edge computing. In the method, the performance of multiple models on edge devices can be improved, while reducing communication overhead and ensuring data privacy.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A multi-model collaborative knowledge distillation optimization method based on edge computing is applied to an end-cloud collaborative system, wherein the end-cloud collaborative system includes a server and a plurality of edge devices connected to the server, each of the edge devices is connected to a plurality of clients in an area, and includes the following steps:

[0008] Several edge devices respectively build local models and train the local models through the collected local data to obtain trained local models. Several edge devices upload the trained models to the server.

[0009] Perform real-time performance evaluation on the local model on the edge device to determine whether the performance evaluation result meets the preset conditions. If not, update the local model. The updated model includes the following: 1) adopt a multi-model collaborative distillation strategy to exchange knowledge between local models on different edge devices; 2) process data heterogeneity and imbalance, adopt a self-distillation method and multi-level branch regularization technology to enhance the model's adaptability to local data and improve global model performance; 3) perform adaptive resource scheduling to dynamically adjust the computing resources of the edge device; 4) the server aggregates the local model and sends the model update content in this round of iteration to the edge device, and performs task decision support based on the updated local model; if yes, there is no need to update the local model in this round of iteration, and task decision support is performed based on the local model.

[0010] Preferably, real-time performance evaluation of the local model on the edge device is performed, specifically comprising the following steps:

[0011] Determine a performance evaluation indicator, wherein the performance evaluation indicator includes one or a combination of accuracy, recall, and F1 score;

[0012] The local model output collected on the edge device is compared with the real-time data to determine whether the comparison result is lower than the preset performance threshold. If so, knowledge distillation is started to update the local model performance and determine the degree of knowledge distillation; if not, there is no need to update the local model.

[0013] Preferably, the performance evaluation index adopts accuracy, and the degree of knowledge distillation is controlled by adjusting the weight factor in the distillation loss function, and the weight factor is as follows:

[0014]

[0015] Among them, f(·,·) is a mapping function that dynamically adjusts α, θ according to the gap between the performance evaluation index and the performance threshold. perf is the preset performance threshold.

[0016] Preferably, a collaborative distillation framework is used to exchange knowledge between local models on different edge devices, specifically including the following steps:

[0017] Knowledge exchange is carried out through federated learning. Specifically, in each iteration of federated learning, the server collects the model weights from each edge device, generates a global model weight, and distributes the global model weight to the edge devices participating in the learning. The formula is as follows:

[0018]

[0019] Among them, W global is the global model weight, W k is the model weight of the kth edge device, and K is the number of edge devices participating in the learning;

[0020] Cosine similarity is used to measure the similarity between tasks, and its calculation formula is:

[0021]

[0022] Among them, S ij is the similarity measure between the i-th task and the j-th task, W i and W j are the model weight vectors of the two tasks respectively;

[0023] Minimizing the difference between the global model and the models of each edge device while taking into account the similarity between tasks, the optimization problem can be expressed as:

[0024]

[0025] in, is the loss function, which measures the difference between the global model and the edge device model, λ k is the weight associated with edge device k, μ ij is the weight associated with the similarity between tasks i and j;

[0026] A personalized head is introduced based on the global model. The personalized head is expressed by the following formula:

[0027]

[0028] in, is the final model weight on edge device k, is the personalized header weight added for edge device k.

[0029] Preferably, the following steps are also included: in federated learning, only the updated part of the model weights is uploaded and downloaded between the edge device and the global model.

[0030] Preferably, a self-distillation method and a multi-level branch regularization technique are used to enhance the adaptability of the model to local data and improve the global model performance, which specifically includes the following steps:

[0031] By performing client self-distillation on the client, the objective function of client self-distillation can be expressed as:

[0032]

[0033] in, is the cross entropy loss, f(x) is the output of the client model, y is the true label, α is the weight factor, β is the weight of the distillation loss, P(z) and Q(z) are the output distributions under different dropout masks, respectively. is the KL divergence loss;

[0034] The server uses multi-level branch regularization technology to deal with the heterogeneity and imbalance of data between clients. The objective function of multi-level branch regularization is:

[0035]

[0036] Among them, H is the number of layers in the model, γ i is the regularization weight of the corresponding level, is the loss function of the i-th layer, is the aggregation loss;

[0037] A heterogeneity-aware data sampling strategy is adopted. According to the distribution characteristics of the data on the edge devices, the sampling probability of samples on each device is dynamically adjusted to ensure that the server can aggregate more representative data. The sampling strategy can be expressed by the following formula:

[0038]

[0039] Among them, P(x) is the sampling probability of sample x, d(x) is the distribution distance measure of sample x, and λ d is a parameter that controls the effect of distribution distance;

[0040] An imbalance correction mechanism is introduced, which improves the model's ability to recognize minority samples by giving higher weights to minority samples. The loss function of imbalance correction is:

[0041]

[0042] Where C is the total number of categories, ω c is the weight of category c, is the cross entropy loss for category c;

[0043] The overall objective function of end-to-end training can be expressed as:

[0044]

[0045] Preferably, the adaptive resource scheduling and dynamic adjustment of computing resources of edge devices specifically include the following steps:

[0046] Evaluate the computing power and network status of each edge device. The evaluation indicators of computing power include the CPU and memory usage of the device and the time required to process a specific task. The evaluation indicators of network status include transmission time and packet loss rate.

[0047] The current system load is determined based on the evaluation results of computing power and network conditions, and the resource requirements of the local model to complete the task are predicted. Resources are dynamically allocated based on the current system load and predicted resource requirements to optimize communication overhead and computing load.

[0048] Preferably, the optimization of communication overhead can be expressed as:

[0049]

[0050] Among them, O comm is the communication overhead, K is the number of edge devices, ΔW k is the model weight update amount of the kth edge device, is ΔW k The square of the norm of , which indicates the magnitude of the weight update, is the normalized communication cost of the weight model of edge device k in the previous iteration, ensuring that the communication costs of different devices and different rounds are comparable.

[0051] Based on the above technical solution, the beneficial effects of the present invention are:

[0052] 1) The present invention performs real-time performance evaluation on the local model on the edge device and dynamically decides whether to perform knowledge distillation based on the evaluation results. This method enables the model to adapt to changes in data distribution more quickly, especially when facing non-independent and identically distributed data, which can effectively improve the generalization ability of the model.

[0053] 2) The present invention adopts a multi-model collaborative distillation strategy, allowing knowledge exchange between models on different edge devices, and achieves collaborative optimization of multiple tasks by sharing key parts of the model. This collaborative mechanism not only improves the performance of a single model, but also enhances the efficiency of the entire system when processing multiple tasks.

[0054] 3) The present invention can handle data heterogeneity and imbalance, and adopts self-distillation method and multi-level branch regularization technology to effectively solve the problem of inconsistent data distribution among edge devices. The adaptability of the model to local data is enhanced through local self-distillation, while the multi-level branch regularization technology on the server side improves the performance of the global model.

[0055] 4) The present invention can perform adaptive resource scheduling, dynamically adjust resource allocation according to the computing power of edge devices and network conditions, and optimize communication overhead and computing load. Experimental data show that while maintaining model performance, the method of the present invention can significantly reduce 20% of data transmission during the model update process.

[0056] 5) The present invention improves computing resource utilization: Through adaptive resource scheduling, the present invention can allocate computing resources more reasonably, so that the computing power of edge devices is fully utilized while avoiding resource waste. This method is particularly effective on resource-constrained edge devices.

[0057] 6) The present invention improves system robustness: The method of the present invention uses a multi-model collaborative distillation strategy and resource adaptive scheduling, so that the system can better adapt to the dynamic changes in data distribution and fluctuations in the communication environment, thereby improving the robustness of the system.

[0058] 7) The present invention accelerates model convergence: Through multi-model collaborative distillation strategy and resource adaptive scheduling, the method of the present invention can accelerate the convergence speed of the model, especially when facing large-scale data sets, and can achieve ideal model performance more quickly.

[0059] 8) Data privacy protection: The method of the present invention avoids the direct uploading of sensitive data and enhances data privacy protection by performing client model training and knowledge distillation on edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The present invention is a flowchart of a multi-model collaborative knowledge distillation optimization method based on edge computing in an embodiment. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0062] like Figure 1As shown, this embodiment provides a multi-model collaborative knowledge distillation optimization method based on edge computing, which is applied to the application of the intelligent manufacturing field of industrial Internet of Things in the end-cloud collaborative system. The end-cloud collaborative system includes a server and several edge devices connected to the server, and each edge device is connected to several clients in the area, including the following steps:

[0063] Several edge devices build local models respectively and train the local models through the collected local data to obtain the trained local models. Several edge devices upload the trained models to the server. The collected local data includes machine status, production parameters and environmental variables. The edge devices also pre-process the collected local data, extract production features, and perform preliminary analysis.

[0064] Perform real-time performance evaluation on the local model on the edge device to determine whether the performance evaluation result meets the preset conditions. If not, update the local model. The updated model includes the following: 1) Use a collaborative distillation framework to exchange knowledge between local models on different edge devices; 2) Process data heterogeneity and imbalance, use self-distillation methods and multi-level branch regularization technology to enhance the model's adaptability to local data and improve global model performance; 3) Perform adaptive resource scheduling to dynamically adjust the computing resources of edge devices; 4) The server aggregates the local model and sends the model update content during this round of iteration to the edge device, and performs task decision support based on the updated local model to optimize production efficiency; if yes, there is no need to update the local model in this round of iteration, and task decision support is performed based on the local model.

[0065] In one embodiment, a multi-model collaborative knowledge distillation optimization method based on edge computing provides a specific process for real-time performance evaluation of local models on edge devices. This process is to perform real-time performance evaluation of models on edge devices through a lightweight evaluation mechanism to determine the performance of the model on the current task, and based on the evaluation results, decide whether knowledge distillation is needed and the degree of distillation. The following are the technical details of the model adaptability evaluation process:

[0066] 1) Selection and calculation of performance evaluation indicators In order to evaluate the adaptability of the model, we first need to select appropriate performance evaluation indicators. Common indicators include accuracy, recall, F1 score, etc. These indicators can fully reflect the performance of the model on a specific task. Given a model M on an edge device and its performance on a test set D test The performance on the above, the accuracy Acc can be calculated as:

[0067]

[0068] in, It is an indicator function. When the model's prediction M(x) is equal to the true label y, the function value is 1, otherwise it is 0.

[0069] 2) Real-time performance monitoring Real-time performance monitoring is achieved by continuously collecting model outputs on edge devices and comparing them with real-time data. In order to reduce the computational burden, a sampling method can be used. The monitoring process can be expressed as:

[0070]

[0071] Among them, D stream is a sample set in the real-time data stream, x t and t are the input and label at time t respectively.

[0072] 3) Adaptive decision logic Based on the real-time monitored performance indicators, a decision logic needs to be designed to determine whether to trigger the knowledge distillation process. The decision logic can be based on the comparison of performance evaluation indicators with preset performance thresholds. If the performance of the model is lower than the performance threshold, knowledge distillation is initiated to improve the model performance:

[0073]

[0074] Among them, B Distill is a Boolean variable indicating whether knowledge distillation is performed, θ perf is the preset performance threshold.

[0075] 4) Determination of the degree of distillation Once the decision is made to perform knowledge distillation, the degree of distillation needs to be determined, that is, to what extent the model should be updated to adapt to the new data distribution. The degree of distillation can be controlled by adjusting the weight factor in the distillation loss function. The determination of the weight factor α can be based on the degree of performance degradation:

[0076]

[0077] Here, f(·,·) is a mapping function that dynamically adjusts α according to the gap between the performance indicator and the threshold (used for self-distillation objective function adjustment in Formula 10).

[0078] 5) Feedback mechanism for model adaptability evaluation In order to make the model adaptability evaluation more accurate, a feedback mechanism can be introduced. By analyzing the improvement of model performance after knowledge distillation, the threshold θ of the evaluation index can be adjusted. perf and mapping function f. The purpose of the feedback mechanism is to make the model fitness evaluation more accurate through iterative optimization.

[0079] Through the above steps, a dynamic and adaptive evaluation and decision-making process can be provided for the models on edge devices, thus providing a solid foundation for the implementation of knowledge distillation strategies. Its design fully considers the resource constraints and device heterogeneity in edge computing environments, and provides technical support for the implementation of effective multi-model collaborative knowledge distillation optimization methods.

[0080] In one embodiment, a multi-model collaborative knowledge distillation optimization method based on edge computing provides a specific process of using a multi-model collaborative distillation strategy to exchange knowledge between local models on different edge devices. The multi-model collaborative distillation strategy aims to achieve collaborative optimization of multiple tasks through knowledge exchange between models on different edge devices. The design takes into account the diversity of edge devices and the heterogeneity of tasks, and promotes mutual learning and performance improvement between multiple models by sharing key parts of the model. The following are the technical details of the multi-model collaborative distillation strategy:

[0081] 1) Construction of the collaborative distillation framework The design of the collaborative distillation framework allows models on edge devices to share knowledge through federated learning. In each iteration of federated learning, the server collects model weights from each edge device, then generates global model weights and distributes these weights to the edge devices participating in the learning. The process can be expressed by the following formula:

[0082]

[0083] Among them, W global is the global model weight, W k is the model weight of the kth edge device, and K is the number of edge devices participating in the learning.

[0084] 2) Task Similarity Measurement In order to effectively perform collaborative distillation, it is necessary to measure the similarity between different tasks. The present invention uses cosine similarity to measure the similarity between tasks, and its calculation formula is:

[0085]

[0086] Among them, S ij is the similarity measure between the i-th task and the j-th task, W i and W j are the model weight vectors of the two tasks respectively.

[0087] 3) Collaborative Optimization of Knowledge Distillation In the collaborative distillation framework, each edge device not only learns its own task, but also optimizes the global model by measuring the similarity with other tasks. The goal of the optimization is to minimize the difference between the global model and the models of each edge device while taking into account the similarity between tasks. The optimization problem can be expressed as:

[0088]

[0089] in, is the loss function, which measures the difference between the global model and the edge device model, λ k is the weight associated with edge device k, μ ij is the weight associated with the similarity between tasks i and j.

[0090] 4) Multi-task learning and personalization In the process of multi-model collaborative distillation, the model on each edge device not only learns shared knowledge, but also needs to learn personalized knowledge to solve specific tasks. The present invention achieves this by adding a personalized head on the basis of the global model. The introduction of the personalized head can be expressed by the following formula:

[0091]

[0092] in, is the final model weight on edge device k, is the personalized header weight added for edge device k.

[0093] 5) Optimization of communication efficiency In order to reduce the communication overhead in federated learning, the present invention adopts model compression technology. In addition, by uploading and downloading only the updated part of the model weights instead of the entire model weights, the communication cost is further reduced. This method can be expressed by the following formula:

[0094]

[0095] Where, ΔW k is the model weight update amount on edge device k, is the model weight in the previous iteration.

[0096] Through the multi-model collaborative distillation strategy, effective collaboration between multiple models is achieved in the edge computing environment, which not only improves the performance of each model, but also reduces communication overhead, making the multi-model collaborative knowledge distillation optimization method possible on resource-constrained edge devices.

[0097] In one embodiment, a multi-model collaborative knowledge distillation optimization method based on edge computing provides a specific process for processing data heterogeneity and imbalance, using a self-distillation method and a multi-level branch regularization technology to enhance the adaptability of the model to local data and improve the performance of the global model. This process solves the problem of inconsistent data distribution between edge devices. By performing client self-distillation on the client side and introducing multi-level branch regularization technology on the server side, the adaptability of the model to local data is enhanced, and the performance of the global model is improved. The following are the technical details of the data heterogeneity and imbalance processing process:

[0098] 1) Client-side self-distillation mechanism The self-distillation mechanism is a knowledge distillation technology introduced in the client model, which is achieved by simulating the teacher-student architecture during the training process. In this architecture, the client model is both a teacher and a student, refining and fusing knowledge through self-learning. The goal of self-distillation is to improve the generalization ability of the model on local data while reducing dependence on the global model. The client-side self-distillation mechanism is a knowledge distillation method performed on edge devices, which allows the model to improve its adaptability to local data by self-learning the output distribution of the model under different dropout masks when facing non-independent and identically distributed data. The objective function of client-side self-distillation can be expressed as:

[0099]

[0100] in, is the cross entropy loss, f(x) is the output of the client model, y is the true label, α is the weight factor, β is the weight of the distillation loss, P(z) and Q(z) are the output distributions under different dropout masks, respectively. is the KL divergence loss.

[0101] 2) Multi-level branch regularization The server uses multi-level branch regularization technology to deal with the heterogeneity and imbalance of data between clients. This method encourages the model to learn more robust feature representations at different levels by applying regularization at different levels of the model. The objective function of multi-level branch regularization is:

[0102]

[0103] Among them, H is the number of layers in the model, γ i is the regularization weight of the corresponding level, is the loss function of the i-th layer, is the aggregation loss.

[0104] 3) Heterogeneity-aware data sampling In order to further deal with data heterogeneity, the present invention adopts a heterogeneity-aware data sampling strategy. This strategy dynamically adjusts the sampling probability of samples on each device according to the distribution characteristics of the data on the edge device to ensure that the server can aggregate more representative data. The sampling strategy can be expressed by the following formula:

[0105]

[0106] Among them, P(x) is the sampling probability of sample x, d(x) is the distribution distance measure of sample x, and λ d is a parameter that controls the influence of distribution distance.

[0107] 4) Imbalance Correction For the problem of data imbalance, the present invention introduces an imbalance correction mechanism. This mechanism improves the model's ability to recognize minority samples by giving higher weights to minority samples. The loss function of imbalance correction is:

[0108]

[0109] Where C is the total number of categories, ω c is the weight of category c, is the cross entropy loss for class c.

[0110] 5) End-to-end training strategy The end-to-end training strategy for data heterogeneity and imbalance processing involves client self-distillation, multi-level branch regularization, and joint optimization of heterogeneity-aware sampling and imbalance correction. In this way, the model can effectively utilize data on edge devices while ensuring privacy and improve the performance of the global model. The objective function of end-to-end training can be expressed as:

[0111]

[0112] The data heterogeneity and imbalance processing of the present invention can effectively solve the problem of inconsistent data distribution on edge devices and improve the generalization ability and robustness of the model. Its design takes into account the actual challenges in the edge computing environment and provides technical support for the implementation of an effective multi-model collaborative knowledge distillation optimization method.

[0113] In one embodiment, a multi-model collaborative knowledge distillation optimization method based on edge computing provides a specific process for adaptive resource scheduling and dynamic adjustment of computing resources of edge devices, which is used to optimize the knowledge distillation process in the edge computing environment. Dynamically adjust resource allocation according to the computing power and network status of the edge device to optimize communication overhead and computing load. The following are the technical details of the resource adaptive scheduling process:

[0114] 1) Evaluation of edge device computing power First, the computing power of each edge device needs to be evaluated. This can be achieved by monitoring the device's CPU and memory usage, the time required to process specific tasks, and other indicators. The evaluation of computing power can be expressed as:

[0115] C e =f(U CPU ,U MEM , T task ) (15)

[0116] Among them, C e is the computing power of the edge device, f is the evaluation function, and U CPU and U MEM are the CPU and memory usage, Ttask It is the time required to process a specific task.

[0117] 2) Network status monitoring Network status monitoring is crucial for resource scheduling because it directly affects the efficiency of data transmission. Network status can be evaluated by monitoring indicators such as data packet transmission time and packet loss rate. The evaluation of network status can be expressed as:

[0118]

[0119] Among them, N s is the network status, g is the evaluation function, T trans is the transmission time of the data packet, is the packet loss rate.

[0120] 3) Resource allocation strategy Based on the computing power and network conditions of edge devices, the resource allocation strategy dynamically adjusts the resource allocation of each device. Resource allocation can be the number of processor cores, memory allocation, or network bandwidth. The resource allocation strategy can be expressed as:

[0121] R alloc =h(C e ,N s ) (17)

[0122] Among them, R alloc is the resource allocation amount, h is the allocation function, and resource allocation is determined based on computing power and network conditions.

[0123] 4) Adaptive adjustment mechanism The adaptive adjustment mechanism dynamically adjusts resource allocation according to the current system load and task requirements. This mechanism can predict future resource requirements based on the prediction model and adjust resource allocation accordingly. Adaptive adjustment can be expressed as:

[0124]

[0125] in, is the resource allocation at time t, ΔR adjustIt is the amount of resources adjusted according to the prediction results. The prediction results can be implemented by using a prediction model built with machine learning technology (neural networks such as CNN and RNN). The prediction model can predict the specific demand for resources for a specific task based on historical and real-time data. For example, a prediction model based on time series analysis may predict that during a certain period of the day, due to increased user activity, the system will need more computing resources to handle the surge in requests. Once the prediction model generates predicted values ​​for resource demand, the system will dynamically adjust resource allocation based on these predicted values. If the prediction shows that resource demand will increase, the system can pre-allocate more resources to avoid performance bottlenecks; if the prediction of resource demand will decrease, the system can reduce resource allocation, thereby saving costs and improving resource utilization efficiency.

[0126] 5) Communication Overhead Optimization In order to reduce communication overhead, the present invention adopts model compression techniques such as parameter quantization and sparse coding. In addition, by uploading and downloading only the updated part of the model weights instead of the entire model weights, the communication cost is further reduced. The optimization of communication overhead can be expressed as:

[0127]

[0128] Among them, O comm is the communication overhead, K is the number of edge devices, ΔW k is the model weight update amount for the kth device, is ΔW k The square of the norm of , which indicates the magnitude of the weight update, is the normalized communication cost of the weight model of edge device k in the previous iteration, ensuring that the communication costs of different devices and different rounds are comparable.

[0129] Through adaptive resource scheduling, it is possible to significantly reduce the communication and computing overhead in the edge computing environment while ensuring the effect of knowledge distillation. Its design takes into account the dynamic changes and heterogeneity of edge devices, providing important technical support for the realization of efficient multi-model collaborative knowledge distillation optimization methods.

[0130] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0131] The above is only a preferred implementation of a multi-model collaborative knowledge distillation optimization method based on edge computing disclosed in the present invention, and is not intended to limit the protection scope of the embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the protection scope of the embodiments of this specification.

Claims

1. A multi-model collaborative knowledge distillation optimization method based on edge computing, characterized in that: Applied in a terminal-cloud collaborative system, the terminal-cloud collaborative system includes a server and a plurality of edge devices connected to the server, each of the edge devices is connected to a plurality of clients in an area, and includes the following steps: Several edge devices respectively build local models and train the local models through the collected local data to obtain trained local models. Several edge devices upload the trained models to the server. Perform real-time performance evaluation on the local model on the edge device to determine whether the performance evaluation result meets the preset conditions. If not, update the local model. The update model includes the following: 1) adopt a multi-model collaborative distillation strategy to exchange knowledge between local models on different edge devices; 2) process data heterogeneity and imbalance, adopt a self-distillation method and multi-level branch regularization technology to enhance the model's adaptability to local data and improve global model performance; 3) perform resource adaptive scheduling and dynamically adjust the computing resources of the edge device; 4) the server aggregates the local model and sends the model update content in this round of iteration to the edge device, and provides task decision support based on the updated local model; If so, there is no need to update the local model in this iteration, and task decision support is provided based on the local model.

2. According to the multi-model collaborative knowledge distillation optimization method based on edge computing in claim 1, it is characterized in that: Real-time performance evaluation of local models on edge devices includes the following steps: Determine a performance evaluation indicator, wherein the performance evaluation indicator includes one or a combination of accuracy, recall, and F1 score; The local model output collected on the edge device is compared with the real-time data to determine whether the comparison result is lower than the preset performance threshold. If so, knowledge distillation is started to update the local model performance and determine the degree of knowledge distillation; if not, there is no need to update the local model.

3. According to the edge computing-based multi-model collaborative knowledge distillation optimization method of claim 2, it is characterized in that: The performance evaluation index adopts accuracy, and the degree of knowledge distillation is controlled by adjusting the weight factor in the distillation loss function, and the weight factor is as follows: Among them, f(·,·) is a mapping function that dynamically adjusts α, θ according to the gap between the performance evaluation index and the performance threshold. perf is the preset performance threshold.

4. According to the multi-model collaborative knowledge distillation optimization method based on edge computing in claim 1, it is characterized in that: The collaborative distillation framework is used to exchange knowledge between local models on different edge devices. The specific steps include: Knowledge exchange is carried out through federated learning. Specifically, in each iteration of federated learning, the server collects model weights from each edge device, generates a global model weight, and distributes the global model weight to the edge devices participating in the learning. The formula is as follows: Among them, W global is the global model weight, W k is the model weight of the kth edge device, and K is the number of edge devices participating in the learning; Cosine similarity is used to measure the similarity between tasks, and its calculation formula is: Among them, S ij is the similarity measure between the i-th task and the j-th task, W i and W j are the model weight vectors of the two tasks respectively; Minimizing the difference between the global model and the models of each edge device while taking into account the similarity between tasks, the optimization problem can be expressed as: in, is the loss function, which measures the difference between the global model and the edge device model, λ k is the weight associated with edge device k, μ ij is the weight associated with the similarity between tasks i and j; A personalized head is introduced based on the global model. The personalized head is expressed by the following formula: in, is the final model weight on edge device k, is the personalized header weight added for edge device k.

5. According to the edge computing-based multi-model collaborative knowledge distillation optimization method of claim 4, it is characterized in that: The following steps are also included: In federated learning, only the updated part of the model weights is uploaded and downloaded between the edge device and the global model.

6. According to the edge computing-based multi-model collaborative knowledge distillation optimization method of claim 1, it is characterized in that: The self-distillation method and multi-level branch regularization technology are used to enhance the model's adaptability to local data and improve the global model performance. The specific steps include: By performing client self-distillation on the client, the objective function of client self-distillation can be expressed as: in, is the cross entropy loss, f(x) is the output of the client model, y is the true label, α is the weight factor, β is the weight of the distillation loss, P(z) and Q(z) are the output distributions under different dropout masks, respectively. is the KL divergence loss; The server uses multi-level branch regularization technology to deal with the heterogeneity and imbalance of data between clients. The objective function of multi-level branch regularization is: Among them, H is the number of layers in the model, γ i is the regularization weight of the corresponding level, is the loss function of the i-th layer, is the aggregation loss; A heterogeneity-aware data sampling strategy is adopted. According to the distribution characteristics of the data on the edge devices, the sampling probability of samples on each device is dynamically adjusted to ensure that the server can aggregate more representative data. The sampling strategy can be expressed by the following formula: Among them, P(x) is the sampling probability of sample x, d(x) is the distribution distance measure of sample x, and λ d is a parameter that controls the effect of distribution distance; An imbalance correction mechanism is introduced, which improves the model's ability to recognize minority samples by giving higher weights to minority samples. The loss function of imbalance correction is: Where C is the total number of categories, ω c is the weight of category c, is the cross entropy loss for category c; The overall objective function of end-to-end training can be expressed as:

7. The multi-model collaborative knowledge distillation optimization method based on edge computing according to claim 1 is characterized in that: The adaptive resource scheduling and dynamic adjustment of computing resources of edge devices specifically include the following steps: Evaluate the computing power and network status of each edge device. The evaluation indicators of computing power include the CPU and memory usage of the device and the time required to process a specific task. The evaluation indicators of network status include transmission time and packet loss rate. The current system load is determined based on the evaluation results of computing power and network conditions, and the resource requirements of the local model to complete the task are predicted. Resources are dynamically allocated based on the current system load and predicted resource requirements to optimize communication overhead and computing load.

8. The multi-model collaborative knowledge distillation optimization method based on edge computing according to claim 7 is characterized in that: The optimization of communication overhead can be expressed as: Among them, O comm is the communication overhead, K is the number of edge devices, ΔW k is the model weight update amount of the kth edge device, is ΔW k The square of the norm of , which indicates the magnitude of the weight update, is the normalized communication cost of the weight model of edge device k in the previous iteration, ensuring that the communication costs of different devices and different rounds are comparable.

Images