Knowledge distillation optimization method guided by adaptive noise sample in end-cloud collaborative environment
By introducing adaptive noise sample guidance and ant colony optimization algorithms in the end-cloud collaborative environment, the problems of inefficiency and insufficient resources of traditional federated learning in Non-IID data processing are solved, and efficient knowledge distillation and resource optimization are achieved, improving the Model performance and communication efficiency are suitable for a variety of intelligent system applications.
Patent Information
- Application Number
- CN202411075649.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
In the end-cloud collaborative environment, traditional federated learning algorithms are inefficient when processing Non-IID data, making it difficult to effectively integrate model updates from different clients, and the computing resources of end-equipment are limited, which increases communication costs and computing burden. The existing knowledge distillation method The special needs of the end-cloud collaborative environment are not fully considered, especially in dealing with the heterogeneity of client data distribution.
Propose a knowledge distillation optimization method for adaptive noise sample guidance in the end-cloud collaborative environment. By training the teacher model on the end-to-end device and using the adaptive distillation intensity adjustment mechanism, noise samples related to the end-to-end device are generated, combining weight comparison regularization and federal Average model aggregation strategy, integrate the end device model, and introduce ant colony optimization algorithm for resource optimization configuration, and dynamically adjust computing tasks and cloud resource allocation.
Significantly improves model training efficiency and communication efficiency, reduces the communication cost of model updates, and improves the performance of global models. It is suitable for intelligent system application scenarios such as intelligent health, intelligent manufacturing and traffic management, and can quickly adapt to data distribution changes and protect data privacy. .
Smart Images

Figure CN119005293A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of end-cloud collaboration technology, and in particular to a knowledge distillation optimization method guided by adaptive noise samples in an end-cloud collaboration environment. Background Art
[0002] In today's information technology field, the end-cloud collaborative architecture has become the core technology for improving the data processing capabilities of smart devices. Under this architecture, model training efficiency and communication costs have become technical problems that need to be solved urgently. Traditional federated learning (FL) algorithms achieve distributed learning by averaging model parameters of multiple clients, but they have obvious shortcomings when processing non-independent and identically distributed (Non-IID) data. In practical applications, client data distribution is often heterogeneous, which makes it difficult for the global model to capture the unique data characteristics of each client, which in turn affects the model's generalization ability and performance.
[0003] At the same time, due to the limited computing resources of the end device, the traditional FL algorithm requires the client to perform a lot of model training and parameter updates, which not only increases the computing burden of the end device, but also increases the communication cost. The client needs to communicate frequently with the cloud server to transmit model updates, which has problems in data privacy protection and network bandwidth utilization. Especially in the mobile network environment, the mobility of the client and the instability of the network connection further aggravate the problem.
[0004] As an effective model compression and acceleration technology, knowledge distillation has shown significant advantages in single model training. However, in an end-cloud collaborative environment, how to effectively use knowledge distillation technology to optimize model training, especially when dealing with Non-IID data distribution and limited computing resources, remains an unresolved problem. Existing knowledge distillation methods mainly focus on how to extract knowledge from large models to train small models, but do not fully consider the special needs of the end-cloud collaborative environment.
[0005] Recent studies have proposed some solutions that attempt to improve model performance through techniques such as regularization, model selection, or knowledge distillation. For example, some methods reduce the model's dependence on a specific data distribution by applying local regularization on the client, while others reduce communication costs by selectively aggregating client models. However, the above methods still have limitations when dealing with the heterogeneity of client data distribution. On the end device, the model is prone to overfitting; on the cloud, it is difficult for the model to effectively integrate diverse data from different clients, resulting in limited performance of the final model.
[0006] In summary, the existing technologies have the following major problems and defects in model training in the end-cloud collaborative environment: First, the traditional FL algorithm is inefficient when processing Non-IID data distribution, and it is difficult to effectively integrate model updates from different clients; second, the computing resources of the end device are limited, and the FL algorithm requires frequent model updates and communications, which increases the communication cost and computing burden of the end device; third, the existing knowledge distillation method does not fully consider the special needs of the end-cloud collaborative environment, especially in dealing with the heterogeneity of client data distribution; finally, there is a lack of an optimization strategy that can effectively utilize the computing power of the end device, reduce the computing pressure on the cloud, and improve the global model performance while maintaining communication efficiency. Summary of the invention
[0007] In order to solve the above technical problems, the present invention proposes a knowledge distillation optimization method guided by adaptive noise samples in an end-cloud collaborative environment, which can improve the model training efficiency and communication efficiency under the end-cloud collaborative architecture while protecting data privacy; in addition, the ant colony optimization algorithm is introduced to optimize the resource configuration of the end-cloud collaborative system, dynamically adjust the computing tasks of the end device and the allocation of cloud resources, achieve balanced distribution of computing load and improvement of model performance, and provide an efficient and secure solution for knowledge distillation training under the end-cloud collaborative architecture. In practical applications, the present invention can significantly reduce the communication cost in the model update process and improve the performance of the global model, and is suitable for intelligent system application scenarios such as smart health, smart manufacturing and traffic management.
[0008] In order to achieve the above object, the technical solution of the present invention is as follows:
[0009] The method for optimizing knowledge distillation guided by adaptive noise samples in a device-cloud collaborative environment is characterized in that it is applied to a device-cloud collaborative system, wherein the device-cloud collaborative system includes a plurality of end devices that are communicatively connected to a cloud server, and includes the following steps:
[0010] On each end device, the corresponding teacher model is trained using the local data set to obtain a trained teacher model; on each end device, the student model is trained using the trained teacher model through knowledge distillation, and the distillation strength is dynamically adjusted using an adaptive distillation strength adjustment mechanism during training to obtain a trained student model;
[0011] The cloud server generates noise samples related to the student model in the end device, and updates the noise samples by minimizing the difference between the noise sample distribution and the real data distribution through an adaptive algorithm. The updated noise samples are input into the student model for training to obtain the corresponding end device model.
[0012] The cloud server uses a model aggregation strategy of weighted comparison regularization and federated averaging to integrate the device model updates of several devices, form a global model and send it to all devices.
[0013] The terminal devices use the sent global model to complete the prediction task, and feed back the prediction results, and iteratively optimize the global model according to the fed-back prediction results.
[0014] Preferably, the distillation loss function used in knowledge distillation is defined as follows:
[0015]
[0016] Among them, λ is a hyperparameter used to balance the two loss terms, H soft (·,·) and H kl (·,·) represent the cross entropy loss and KL divergence function respectively, and softmax(M(x)) represents the output of model M for sample x.
[0017] Preferably, the adaptive distillation strength adjustment mechanism is as follows:
[0018]
[0019] Among them, S α is the step size for adjusting the distillation intensity, and Represent the accuracy of the student model and the teacher model on the validation set respectively.
[0020] Preferably, the cloud server generates noise samples related to the student model in the terminal device, and updates the noise samples by minimizing the difference between the noise sample distribution and the real data distribution through an adaptive algorithm, and inputs the updated noise samples into the student model for training, which specifically includes the following steps:
[0021] The process of generating noise samples can be expressed as:
[0022]
[0023] Among them, P noise is the noise distribution generating function with parameter θ. The initial noise sample can be obtained through the teacher model M t Output To guide, that is:
[0024]
[0025] Among them, g(·) is the mapping function from the teacher model output to the soft label,
[0026] An adaptive algorithm is used to minimize the difference between the noise sample distribution and the true data distribution. The formula is as follows:
[0027]
[0028] The maximization of mutual information is achieved through the gradient ascent algorithm, namely:
[0029]
[0030] Where η is the learning rate, represents the gradient,
[0031] The distillation process can be expressed as:
[0032]
[0033] Here, H(·,·) represents the cross entropy loss.
[0034] Preferably, the cloud server uses a model aggregation strategy of weighted comparison regularization and federated averaging to integrate the terminal device model updates of several terminal devices to form a global model, which specifically includes the following steps:
[0035] The weight contrast regularization term can be expressed as:
[0036]
[0037] Among them, λ j is a hyperparameter used to balance the weight differences of different layers, M global,j and M i,j are the weights of the global model and the i-th end device model at the j-th layer,
[0038] The federated average aggregation can be expressed as:
[0039]
[0040] Where N is the number of participating devices, M i is the update of the i-th end device model,
[0041] By aggregating the loss function To evaluate the model performance, and adjust the hyperparameter λ in the weight comparison regularization term based on the model performance feedback j , the formula is as follows:
[0042]
[0043] Where η is the learning rate, Denotes the distillation loss function with respect to λ j The gradient of represents the global dataset, where w i is the weight of the ith device, is the model M on the end device i i In local dataset Local losses on.
[0044] Preferably, it also includes introducing an ant colony optimization algorithm to optimize resource configuration of the end-cloud collaborative system, and dynamically adjust the end device computing tasks and cloud resource allocation, which specifically includes the following steps:
[0045] Assume T = {t1, t2, …, t N} represents the task set on the end device, R = {r1, r2, …, r M} represents the available cloud resource set, each task t i It needs to consume a certain amount of computing resources c i , and each cloud resource r j With corresponding computing power C j , the optimization goal is to find a mapping function f:T→R so that the computing requirements of all tasks are met and do not exceed the total capacity of cloud resources, that is:
[0046]
[0047] Where N is the number of end devices, M is the number of cloud resources, and c i is the amount of computing resources required to execute the task on the end device i, C j is the computing power of the cloud resource j,
[0048] The ant selects the next node to visit based on the pheromone concentration and heuristic information on the path. The pheromone update rule is as follows:
[0049]
[0050] Where ρ is the pheromone volatility coefficient, represents the pheromone concentration on the path (i, j) in the tth iteration, Δτ ij It represents the amount of pheromone deposited by the ant on the path (i, j), which is related to the task t i Assign to resource r j The efficiency is proportional to the transfer probability P when the ant chooses the next resource. ij Determined by the following formula:
[0051]
[0052] in, is the component of pheromone concentration, α reflects the concentration of pheromone on path (i, j), is the component of heuristic information, β is the weight parameter of heuristic information, and are the pheromone concentration and heuristic information when the ant chooses to transfer from path k to path j, N iRepresents the task t i Start with a collection of selectable resources,
[0053] In each iteration, the ants move according to the transition probability P. ij Select the next resource to be accessed, determine the iteration termination condition, until the iteration number threshold is reached or other termination conditions are met, and assign the corresponding task to the corresponding resource. The decision-making process of task allocation can be expressed as:
[0054]
[0055] Among them, E(r j ) indicates resource r j Energy consumption, D(t i ,r j ) represents task t i Assign to resource r j The resulting delay.
[0056] Based on the above technical solution, the beneficial effects of the present invention are:
[0057] 1) The present invention optimizes the distribution of Non-IID data: The weighted contrast regularization and federated average model aggregation strategy adopted by the present invention effectively alleviate the challenges brought by the distribution of Non-IID data. Experimental results show that on the CIFAR-100 dataset, compared with the traditional processing method CKDF, the stability and performance of the model are significantly improved when processing the distribution of Non-IID data, and the average accuracy is increased by 2.5%.
[0058] 2) The present invention reduces model training costs: Through the knowledge distillation optimization strategy of the present invention, the terminal device can efficiently extract significant knowledge under limited computing resources, thereby reducing dependence on cloud computing resources. Compared with traditional methods, the present invention further reduces training costs while maintaining model performance, which has certain practical value for terminal devices with limited resources.
[0059] 3) The present invention reduces the reliance on labeled data: Through the self-distillation strategy and noise sample generation technology, the present invention reduces the need for a large amount of labeled data, allowing model training to be performed at a lower data labeling cost. This means that when achieving the same level of model performance, the amount of labeled data required can be greatly reduced, thereby significantly reducing the cost and time of data labeling.
[0060] 4) Strong adaptability of the present invention: The technical solution of the present invention enables the end-cloud collaborative learning system to better adapt to the dynamic changes in data distribution and fluctuations in the communication environment. By introducing a dynamic adjustment mechanism, the system can quickly respond to environmental changes and maintain or improve model performance. When faced with rapid changes in data distribution, the method of the present invention can quickly adapt to new data distributions, while traditional learning methods may require more time and resources to adapt to these changes.
[0061] 5) The present invention protects the data privacy of the end device: The present invention reduces the data transmission between the end device and the cloud by performing knowledge extraction and model training on the end device, thereby reducing the risk of data leakage. Experiments show that in the classification task of the CIFAR-100 dataset, by applying the self-distillation strategy on the end device, the present invention reduces the data transmission requirements by 15% compared with the traditional federated learning method FedAvg, effectively protecting user privacy. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 is a flow chart of a knowledge distillation optimization method guided by adaptive noise samples in a device-cloud collaborative environment in an embodiment;
[0063] Figure 2 The present invention is a schematic block diagram of a knowledge distillation optimization method process guided by adaptive noise samples in a terminal-cloud collaborative environment in an embodiment. DETAILED DESCRIPTION
[0064] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0065] like Figure 1 , 2 As shown, this embodiment provides a knowledge distillation optimization method guided by adaptive noise samples in a terminal-cloud collaborative environment, which is applied to a terminal-cloud collaborative system for intelligent health monitoring of users. The terminal-cloud collaborative system includes several terminal devices connected to a cloud server for communication, including the following steps:
[0066] Step 1: On each terminal device, the corresponding teacher model is trained through the local data set to obtain a trained teacher model; on each terminal device, the trained teacher model is used to train the student model through knowledge distillation, and the adaptive distillation strength adjustment mechanism is used during training to dynamically adjust the distillation strength to obtain a trained student model.
[0067] In this embodiment, health-related data such as heart rate, number of steps, sleep quality, etc. are collected on the end devices (such as smartphones and wearable devices), and the self-distillation strategy is applied to extract key health monitoring knowledge and enhance the generalization ability and robustness of the model. The self-distillation strategy is described in detail as follows, which extracts core knowledge from the end devices to enhance the robustness of the model and reduce the communication requirements with the cloud.
[0068] 1) Selection and training of teacher model: The first step of the self-distillation strategy is to select a teacher model M with excellent performance t , which is based on the local data set on the end device Train and evaluate its performance through evaluation indicators such as accuracy and loss value. The selection process of the teacher model can be expressed as:
[0069]
[0070] in, represents the set of all potential models, is the loss function used to measure the performance of the model on the local dataset.
[0071] 2) Initialization of the student model: student model M s It is a model with simplified structure and reduced parameters, which aims to extract core knowledge from the teacher model. The student model can be initialized in many ways, such as random initialization, fine-tuning based on pre-trained models, or using transfer learning methods. The initialization process of the student model can be expressed as:
[0072]
[0073] Among them, f init is the initialization function, Represents the set of possible initialization strategies for the student model.
[0074] 3) Design of distillation loss function: Distillation loss function Used to measure the difference between the output of the student model and the output of the teacher model. A composite loss function that combines cross entropy loss and KL divergence is used to ensure that the student model not only learns the correct category labels, but also captures the probability distribution of the teacher model output. The distillation loss function is defined as follows:
[0075]
[0076] Among them, λ is a hyperparameter used to balance the two loss terms, H soft (·,·) and H kl (·,·) represent the cross entropy loss and KL divergence function respectively, and softmax(M(x)) represents the output of model M for sample x.
[0077] 4) Adaptive distillation strength adjustment: In order to further improve the performance of the student model, an adaptive distillation strength adjustment mechanism is introduced. This mechanism dynamically adjusts the distillation strength according to the performance of the student model on the validation set. When the performance of the student model is lower than that of the teacher model, the distillation strength is increased to force the student model to learn more knowledge; when the performance of the student model is close to or exceeds that of the teacher model, the distillation strength is reduced to encourage the student model to explore new knowledge. The formula for adaptive distillation strength adjustment is:
[0078]
[0079] Among them, S α is the step size for adjusting the distillation intensity, and Represent the accuracy of the student model and the teacher model on the validation set respectively.
[0080] Through the above self-distillation strategy, the end device can effectively extract significant knowledge and maintain the robustness of the model under limited computing resources, which not only improves the performance of the model on the end device, but also lays a solid foundation for knowledge aggregation in the cloud. The subsequent method description will further describe in detail the noise sample generation and adaptive algorithm, as well as the working principle of weight contrast regularization and federated average model aggregation and its role in the overall technical solution.
[0081] Step 2: The cloud server generates noise samples related to the student model in the terminal device, and updates the noise samples by minimizing the difference between the noise sample distribution and the true data distribution through an adaptive algorithm. The updated noise samples are input into the student model for training to obtain the corresponding terminal device model.
[0082] In this embodiment, considering that the data distribution of the end devices may be different, direct aggregation may lead to a decrease in the performance of the global model. In order to solve this problem, a noise sample generation and adaptive algorithm is proposed, which aims to generate noise samples on the cloud server, simulate the distribution of physiological data of different users, and optimize the sample quality through an adaptive algorithm. The specific description is as follows:
[0083] 1) Generation of noise samples: Noise samples It is generated in the cloud to simulate the data distribution on the end device. These samples are not real data, but can reflect the characteristics of data distribution. The process of generating noise samples can be expressed as:
[0084]
[0085] Among them, P noise is the noise distribution generating function with parameter θ. The initial noise sample can be obtained through the teacher model M t Output To guide, that is:
[0086]
[0087] Here, g(·) is the mapping function from the teacher model output to the soft label.
[0088] 2) Optimization of adaptive algorithm: In order to improve the quality of noise samples and make them closer to the real data distribution P real , an adaptive algorithm is used to adjust the distribution parameter θ of the noise sample. The goal of the adaptive algorithm is to minimize the difference between the noise sample distribution and the true data distribution, which can be achieved by maximizing the mutual information between the two:
[0089]
[0090] Maximization of mutual information can be achieved through the gradient ascent algorithm, namely:
[0091]
[0092] Where η is the learning rate, Represents the gradient.
[0093] 3) Application of noise samples in the distillation process: Noise samples play a key role in the distillation process. They are not only used to simulate the data distribution of the end device, but also used to guide the student model M s During the distillation process, the soft labels of the noise samples It is used as an additional target of the student model, which enables the student model to learn more generalized feature representations. The distillation process can be expressed as:
[0094]
[0095] Here, H(·,·) represents the cross entropy loss.
[0096] By generating noise samples and adaptive algorithms, the cloud server can effectively simulate the data distribution on the end device, providing a more realistic sample basis for knowledge distillation. This method not only improves the knowledge transfer efficiency of the end device model, but also enhances the adaptability of the cloud model to diversified data, thereby improving the performance of the global model under the end-cloud collaborative architecture.
[0097] In step 3, the cloud server uses the model aggregation strategy of weighted comparison regularization and federated averaging to integrate the device model updates of several devices, form a global model and send it to all devices.
[0098] In this embodiment, in order to improve the performance of the global model and reduce the problems caused by non-independent and identically distributed (Non-IID) data, a model aggregation strategy combining weighted contrast regularization and federated averaging is proposed to integrate model updates from multiple end devices to form a global health monitoring model. The specific instructions are as follows:
[0099] 1) Design of weight contrast regularization term: Weight contrast regularization term R weight It is used to quantify and reduce the weight differences between different end-device models. This is done by comparing each end-device model M i With the global model M global The weight contrast regularization term can be expressed as:
[0100]
[0101] Among them, λ j is a hyperparameter used to balance the weight differences of different layers, M global,j and M i,j are the weights of the global model and the i-th end device model at the j-th layer, respectively.
[0102] 2) Implementation of federated average model aggregation: Federated average model aggregation is a key step in end-cloud collaborative learning. It forms a global model by aggregating model updates from all participating end devices. In the present invention, federated average aggregation not only considers the average of model parameters, but also combines weight contrast regularization terms to optimize the aggregation process. Federated average aggregation can be expressed as:
[0103]
[0104] Where N is the number of participating devices, M i is the update of the i-th end device model.
[0105] 3) Optimization during aggregation: In order to optimize the global model during the aggregation process, an iterative optimization strategy is adopted. After each aggregation, the global model will be used to guide the training of the end device model and the aggregation loss function will be used to optimize the global model during the aggregation process. Then, adjust the hyperparameter λ in the weight comparison regularization term based on the model performance feedback. j . This process can be expressed as:
[0106]
[0107] Where η is the learning rate, Denotes the distillation loss function with respect to λ j The gradient of Represents a global dataset. where w iis the weight of the ith device, is the model M on the end device i i In local dataset Local losses on.
[0108] Through weighted contrast regularization and federated average model aggregation, we can effectively integrate the updates of the end device model and alleviate the problems caused by Non-IID data distribution. This method not only improves the performance of the global model, but also enhances the model's adaptability to diverse data, thereby improving the overall learning efficiency under the end-cloud collaborative architecture.
[0109] Step 4: Some of the terminal devices use the sent global model to complete the prediction task, and feed back the prediction results, and iteratively optimize the global model according to the fed-back prediction results.
[0110] In this embodiment, the end device uses the global model (global health monitoring model) sent down to perform real-time health monitoring. Users feedback the monitoring results, and the model is iteratively optimized based on the feedback.
[0111] In an embodiment, the knowledge distillation optimization method guided by adaptive noise samples in a device-cloud collaborative environment further includes introducing an ant colony optimization algorithm to optimize resource configuration of the device-cloud collaborative system, and dynamically adjusting the allocation of device computing tasks and cloud resources, specifically including the following steps:
[0112] 1) Problem modeling and ant colony initialization: For the problem of end-cloud resource optimization, we must first build a reasonable mathematical model. Let T = {t1, t2, …, t N} represents the task set on the end device, R = {r1, r2, …, r M} represents the set of available cloud resources. Each task t i It needs to consume a certain amount of computing resources c i , and each cloud resource r j With corresponding computing power C j The optimization goal is to find a mapping function f:T→R so that the computing requirements of all tasks are met and do not exceed the total capacity of cloud resources, that is:
[0113]
[0114] Where N is the number of end devices, M is the number of cloud resources, and c i is the amount of computing resources required to execute the task on the end device i, C j is the computing power of cloud resource j. At initialization, each ant is located on a randomly selected terminal device, ready to start looking for the optimal resource allocation solution.
[0115] 2) Pheromone update and transition probability: In the ACO algorithm, ants choose the next node to visit based on the pheromone concentration and heuristic information on the path. Pheromones are a virtual chemical substance used to guide the search direction of ants. The pheromone update rules are as follows:
[0116]
[0117] Where ρ is the pheromone volatility coefficient, represents the pheromone concentration on the path (i, j) in the tth iteration, Δτ ij It represents the amount of pheromone deposited by the ant on the path (i, j), which is related to the task t i Assign to resource r j When the ant chooses the next resource, the transfer probability P ij Determined by the following formula:
[0118]
[0119] in, is the component of pheromone concentration, α reflects the concentration of pheromone on path (i, j). The higher the pheromone concentration, the more ants choose this path, indicating that this is a better solution. is the weight of the heuristic information, and β is the weight parameter of the heuristic information. Heuristic information is usually based on specific properties of the problem, such as the availability of resources or the urgency of the task, to guide the ants' choices. and are the pheromone concentration and heuristic information when the ant chooses to transfer from path k to path j. i Represents the task t i Departures a selectable collection of resources.
[0120] 3) Task allocation and resource scheduling: In each iteration, ants allocate resources according to the transfer probability P. ij Select the next resource to be accessed and assign the corresponding task to it. The decision process of task assignment can be expressed as:
[0121]
[0122] Among them, E(r j ) indicates resource r j Energy consumption, D(t i ,r j ) represents task t i Assign to resource r j The goal of the algorithm is to minimize the energy consumption and delay of the entire system while meeting the task requirements.
[0123] 4) Convergence judgment and resource optimization configuration: Convergence judgment is an important part of the ACO algorithm, which is used to evaluate whether the algorithm is close to the optimal solution. If the quality of the solution does not improve significantly after multiple consecutive iterations, or the speed of improvement of the solution quality is lower than the preset threshold, the algorithm is considered to have reached a convergence state. In addition, in order to avoid infinite loops, the algorithm can also set a maximum number of iterations T max . Through iterative search and convergence judgment, the ACO algorithm can achieve optimal resource allocation in the end-cloud collaborative environment. The algorithm not only focuses on the satisfaction of the needs of a single task, but also pays attention to the performance and efficiency of the entire system. By dynamically adjusting the resource allocation plan, it achieves the goal of minimizing system energy consumption and latency. This method provides an effective optimization strategy for resource management in the end-cloud collaborative environment, which helps to improve the overall performance of the system.
[0124] The ant colony optimization algorithm provides a practical solution for optimizing the configuration of end-cloud resources. By simulating the foraging behavior of ants, the algorithm can find the best match between tasks and resources in a complex end-cloud environment, achieving efficient resource utilization and maximizing system performance. This method provides new ideas and means for resource management in an end-cloud collaborative environment, which is of great significance for improving system stability and efficiency.
[0125] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0126] The above description is only a preferred implementation of the knowledge distillation optimization method guided by adaptive noise samples in the end-cloud collaborative environment disclosed by the present invention, and is not intended to limit the protection scope of the embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included in the protection scope of the embodiments of this specification.
Claims
1. The knowledge distillation optimization method guided by adaptive noise samples in the end-cloud collaborative environment is characterized by: Applied to a terminal-cloud collaborative system, the terminal-cloud collaborative system includes a number of terminal devices that are connected to a cloud server for communication, including the following steps: On each end device, the corresponding teacher model is trained using the local data set to obtain a trained teacher model; on each end device, the student model is trained using the trained teacher model through knowledge distillation, and the distillation strength is dynamically adjusted using an adaptive distillation strength adjustment mechanism during training to obtain a trained student model; The cloud server generates noise samples related to the student model in the end device, and updates the noise samples by minimizing the difference between the noise sample distribution and the real data distribution through an adaptive algorithm. The updated noise samples are input into the student model for training to obtain the corresponding end device model. The cloud server uses a model aggregation strategy of weighted comparison regularization and federated averaging to integrate the device model updates of several devices, form a global model and send it to all devices. The terminal devices use the sent global model to complete the prediction task, and feed back the prediction results, and iteratively optimize the global model according to the fed-back prediction results.
2. According to claim 1, the knowledge distillation optimization method guided by adaptive noise samples in the end-cloud collaborative environment is characterized in that: The distillation loss function used in knowledge distillation is defined as follows: Among them, λ is a hyperparameter used to balance the two loss terms, H soft (·,·) and H kl (·,·) represent the cross entropy loss and KL divergence function respectively, and softmax(M(x)) represents the output of model M for sample x.
3. The knowledge distillation optimization method guided by adaptive noise samples in the end-cloud collaborative environment according to claim 2 is characterized in that: Adaptive distillation strength adjustment mechanism, the formula is as follows: Among them, S α is the step size for adjusting the distillation intensity, and Represent the accuracy of the student model and the teacher model on the validation set respectively.
4. The knowledge distillation optimization method guided by adaptive noise samples in a terminal-cloud collaborative environment according to claim 1 is characterized in that: The cloud server generates noise samples related to the student model in the terminal device, and updates the noise samples by minimizing the difference between the noise sample distribution and the real data distribution through an adaptive algorithm. The updated noise samples are input into the student model for training, which specifically includes the following steps: The process of generating noise samples can be expressed as: Among them, P noise is the noise distribution generating function with parameter θ. The initial noise sample can be obtained through the teacher model M t Output To guide, that is: Among them, g(·) is the mapping function from the teacher model output to the soft label, An adaptive algorithm is used to minimize the difference between the noise sample distribution and the true data distribution. The formula is as follows: The maximization of mutual information is achieved through the gradient ascent algorithm, namely: Where η is the learning rate, represents the gradient, The distillation process can be expressed as: Here, H(·,·) represents the cross entropy loss.
5. The knowledge distillation optimization method guided by adaptive noise samples in a device-cloud collaborative environment according to claim 1 is characterized in that: The cloud server uses the model aggregation strategy of weighted comparison regularization and federated averaging to integrate the device model updates of several devices to form a global model, which includes the following steps: The weight contrast regularization term can be expressed as: Among them, λ j is a hyperparameter used to balance the weight differences of different layers, M global,j and M ,j are the weights of the global model and the i-th end device model at the j-th layer, The federated average aggregation can be expressed as: Where N is the number of participating devices, M i is the update of the i-th end device model, By aggregating the loss function To evaluate the model performance, and adjust the hyperparameter λ in the weight comparison regularization term based on the model performance feedback j , the formula is as follows: Where η is the learning rate, Denotes the distillation loss function with respect to λ j The gradient of represents the global dataset, where w i is the weight of the ith device, is the model M on the end device i i In local dataset Local losses on.
6. The knowledge distillation optimization method guided by adaptive noise samples in a terminal-cloud collaborative environment according to claim 1 is characterized in that: It also includes introducing an ant colony optimization algorithm to optimize resource configuration of the end-cloud collaborative system and dynamically adjust the end device computing tasks and cloud resource allocation, specifically including the following steps: Assume T = {t1, t2, …, t N } represents the task set on the end device, R = {r1, r2, …, r M } represents the available cloud resource set, each task t i It needs to consume a certain amount of computing resources c i , and each cloud resource r j With corresponding computing power C j , the optimization goal is to find a mapping function f:T→R so that the computing requirements of all tasks are met and do not exceed the total capacity of cloud resources, that is: Where N is the number of end devices, M is the number of cloud resources, and c i is the amount of computing resources required to execute the task on the end device i, C j is the computing power of the cloud resource j, The ant selects the next node to visit based on the pheromone concentration and heuristic information on the path. The pheromone update rule is as follows: Where ρ is the pheromone volatility coefficient, represents the pheromone concentration on the path (i, j) in the tth iteration, Δτ ij It represents the amount of pheromone deposited by the ant on the path (i, j), which is related to the task t i Assign to resource r j The efficiency is proportional to the transfer probability P when the ant chooses the next resource. ij Determined by the following formula: in, is the component of pheromone concentration, α reflects the concentration of pheromone on path (i, j0, is the component of heuristic information, β is the weight parameter of heuristic information, and are the pheromone concentration and heuristic information when the ant chooses to transfer from path k to path j, N i Represents the task t i Start with a collection of selectable resources, In each iteration, the ants move according to the transition probability P ij Select the next resource to be accessed, determine the iteration termination condition, until the iteration number threshold is reached or other termination conditions are met, and assign the corresponding task to the corresponding resource. The decision-making process of task allocation can be expressed as: Among them, E(r j ) indicates resource r j Energy consumption, D(t i ,r j ) represents task t i Assign to resource r j The resulting delay.
Citation Information
Cited By
Cloud edge size model collaboration method and system based on federal learning, and electronic equipment
CN119496824A
Distributed single-mode small model data synchronization method applied to intelligent electric energy meter
CN120317150A
Distributed Single-Modal Small Model Data Synchronization Method Applied to Smart Electric Energy Meters
CN120317150B
