Semi-asynchronous federated learning processing method and system based on heterogeneous perception scheduling

Abstract

The invention discloses a semi-asynchronous federated learning processing method and system based on heterogeneous perception scheduling. A parameter server firstly initializes global model parameters and an aggregation time threshold, and distributes a model to an edge end device. And after receiving, the side end equipment performs local iteration updating, calculates a comprehensive importance value and uploads the comprehensive importance value to the server, and notifies that the server prepares to participate in aggregation. And when the time of the server reaches a threshold value, recording the updated equipment as a pre-scheduling set, selecting part of equipment through a scheduling strategy to upload local update, and recording the lagging degree. And the server aggregates the received update to generate a new global model, and adjusts the time threshold of the next round according to the completion condition of the current round of equipment. And the new model is broadcasted to the edge end equipment, and the next round of iteration is carried out until the model converges. According to the method, the time threshold is used as an aggregation boundary, waiting time delay between devices is effectively reduced, model convergence is accelerated, local optimum is avoided, and model precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed collaborative training, and particularly to a semi-asynchronous federated learning processing method and system based on heterogeneous perception scheduling. Background Art

[0002] With the rapid development of the Internet of Everything and edge computing technologies, machine learning is gradually transforming from a data-centric centralized training mode to a distributed training mode. Traditional centralized learning relies on collecting all data to a central server for training. In this process, not only data privacy and security issues are faced, but also huge communication and storage costs are required. Against this background, federated learning (FL) without uploading local data is proposed. FL allows users (devices) to train based on local data and only upload the local model or its updates after training, avoiding transmitting the original data to the cloud server. This method can not only improve data security but also help reduce communication load. In many practical scenarios of federated learning applications, such as intelligent transportation, smart city, and intelligent healthcare scenarios, a large amount of data is generated on terminal devices (such as smartphones, sensors, and edge servers). There may be significant differences in computing power, data distribution, and communication bandwidth among these edge devices, making efficient federated collaborative optimization extremely challenging. Federated learning under these differences is also called heterogeneous federated learning (HFL). The heterogeneity is divided into two categories: system heterogeneity and statistical heterogeneity. The differences in system heterogeneity are reflected in the hardware conditions (CPU, GPU, memory, and network communication capabilities, etc.) of each device. Statistical heterogeneity refers to the difference in the data sets on devices, that is, the so-called non-independent and identically distributed (Non-IID) problem. System heterogeneity greatly reduces the speed of joint model training, while statistical heterogeneity seriously affects the number of communications required for the model to be trained to a specified accuracy, resulting in a sharp increase in communication overhead. Therefore, how to achieve low-cost and efficient model training while respecting data privacy and facing system heterogeneity and statistical heterogeneity has become a highly concerned issue in federated learning.

[0003] Difficulty in balancing training efficiency and model accuracy: Traditional synchronous federated learning (FL) methods usually need to aggregate the local model updates of all edge devices to maintain model accuracy. However, in a heterogeneous training environment, due to the significant differences in computing power among different devices, the training latency of each round of iteration is often limited by the slowest device, resulting in a decrease in overall training efficiency. In addition, the all-device aggregation mechanism also incurs high communication overhead. To improve training efficiency, researchers have begun to explore asynchronous federated learning methods. The core idea is to allow the server to perform aggregation immediately after receiving the model updates uploaded by some devices, thereby reducing waiting time and increasing system throughput. However, this asynchronous mechanism faces severe challenges in statistical heterogeneous scenarios - the time difference in device updates will introduce a lag effect, making it difficult to guarantee the convergence of the global model.

[0004] Poor aggregation performance of some devices: In current research, the strategies for some devices to participate in aggregation usually evaluate the importance of updates based on a single metric (such as loss value, model update paradigm, or update lag) generated during the local training process. However, this single-dimensional evaluation mechanism is difficult to ensure that the data distribution of the selected device set can effectively approximate the global distribution, and it cannot properly handle the volatility and diversity of device contributions in heterogeneous environments. At the same time, to address the problem of non-independent and identically distributed (Non-IID) data, some studies introduce data diversity or data volume as screening criteria in device scheduling, preferentially selecting devices with a wide coverage of data distribution or a large amount of local samples to participate in aggregation. However, these methods only focus on data heterogeneity and ignore the heterogeneity of the devices themselves (such as computing power, communication conditions, etc.), which may exacerbate the unfairness of the system and marginalize resource-constrained devices. Therefore, how to co-optimize the impacts of data heterogeneity and device heterogeneity in the scheduling strategy remains a key issue to be solved urgently. Summary of the Invention

[0005] The object of the present invention is to solve the above problems, and a semi-asynchronous federated learning processing method and system based on heterogeneous-aware scheduling are designed.

[0006] Furthermore, in the above-mentioned semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling, the semi-asynchronous federated learning processing method includes the following steps: Initialize the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distribute the global model to the set of edge devices; When an edge device receives the global model, perform local iteration to update the local model and calculate the local model update; The edge device calculates a comprehensive importance value based on multiple factors and uploads the comprehensive importance value to the server, notifying the server that it is ready to participate in global aggregation; When the time set by the server reaches the threshold, mark the edge devices that have completed local updates as the pre-scheduling set; The server selects a part of the devices to upload their local model updates through a scheduling strategy and records the lag of each device; When the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates; The server adjusts the aggregation time threshold for the next round according to the situation of device completion of local iteration in this round of training; Broadcast the updated global model to the edge devices and start the next round of global iteration until the global model converges.

[0007] Further, in the above semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling, the edge device includes: Suppose there is a group of edge devices under the parameter server , each device trains on its local dataset to jointly train a global model ; Denote as the training samples in including features and labels , the size of is ; The local loss function of device k on the dataset is defined as: where is the model parameter to be optimized, is the loss function corresponding to under the training samples, and the global loss function on all distributed datasets is defined as: ; where is the size of the local dataset of device , is the size of the global dataset; The federated learning optimization goal is to obtain an optimal model vector such that the global loss function is minimized, expressed as: .

[0008] Further, in the above semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling, when an edge device receives the global model, it performs local iteration to update the local model and calculates the local model update, including: When an edge device receives the global model , it performs rounds of local iteration for local update, through: ; where is the learning rate, is the gradient calculated by device based on the global model , after local training is completed, device obtains the next-round local model , device calculates obtain its local model update, denoted as .

[0009] Furthermore, in the above semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling, the edge device calculates a comprehensive importance value according to multiple factors, including: The edge device comprehensively considers the diversity of the device's dataset , the size of the dataset and the aggregation frequency factors, and according to the weighted formula: calculate the comprehensive importance value, which is the weighted result of the three evaluation metrics and , and at the same time upload the comprehensive importance value to the server in the form of a data packet and notify the server that it is ready to participate in the global aggregation.

[0010] Furthermore, in the above semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling, the server selects a part of the devices to upload their local model updates through a scheduling policy and records the lag of each device, including: The server schedules a part of from the pre-scheduled set through a heterogeneous-aware policy a subset of devices with higher values to upload their local model updates. The lag of each edge device is denoted as , and it is defined as , where is the version number of the global model based on which the device performs local training.

[0011] Furthermore, in the above semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling, the server selects a part of the devices to upload their local model updates through a scheduling policy and records the lag of each device, and also includes: Before the model aggregation in federated learning, the server obtains the values of the devices that have completed local training in this round of iteration and preferentially schedules the devices with higher values according to a preset scheduling rate. The lag of the local update uploaded by each device is denoted as , and the greater the lag, the smaller the contribution ratio to the global model.

[0012] Furthermore, in the above semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling, when the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates, including: HASA-FL uses the time threshold as the boundary of global iteration to reduce the waiting latency of fast and slow devices during the training process;​ Reduce the frequency of global model distribution and local model upload, and reduce communication overhead. When the time threshold for determining the iteration boundary is too large, HASA-FL degrades to synchronous federated learning; when the time threshold is too small, HASA-FL degrades to asynchronous federated learning. HASA-FL adjusts the time threshold for the next round according to the training completion situation of edge devices and historical time threshold information.

[0013] Furthermore, in a semi-asynchronous federated learning processing system based on heterogeneous awareness scheduling, the semi-asynchronous federated learning processing system includes the following modules: The parameter initialization module is used to initialize the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distribute the global model to the set of edge devices. The model iteration module is used to perform local iteration to update the local model and calculate the local model update when an edge device receives the global model. The importance calculation module is used for edge devices to calculate the comprehensive importance value according to multiple factors, and upload the comprehensive importance value to the server to notify the server that it is ready to participate in global aggregation. The threshold judgment module is used to mark the edge devices that have completed local updates as the pre-scheduling set when the time set by the server reaches the threshold. The lag record module is used for the server to select a part of the devices to upload their local model updates through the scheduling policy and record the lag of each device. The scheduling set update module is used to iteratively generate a new global model by aggregating the received local updates when the local updates in the scheduling set arrive. The threshold adjustment module is used for the server to adjust the aggregation time threshold for the next round according to the situation of device completion of local iteration in this round of training. The model broadcast module is used to broadcast the updated global model to the edge devices to start the next round of global iteration until the global model converges.

[0014] Furthermore, in a semi-asynchronous federated learning processing system based on heterogeneous awareness scheduling, the semi-asynchronous federated learning processing system includes the following modules: The loss function definition module is used to assume that there is a group of edge devices under the parameter server , each device trains on its local dataset to jointly train a global model , is d-dimensional real space; denote as the training samples in, including features and labels , has a size of ; The local loss function of device k on the dataset is defined as: ; where are the model parameters to be optimized, is the corresponding loss function under the training sample . The global loss function on all distributed datasets is defined as: ; The model vector obtaining module, the optimization objective for federated learning is to obtain an optimal model vector , such that is minimized, expressed as: .

[0015] Furthermore, in a semi - asynchronous federated learning processing system based on heterogeneous - awareness scheduling, the semi - asynchronous federated learning processing system includes the following modules: The iterative update module, when an edge device receives the global model , it performs rounds of local iteration for local update, through: ; where is the learning rate. After local training is completed, device calculates to obtain its local model update, denoted as .

[0016] The beneficial effects are as follows: By initializing the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distributing the global model to the set of edge devices; when an edge device receives the global model, it performs local iteration to update the local model and calculates the local model update; the edge device calculates a comprehensive importance value based on multiple factors and uploads the comprehensive importance value to the server to notify the server that it is ready to participate in global aggregation; when the time set by the server reaches the threshold, the edge devices that have completed local updates are recorded as the pre-scheduling set; the server selects a part of the devices to upload their local model updates through a scheduling strategy and records the lag of each device; when the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates; the server adjusts the aggregation time threshold for the next round according to the situation of device local iteration in this round of training; broadcasts the updated global model to the edge devices to start the next round of global iteration until the global model converges. 1. The present invention coordinates the advantages of synchronous federated learning and asynchronous federated learning, and proposes semi-asynchronous federated learning with an adaptive time threshold, using the time threshold as the boundary of federated aggregation, reducing the waiting delay between devices and accelerating model convergence; 2. Secondly, the present invention designs a heterogeneous-aware device scheduling strategy, comprehensively considering the statistical heterogeneity and system heterogeneity of edge devices during the scheduling process, preventing the model from converging to a local optimum and improving model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention.

[0018] Figure 1 Schematic diagram of the first embodiment of a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention; Figure 2 Schematic diagram of the first embodiment of a semi-asynchronous federated learning processing system based on heterogeneous-aware scheduling in an embodiment of the present invention; Figure 3 Schematic diagram of HASA-FL of a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention; Figure 4 Schematic diagram of a synchronous / asynchronous aggregation of a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention, where Figure 4 (a) is a schematic diagram of synchronous aggregation, Figure 4 (b) is a schematic diagram of asynchronous aggregation; Figure 5 Schematic diagram of a semi-asynchronous aggregation with an adaptive time threshold of a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention; Figure 6 Schematic diagram of heterogeneous-aware device scheduling for a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention, where Figure 6 (a) is the model update vector before scheduling, Figure 6 (b) is the weighted value evaluation of the model; Figure 7 Schematic diagram of the influence of different federated learning aggregation mechanisms on the test accuracy over time for a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention, where Figure 7 (a) is a schematic diagram of the accuracy performance of the convolutional model CNN on the image dataset FMNIST, Figure 7 (b) is a schematic diagram of the accuracy performance of the residual model ResNet18 on the image dataset Cifar-10; Figure 8 Schematic diagram of the influence of different scheduling rates on the model performance for a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention, where Figure 8 (a) is a curve graph of the change of model accuracy with the number of training rounds, Figure 8 (b) is a curve graph of the change of model loss with the number of training rounds; Figure 9 Schematic diagram of the influence of different scheduling strategies on the model performance for a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention, where Figure 9 (a) is a schematic diagram of the accuracy performance of the convolutional model CNN on the image dataset FMNIST, Figure 9 (b) is a schematic diagram of the accuracy performance of the residual model ResNet18 on the image dataset Cifar-10; Figure 10 Schematic diagram of the influence of different scheduling factors in the proposed scheduling strategy on the model performance for a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention, where Figure 10 (a) is a curve graph of the change of model accuracy with the number of training rounds, Figure 10 (b) is a curve graph of the change of model loss with the number of training rounds; Figure 11 Schematic diagram of the composition of an edge device, a parameter server, and a storage medium provided for a semi-asynchronous federated learning processing method based on heterogeneous-aware scheduling in an embodiment of the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0020] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.

[0021] Embodiment 1 The present invention will be specifically described below with reference to the accompanying drawings, as Figure 1 shown, a semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling, the semi-asynchronous federated learning processing method includes the following steps: Step 101, initialize the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distribute the global model to the set of edge devices; Step 102, when an edge device receives the global model, perform local iteration to update the local model, and calculate the local model update; Specifically, in this embodiment, when an edge device receives the global model after that, it performs rounds of local iteration for local update, by: ; where is the learning rate, after local training is completed, the device calculates to obtain its local model update, denoted as .

[0022] Step 103, the edge device calculates a comprehensive importance value according to multiple factors, and uploads the comprehensive importance value to the server, notifying the server that it is ready to participate in global aggregation; Specifically, in this embodiment, the edge device comprehensively considers the dataset diversity of the device , the dataset size and the aggregation frequency factors, according to the weighted formula: calculate the comprehensive importance value, and at the same time upload the comprehensive importance value to the server in the form of data packets, and notify the server that it is ready to participate in global aggregation.

[0023] Step 104: When the time set by the server reaches the threshold, mark the edge devices that have completed local updates as the pre-scheduling set; Step 105: The server selects a part of the devices to upload their local model updates through a scheduling policy and records the lag of each device; Specifically, in this embodiment, the server schedules a part of devices with higher values to upload their local model updates. The lag of each edge device is recorded as , and it is defined as , where is the version number of the global model based on which the device performs local training.

[0024] Before aggregating the models in federated learning, the server obtains the values of the devices that have completed local training in this round of iteration and preferentially schedules devices with higher values according to a preset scheduling rate. The lag of the local updates uploaded by each device is recorded as

[0025] , and the greater the lag, the smaller the contribution ratio to the global model. Specifically, in this embodiment, HASA-FL uses the time threshold as the boundary of global iteration to reduce the waiting time delay of fast and slow devices during training; Reduce the frequency of global model distribution and local model upload, reduce communication overhead. When the time threshold for determining the iteration boundary is too large, HASA-FL degrades to synchronous federated learning; when the time threshold is too small, HASA-FL degrades to asynchronous federated learning; HASA-FL adjusts the time threshold for the next round according to the training completion situation of the edge devices and historical time threshold information.

[0026] Step 107: The server adjusts the aggregation time threshold for the next round according to the local iteration completion situation of the devices in this round of training; Step 108: Broadcast the updated global model to the edge devices and start the next round of global iteration until the global model converges.

[0027] Specifically, in this embodiment, there is a group of edge devices under the parameter server of the edge device , and each device trains on its local dataset to jointly train a global model ; Denote as The training samples in include features and labels . The size of is . The local loss function of device k on the dataset is defined as: where are the model parameters to be optimized, is corresponding to the loss function under the training samples . The global loss function on all distributed datasets is defined as: ; The objective of federated learning optimization is to obtain an optimal model vector such that is minimized, expressed as: .

[0028] Its beneficial effects are as follows: By initializing the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distributing the global model to the set of edge devices; when an edge device receives the global model, it performs local iterations to update the local model and calculates the local model update; the edge device calculates the comprehensive importance value according to multiple factors and uploads the comprehensive importance value to the server, notifying the server that it is ready to participate in the global aggregation; when the time set by the server reaches the threshold, the edge devices that have completed local updates are recorded as the pre-scheduling set; the server selects a part of the devices to upload their local model updates through the scheduling strategy and records the lag of each device; when the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates; the server adjusts the aggregation time threshold for the next round according to the situation of device completion of local iterations in this round of training; broadcasts the updated global model to the edge devices, and starts the next round of global iteration until the global model converges. 1. The present invention coordinates the advantages of synchronous federated learning and asynchronous federated learning, and proposes semi-asynchronous federated learning with an adaptive time threshold, using the time threshold as the boundary of federated aggregation, reducing the waiting delay between devices and accelerating the model convergence; 2. Secondly, the present invention designs a heterogeneous-aware device scheduling strategy, comprehensively considering the statistical heterogeneity and system heterogeneity of edge devices during the scheduling process, preventing the model from converging to a local optimum and improving the model accuracy.

[0029] Specifically, the present invention can also be implemented through the following steps: Step S1: Initialize the global model parameters in the parameter server (PS) and the time threshold for the next round of aggregation, and send the global model to the set of edge devices ; Step S2: Local training. When an edge device receives the global model it performs rounds of local iteration for local update, through (6); where is the learning rate. After local training is completed, the device calculates to obtain its local model update, denoted as .

[0030] Step S3: Calculate the comprehensive importance value. The edge device comprehensively considers factors such as the diversity of the device's dataset , the size of the dataset and the aggregation frequency and other factors, and calculates the comprehensive importance value according to the weighted formula: At the same time, upload this value to the server in the form of a data packet and notify the server that it is ready to participate in global aggregation.

[0031] Step S4: When the time set by the server reaches the threshold, the edge devices that have completed local updates are recorded as the pre-scheduling set .

[0032] Step S5: The server schedules a part of the devices with higher values to upload their local model updates. The lag of each edge device is denoted as , and we define , where is the version number of the global model based on which the device performed local training; Step S6: When the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates, through (7); Step S7: The server adjusts the aggregation time threshold for the next round according to the situation of device completion of local iteration in this round of training; Step S8: Broadcast the updated global model to the edge devices and start the next round of global iteration until the global model converges.

[0033] Consider a set of edge devices under the parameter server , and each device trains on its local dataset 。Denote as the training samples (including features and labels ) in . The local loss function of device k on the dataset is defined as (8); where are the model parameters to be optimized and is the corresponding loss function under the training samples (9); The ultimate optimization goal of federated learning is to obtain an optimal model vector such that is minimized, expressed as (10).

[0034] According to the above technical solution, in step S6, it further includes: HASA-FL uses the time threshold as the boundary of global iteration, thereby reducing the waiting latency of fast and slow devices during training, while reducing the frequency of global model distribution and local model upload, and reducing communication overhead. However, the size of the time threshold will further affect the convergence speed of the model. When the time threshold determining the iteration boundary is too large, HASA-FL will degenerate into synchronous federated learning; when the time threshold is too small, HASA-FL will also degenerate into asynchronous federated learning. Therefore, a suitable time threshold is crucial for the rapid convergence of the model. HASA-FL adjusts the time threshold of the next round according to the training completion situation of edge devices in a certain round and historical time threshold information, ensuring that the time threshold change can adapt to the computing power heterogeneity difference of edge devices and guaranteeing the stability of the time threshold change.

[0035] According to the above technical solution, in step S5, it further includes: Before the model aggregation in federated learning, the server obtains the value of the devices that have completed local training in this round of iteration, and preferentially schedules the devices with higher value according to the preset scheduling rate. At the same time, due to the existence of update asynchrony, the local updates uploaded by each device have a certain lag, denoted as 。The lag mentioned here refers to the situation where the speed of local training of the device is significantly slower than the speed of global model update, resulting in the inconsistency between the global model on which local updates are based and the latest global model on the server. And aggregating updates with a large lag may bring negative optimization to the global model. Therefore, in our model update formula, the greater the lag, the smaller the contribution ratio to the global model, thus effectively alleviating the negative optimization problem caused by asynchrony.

[0036] The above introduces the embodiments of a semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling of the present invention. Please refer to Figure 2 , in a semi-asynchronous federated learning processing system based on heterogeneous awareness scheduling, the semi-asynchronous federated learning processing system based on heterogeneous awareness scheduling includes the following modules: The parameter initialization module is used to initialize the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distribute the global model to the set of edge devices; The model iteration module is used to perform local iteration to update the local model and calculate the local model update when an edge device receives the global model; The importance calculation module is used for the edge device to calculate the comprehensive importance value according to multiple factors, and upload the comprehensive importance value to the server to notify the server that it is ready to participate in global aggregation; The threshold judgment module is used to mark the edge devices that have completed local updates as the pre-scheduling set when the time set by the server reaches the threshold; The lag record module is used for the server to select a part of the devices to upload their local model updates through the scheduling policy and record the lag of each device; The scheduling set update module is used to iteratively generate a new global model by aggregating the received local updates when the local updates in the scheduling set arrive; The threshold adjustment module is used for the server to adjust the aggregation time threshold for the next round according to the situation of device completion of local iteration in this round of training; The model broadcast module is used to broadcast the updated global model to the edge devices to start the next round of global iteration until the global model converges.

[0037] Embodiment 2 Specifically, the present invention can also be implemented through the following embodiments: Figure 3 This is a schematic diagram of a semi-asynchronous federated learning HASA-FL based on heterogeneous awareness scheduling according to an embodiment of the present invention. This embodiment can be applied to the scenario of distributed heterogeneous collaborative training. The method specifically includes the following steps: Step S1: Initialize the global model parameters and broadcast them to the set of edge devices , initialize the time threshold; Step S2: After the device receives the global model from the PS, it performs rounds of local iteration for local update: (11); where is the learning rate. After local training is completed, the device calculates to obtain its local model update, denoted as .

[0038] Step S3: Calculate the comprehensive importance value. The edge device comprehensively considers factors such as the diversity of the device's dataset , the size of the dataset and the aggregation frequency and other factors, and according to the weighted formula: calculates the comprehensive importance value, and at the same time uploads this value to the server in the form of a data packet and notifies the server that it is ready to participate in the global aggregation.

[0039] By comprehensively considering factors such as dataset diversity and dataset size, the heterogeneous-aware scheduling strategy can effectively improve the convergence performance of the global model while reducing communication overhead in the mode of partial device participation. Specifically, the mathematical description of the heterogeneous-aware scheduling strategy we proposed is as follows: ; is regarded as the comprehensive value of device i in a certain round of global iteration. This value is the weighted result of three evaluation indicators and . The first indicator is used to measure the degree of dataset diversity of device . The larger the value, the more diverse the sample types in the dataset of this device. Conversely, the more scarce. We evaluate (12); where is the total number of categories in is an infinitesimal number, , is the number of samples of the category in the dataset of device accounting for the proportion of . This dataset diversity evaluation indicator can be regarded as our improvement of applying Shannon entropy to machine learning. We call it diversity entropy here. The higher the diversity entropy value, the more diverse the samples in the dataset. When the probabilities of all sample categories are equal, the diversity entropy reaches the maximum value, indicating the highest degree of dataset diversity.​

[0040] The second evaluation indicator For measuring equipment The importance of dataset size, The larger the value, the more data the device relies on for training, and the greater the impact on the global model. The specific evaluation formula is as follows: (13); The third indicator is used to evaluate the equipment in front The relative frequency of participating in aggregation in the round global iteration is denoted as : (14); in is the number of times the device participates in aggregation or is scheduled to upload updates, and R is the current global iteration round. and Inversely proportional, that is, the more times a device participates in aggregation, the smaller the relative frequency. This indicator can balance the fairness issue to other devices caused by some devices being frequently scheduled due to their advantages in the data set.

[0041] Step S3: When the time set by the server reaches the threshold, the edge devices that have completed the local update are recorded as the pre-scheduled set .

[0042] Step S4: The server schedules a part of Higher value equipment Upload their local model updates. Figure 4 As shown in (a), during a global iteration, four devices receive the global model. Perform local training, and the local updates are respectively vector , , and The direction of the vector indicates the direction of the local update, and the length of the vector indicates the size of the local update. Assuming that only two devices are selected for aggregation in each round of training, the number of data samples of device 1 is large, and the data types are not diverse enough. The local update corresponding to device 1 in Figure 4(b) is , although it has an advantage in size, it deviates from the optimal point in direction, so the comprehensive evaluation result is poor. The data types are diverse and the data samples are rich, so Figure 4 The corresponding local model update in (b) , it is easier to approach the central optimal point, so the result of comprehensive evaluation is the best. According to the comprehensive evaluation results of devices 1 to 4, the server will select devices 2 and 3 with better evaluation results, so that the aggregated global model is closer to the global optimal point compared with the above figure. Each edge device The lag of is denoted as , and we define , where is the version number of the global model based on which local training is performed for device ; Step S6: When the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates, through (15); Step S7: The server adjusts the aggregation time threshold for the next round according to the situation of device local iteration completion in this round of training.

[0043] The adaptive time threshold aggregation mechanism proposed by the present invention uses the time threshold as the trigger condition for the server aggregation operation. The technical effect comparison is as Figure 5 shown: Compared with the synchronous aggregation scheme shown in Figure 6 (a), this scheme effectively alleviates the blocking waiting between devices and improves the training efficiency; compared with the asynchronous update scheme in Figure 6 (b), the communication load is significantly improved by optimizing the global model update frequency.

[0044] Regarding the time threshold adjustment problem of semi-asynchronous federated learning in a system heterogeneous environment, the present invention finds that: the number of devices that complete training within a specific aggregation time threshold is positively correlated with the length of this time threshold. When the number of devices that complete training is lower than expected, the main reason is that the aggregation time threshold is set insufficiently (rather than device performance limitations). Based on this technical finding, the present invention proposes an adaptive adjustment method for the aggregation time threshold, which is specifically implemented as follows: (16); where and are respectively the number of devices that fail to complete local training and successfully complete local training in the th round of iteration, is a system parameter between . Generally, when is greater than , is greater than 1, indicating that the number of devices that fail to complete training in the th global round of iteration is relatively large, and the aggregation period for the next round should be appropriately extended, and vice versa.

[0045] Step S8: Broadcast the updated global model to the edge devices and start the next round of global iteration until the global model converges.

[0046] Consider a set of edge devices under the parameter server , each device trains on its local dataset to co-train a global model . Denote as the training samples in (including features and labels ), the size of . The local loss function of device k on dataset (17); where are the model parameters to be optimized, is the corresponding loss function under the training samples . The global loss function on all distributed datasets is defined as (18); The ultimate optimization goal of federated learning is to obtain an optimal model vector such that is minimized, expressed as (19).

[0047] Embodiment 2: Embodiment 2 of the present invention provides an edge device and a storage medium. Figure 7 FIG. Figure 7 shows a schematic composition diagram of an edge device and a storage medium and a cloud device and a storage medium provided in Embodiment 2 of the present invention. The edge device includes: one or more processors; a memory, on which one or more programs are stored. When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above method of the present invention. \[{\mathbf{w}}_i^{t + 1} - {{\mathbf{w}}^t}\] In some embodiments of the present invention, the multiple processors include: A parameter server: performs an aggregation operation on the model when the time threshold condition is met.

[0048] In the embodiments of the present invention, the storage medium includes: A computer-readable storage medium for storing a computer program, which when executed by a processor implements the steps of the above method.

[0049] Example 3: The third embodiment of the present invention analyzes the experimental operation results formed by the above-mentioned first and second embodiments: In this embodiment, the performance of the experimental platform used in this experiment: the processor is an Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz, with 28 cores and 56 threads, a base frequency of 2.00 GHz, 32 * 4 GB of memory, and an NVIDIA RTX 4090 graphics card with a video memory of 24GB. The software environment is Ubuntu 20.04, Python 3.10, CUDA 11.8, and Pytroch 2.0.0. We use multi-threading to simulate multiple edge devices, where each device trains a corresponding deep learning model; Verify the effectiveness of the present invention from four aspects: the comparison of the training time of HASA-FL with different methods, the convergence of different scheduling rates, the comparison of the number of training rounds of different methods, and ablation experiments. Among them, the models used in the experiments include CNN and ResNet-18, and are trained based on multiple widely used public datasets, including MNIST, Fashion-MNIST (FMNIST), and CIFAR-10. We compared HASA-FL with eight methods: synchronous FL (FedAvg), asynchronous FL (FedAsync), semi-asynchronous FL (FedBuff and FedPS), random scheduling (Rdm), Powerofchice, Importance-sampling, and Clustered-sampling. For the aggregation mechanism, we used four aggregation mechanisms and the adaptive time threshold aggregation mechanism (FedDT) proposed by us for comparison. Among them, Fedbuff is a semi-asynchronous FL mechanism based on the number of updates, and updates the global model according to the threshold of the number of received updates set in advance by the server. FedPS is a semi-asynchronous FL mechanism based on a period, and updates the global model whenever the time period specified by the server expires. For the scheduling strategy, we used four scheduling strategies and the heterogeneous awareness strategy (Hetero-aware) proposed by us for comparison. Among them, Rdm is a random scheduling strategy, denoted as, randomly selecting a part of the devices uniformly. The Powerofchice strategy selects a part of the devices with the lowest loss. The Importance-sampling is an importance sampling strategy, which selects a part of the devices with the largest local update norm value. Clustered-sampling is grouped sampling, which groups the training according to model similarity, and selects a part of the devices in each group for aggregation; In this embodiment, a comparative experiment of training the CNN model for 20,000 seconds on the FMNIST dataset and training the ResNet-18 model for 50,000 seconds on the Cifar-10 dataset shows that the FedDT method of the present invention is significantly superior to the FedAvg, FedAsync, FedBuff, and FedPS aggregation algorithms in the prior art in terms of training performance. The experimental data analysis shows that FedDT exhibits the optimal training performance advantage, and this advantage becomes more significant as the dataset complexity increases. The test results of using ResNet18 on the Cifar-10 dataset show that after 50,000 seconds of training, FedDT reaches an accuracy of 86.12%, which is 2.07%, 2.74%, 4.50%, and 16.15% higher than FedPS (84.05%), FedBuff (83.38%), FedAsync (81.62%), and FedAvg (69.97%) respectively. In terms of the time efficiency to reach 80% accuracy, FedDT only requires 27,028 seconds, which shortens the training time by 23.63%, 16.15%, and 38.98% compared to FedPS (35,391 seconds), FedBuff (32,233 seconds), and FedAsync (44,294 seconds) respectively. Due to the waiting delay caused by full synchronization, FedAvg fails to achieve this accuracy index within the specified time limit. These data fully confirm the significant technological progress of the FedDT method of the present invention in terms of convergence speed and model accuracy.

[0050] In this embodiment, HASA-FL adopts the Local Variation Standard Deviation (LVSD), which is defined as the standard deviation of the difference between the accuracy (or loss) of each round and that of the previous round, to measure the fluctuation of the curve in the last 50 rounds. The embodiment of the present invention further studies the impact of the scheduling rate on the performance of the federated learning model. Experiments of training the Resnet-18 model for 200 rounds using the Cifar-10 dataset show that: when the scheduling rate is 20%, the model accuracy / loss curve shows significant oscillation, and the final test accuracy fails to break through 80%, and its accuracy LVSD value reaches 0.057; as the scheduling rate is gradually increased to 100%, the convergence stability of the model is significantly improved, the final test accuracy is increased to 86.12%, and at the same time the accuracy LVSD value is optimized to 0.0055, and the smoothness of the curve is increased by nearly 10 times. However, the experimental data also reveals the positive correlation between the scheduling rate and the communication overhead - in the federated learning scenario with limited communication resources, a 100% scheduling rate will lead to a sharp increase in the uplink communication overhead, especially in a large-scale distributed training environment, which is not practically feasible. To address this technical contradiction, the present invention creatively proposes a new scheduling strategy, which can ensure the model training quality by optimizing the scheduling mechanism while strictly restricting the consumption of communication resources, effectively solving the technical problem that it is difficult to balance communication efficiency and model performance in the prior art.

[0051] In this embodiment, the embodiment of the present invention verifies the superior performance of the proposed Hetero-aware scheduling algorithm compared with benchmark algorithms such as Random, PowerofChoice, ImportanceSampling, and ClusteredSampling through comparative experiments of training the CNN model for 120 rounds on the FMNIST dataset and training the ResNet-18 model for 200 rounds on the Cifar-10 dataset. The experiments adopt different scheduling rate settings (FMNIST 30%, Cifar-10 40%), and focus on the performance of each algorithm under the same scheduling rate condition. The experimental data shows that the Hetero-aware scheduling algorithm is significantly superior to the comparative algorithms in terms of convergence speed, final test accuracy, and test loss indicators, and this advantage is particularly prominent in the complex scenario of training Cifar-10 with ResNet-18. Specifically, under the condition of a 40% scheduling rate, as Figure 10As shown in (b), Hetero-aware is the only algorithm that maintains smooth convergence, and its accuracy curve LVSD value is only 0.0067, which is 4-7 times smoother than the curves of FedAvg (0.030), Power of Choice (0.037), Importance Sampling (0.045) and Clustered Sampling (0.038), which fully proves the strong robustness of the algorithm in a data heterogeneous environment. Analysis shows that the traditional algorithm cannot fully characterize the overall data distribution under a limited scheduling rate, resulting in performance fluctuations. The present invention effectively solves this technical problem through an innovative heterogeneous perception mechanism, and provides an efficient scheduling solution for resource-constrained federated learning systems.

[0052] The embodiment of the present invention verifies the influence of each key factor in the proposed hetero-aware scheduling strategy on the model performance through a systematic ablation experiment. The experiment trains the ResNet-18 model with a 40% scheduling rate and 200 rounds of global iterations on the Cifar-10 dataset, and compares and analyzes the performance differences between the complete hetero-aware method and the partial factor combination (sk+fk, dk+fk) and the benchmark FedAvg. The experimental data show that the complete hetero-aware method exhibits the best overall performance, and its test accuracy is significantly improved compared with the method that only considers the device dataset size and aggregation frequency (sk+fk), and is moderately improved compared with the method that only considers the dataset diversity and aggregation frequency (dk+fk). It is worth noting that the performance of the dk+fk combination is better than the sk+fk combination, and both significantly surpass the FedAvg benchmark method, which confirms that the dataset diversity factor has a more important impact on model performance than the dataset size factor. Technical analysis shows that if the static factor (sk / dk) is used alone as a scheduling indicator, it will lead to the solidification of equipment selection, which violates the basic principles of federated learning. However, the present invention organically combines the dynamic aggregation frequency (fk) with the static factor, which not only ensures the effective use of equipment data, but also achieves a significant improvement in model performance, verifying the technical necessity of multi-factor collaborative scheduling. The experimental results fully demonstrate the complementary role of each scheduling factor in the Hetero-aware method and the scientific nature of the overall design.

[0053] The embodiments of the present invention verify the effectiveness of the proposed method through four groups of systematic experiments: The comparative experiment of training the CNN model for 20,000s on the FMNIST dataset and training the ResNet-18 model for 50,000s on the Cifar-10 dataset shows that FedDT has an accuracy of 86.12% in the IID mode, which is 2.07%, 2.74%, 4.50% and 16.15% higher than FedPS, FedBuff, FedAsync and FedAvg respectively, and the time required to reach 80% accuracy is reduced by 23.63%, 16.15% and 38.98% respectively; The scheduling rate impact experiment shows that when the scheduling rate is increased from 20% to 100%, the LVSD value of the model accuracy is optimized from 0.057 to 0.0055, and the curve smoothness is increased by 10 times; In the comparative experiment of using 30% and 40% scheduling rates on the FMNIST and Cifar-10 datasets respectively, the LVSD value of Hetero-aware reaches 0.0067, which is 4-7 times higher than the curve smoothness of benchmark methods such as FedAvg; The ablation experiment confirms that the complete Hetero-aware method has the best performance, among which the dk+fk combination considering dataset diversity is better than the sk+fk combination considering dataset size, and both are significantly better than FedAvg, verifying the necessity of multi-factor collaborative scheduling. All experimental data consistently show that the present invention is significantly superior to the existing technical solutions in terms of model accuracy, convergence speed and training stability.

[0054] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling, characterized in that The semi-asynchronous federated learning processing method includes the following steps: Initialize the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distribute the global model to the set of edge devices; When an edge device receives the global model, perform local iteration to update the local model and calculate the local model update; The edge device calculates a comprehensive importance value based on multiple factors and uploads the comprehensive importance value to the server, notifying the server that it is ready to participate in global aggregation; When the time set by the server reaches the threshold, mark the edge devices that have completed local updates as the pre-scheduling set; The server selects a part of the devices to upload their local model updates through a scheduling policy and records the lag of each device; When the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates; The server adjusts the aggregation time threshold for the next round according to the situation of the devices completing local iteration in this round of training; Broadcast the updated global model to the edge devices and start the next round of global iteration until the global model converges.

2. The semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling according to claim 1, wherein The edge device includes: Assume there is a set of edge devices under the parameter server , and each device trains on its local dataset to jointly train a global model ; Denote as the training samples in including features and labels , and the size of is ; The local loss function of device k on dataset​ ； Among them are the model parameters to be optimized, is the loss function corresponding to the training samples. The global loss function on all distributed datasets is defined as: ； wherein is the device the size of the local data set, is the size of the global data set; The optimization objective of federated learning is to obtain an optimal model vector , such that is minimized, expressed as: 。 3. A semi-asynchronous federated learning processing method based on heterogeneous perception scheduling according to claim 1, characterized in that When an edge device receives the global model, perform local iteration to update the local model and calculate the local model update, including: When an edge device receives the global model it performs rounds of local iteration for local update by: ； where is the learning rate, and is the gradient calculated by the device based on the global model. After local training is completed, the device obtains the local model for the next round . The device calculates to obtain its local model update, denoted as .

4. A semi-asynchronous federated learning processing method based on heterogeneous perception scheduling according to claim 1, characterized in that, The edge device calculates a comprehensive importance value based on multiple factors, including: The edge device comprehensively considers the diversity of the device's dataset , the size of the dataset and the aggregation frequency factors, and according to the weighted formula: calculate the comprehensive importance value, which is the weighted result of the three evaluation indicators and , at the same time, upload the comprehensive importance value to the server in the form of data packets, and notify the server that it is ready to participate in the global aggregation.

5. A semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling according to claim 1, characterized in that, The server selects a part of the devices to upload their local model updates through a scheduling policy and records the lag of each device, including: The server schedules a part through a heterogeneous awareness strategy of the devices with higher values to upload their local model updates. The latency of each edge device is denoted as , and is defined as , where is the version number of the global model based on when the device is locally trained. ​ 6. The semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling according to claim 1, wherein The server selects a part of the devices to upload their local model updates through a scheduling policy and records the lag of each device, further including: Before the model aggregation in federated learning, the server obtains the values of the devices that have completed local training in this round of iteration, and preferentially schedules the devices with higher values. The lag of the local updates uploaded by each device is recorded as 7. A semi-asynchronous federated learning processing method based on heterogeneous awareness scheduling according to claim 1, wherein When the local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates, including: HASA-FL uses the time threshold as the boundary of global iteration to reduce the waiting delay of fast and slow devices during the training process; Reduce the frequency of global model distribution and local model upload, reduce communication overhead. When the time threshold for determining the iteration boundary is too large, HASA-FL degrades to synchronous federated learning; when the time threshold is too small, HASA-FL degrades to asynchronous federated learning; HASA-FL adjusts the time threshold for the next round according to the situation of the edge devices completing training and historical time threshold information.

8. A semi-asynchronous federated learning processing system based on heterogeneous awareness scheduling, characterized in that, The semi-asynchronous federated learning processing system includes the following modules: A parameter initialization module, which is used to initialize the global model parameters in the parameter server and the time threshold for the next round of aggregation, and distribute the global model to the set of edge devices; A model iteration module, which is used to perform local iteration to update the local model and calculate the local model update when an edge device receives the global model; An importance calculation module, which is used for the edge device to calculate a comprehensive importance value based on multiple factors and upload the comprehensive importance value to the server, notifying the server that it is ready to participate in global aggregation; A threshold judgment module, which is used to mark the edge devices that have completed local updates as the pre-scheduling set when the time set by the server reaches the threshold; A lag recording module, which is used for the server to select a part of the devices to upload their local model updates through a scheduling policy and record the lag of each device; A scheduling set update module, which is used to, after local updates in the scheduling set arrive, the server iteratively generates a new global model by aggregating the received local updates; A threshold adjustment module, which is used for the server to adjust the aggregation time threshold for the next round according to the situation of device local iteration in this round of training; A model broadcast module, which is used to broadcast the updated global model to edge devices to start the next round of global iteration until the global model converges.

9. A semi-asynchronous federated learning processing system based on heterogeneous awareness scheduling as claimed in claim 8, wherein The semi-asynchronous federated learning processing system includes the following modules: A loss function definition module for setting that there is a set of edge devices under the parameter server , and each device trains on its local dataset to jointly train a global model ; Denote as the training samples in including features and labels , of size ; the local loss function of device k on the dataset is defined as: ； Among them are the model parameters to be optimized, is the loss function corresponding to the training samples. The global loss function on all distributed datasets is defined as: ； A model vector obtaining module, where the optimization objective for federated learning is to obtain an optimal model vector , such that is minimized, expressed as: 。 10. A semi-asynchronous federated learning processing system based on heterogeneous perception scheduling as claimed in claim 8, wherein The semi-asynchronous federated learning processing system includes the following modules: Iterative update module, used when an edge device receives the global model After that, it performs rounds of local iteration for local update by: ； where is the learning rate. After local training is completed, the device calculates to obtain its updated local model, denoted as .

Images