Methods, apparatus, and storage media for heterogeneous federated learning training

Abstract

This application relates to the field of federated learning technology, and provides a heterogeneous federated learning training method, apparatus, and storage medium. The method includes: acquiring local resource status information of multiple participating devices, the local resource status information including at least the computational latency, communication latency, and data feature vector of each participating device; clustering the multiple participating devices based on the similarity of the data feature vectors, dividing the multiple participating devices into multiple device clusters; determining the contribution of each participating device in each device cluster and the average waiting time of all participating devices in each device cluster; selecting target devices in the device clusters based on the contribution of each participating device in the device cluster and the average waiting time; assigning an update frequency to each target device, and distributing the global parameters of the network to be trained, so that each target device can start local training of the network to be trained based on the global parameters and the update frequency. This method can improve the accuracy of the global model and achieve efficient convergence in a heterogeneous environment.

Description

Technical Field

[0001] This application relates to the field of federated learning technology, specifically to a method, apparatus, and storage medium for heterogeneous federated learning training. Background Technology

[0002] Federated learning, as an efficient and privacy-preserving distributed learning framework, has demonstrated significant application potential in various fields such as communications, the Internet of Things (IoT), and edge computing. However, due to device heterogeneity and the presence of non-independent and identically distributed data within the system, the performance of the global model in federated learning degrades, and training time costs increase. In heterogeneous systems, the computing and communication capabilities of user devices vary significantly. Devices with limited resources require longer training times, while resource-rich devices often remain idle after training, resulting in low utilization of computing resources. This imbalance in resource allocation triggers a severe long-tail effect, prolonging overall training time, reducing system efficiency, and impacting the convergence speed and performance of the global model.

[0003] Currently, most federated learning research addresses system heterogeneity and data heterogeneity separately. Adjusting the update frequency to adapt to different device capabilities improves training efficiency and global model convergence, but reducing synchronization may weaken the model's generalization ability. On the other hand, methods that combine global and local model parameters enhance the model's generalization ability on non-independent and identically distributed data while maintaining data privacy, but frequent synchronization increases communication overhead and exacerbates system heterogeneity. Summary of the Invention

[0004] The purpose of this application is to provide a heterogeneous federated learning training method, apparatus, and storage medium.

[0005] To achieve the above objectives, the first aspect of this application provides a heterogeneous federated learning training method, the method comprising:

[0006] Acquire local resource status information of multiple participating devices. The local resource status information includes at least the computational latency, communication latency, and data feature vector of each participating device. The data feature vector is obtained by feature extraction from the local dataset. The local dataset is used for local model training of multiple participating devices.

[0007] Clustering of multiple participating devices based on the similarity of data feature vectors, dividing the multiple participating devices into multiple device clusters;

[0008] The contribution of each participating device in each device cluster and the average waiting time of all participating devices in each device cluster are determined. The contribution is determined based on the local resource status information of the participating devices contained in the device cluster to which the participating device belongs.

[0009] For each device cluster, the target device in the cluster is selected based on the contribution and average waiting time of each participating device in the cluster.

[0010] An update frequency is assigned to each target device, and the global parameters of the network to be trained are distributed to each target device so that each target device can start local training of the network to be trained based on the global parameters and the update frequency.

[0011] In this embodiment, selecting the target device in a device cluster based on the contribution and average waiting time of each participating device in the cluster includes: determining the best device in each device cluster based on the contribution of each participating device; constructing a first device subset based on the best device; solving the objective function based on the average contribution and average waiting time of all best devices in the first device subset to obtain a first function value for the first device subset, where the first function value represents the utility score of the device subset and its update frequency, and the update frequency refers to the participation intensity of the target device in a training round; randomly selecting N participating devices from multiple device clusters and adding them to the first device subset to obtain a second device subset; and solving the objective function based on the average contribution and average waiting time of all devices in the second device subset to obtain the first device subset. The second function value of the second device subset; if the second function value is less than or equal to the first function value, the step of randomly selecting N participating devices from multiple device clusters and adding them to the first device subset is executed again to obtain the second device subset, until all participating devices are added to the first device subset to solve the objective function and calculate the corresponding objective function value; if the second function value is greater than the first function value, the second device subset is used as the new first device subset, and the step of randomly selecting N participating devices from multiple device clusters and adding them to the first device subset is executed again to obtain the second device subset, until all participating devices are added to the first device subset to solve the objective function and calculate the corresponding objective function value; the devices included in the final first device subset are determined as the target devices.

[0012] In this embodiment of the application, allocating an update frequency for each target device includes: determining the objective function value corresponding to the subset of target devices to which all target devices are located; determining the initial update frequency of each target device; sequentially adjusting the initial update frequency of at least one target device in the subset of target devices, and determining the objective function value of the subset of target devices after each adjustment, until the maximum value of the objective function value is determined; and determining the update frequency corresponding to each target device at the maximum value as the target update frequency for each target device.

[0013] In the embodiments of this application, the objective function The expression is as follows:

[0014]

[0015] in, The weighting coefficient for the average contribution. The average contribution of the device subset. This represents the average waiting time for a subset of devices.

[0016] In this embodiment, the average contribution is determined according to the following formula. :

[0017]

[0018] in, Belongs to cluster equipment Contribution This is an indicator function.

[0019] In the embodiments of this application, the average waiting time is determined according to the following formulas. Weighting coefficients for average contribution :

[0020]

[0021] in, For the maximum return time, Calculate the latency for the device. For device communication delay, For indicator functions, As the initial value, This refers to the attenuation rate.

[0022] In this embodiment, the maximum return time is determined according to the following formula. :

[0023]

[0024] in, For equipment Update frequency Calculate the latency for the device. This refers to device communication delay.

[0025] In this embodiment of the application, the method further includes: receiving local training results returned by the target device; weighting and aggregating the local training results based on the size of the local dataset of the target device, and updating the global parameters of the network to be trained.

[0026] A second aspect of this application provides a heterogeneous federated learning training apparatus, comprising: a memory configured to store instructions; a processor configured to retrieve instructions from the memory; and a heterogeneous federated learning training method capable of implementing any of the above-mentioned methods when executing the instructions.

[0027] A third aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, cause the processor to be configured to perform any of the above-described heterogeneous federated learning training methods.

[0028] This application proposes a heterogeneous federated learning training method, which clusters devices by acquiring data feature vectors from a local dataset, and constructs an objective function based on device contribution and system waiting time. It adaptively schedules different devices and their update frequencies, thereby accelerating model convergence in the early stages of training and improving training efficiency and generalization ability in the later stages. Under the premise of protecting data privacy, it significantly improves model accuracy and greatly reduces training latency. Attached Figure Description

[0029] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0030] Figure 1 The schematic diagram illustrates a flowchart of a heterogeneous federated learning training method according to an embodiment of this application;

[0031] Figure 2 An overall flowchart of a heterogeneous federated learning method according to an embodiment of this application is illustrated schematically;

[0032] Figure 3 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0034] Figure 1 The illustration schematically shows a flowchart of a heterogeneous federated learning training method according to an embodiment of this application. Figure 1 As shown, in one embodiment of this application, a heterogeneous federated learning training method is provided, including the following steps:

[0035] Step 101: Obtain local resource status information of multiple participating devices. The local resource status information includes at least the computation latency, communication latency, and data feature vector of each participating device. The data feature vector is obtained by feature extraction from the local dataset. The local dataset is used for local model training of multiple participating devices.

[0036] Step 102: Cluster the multiple participating devices based on the similarity of data feature vectors, and divide the multiple participating devices into multiple device clusters;

[0037] Step 103: Determine the contribution of each participating device in each device cluster and the average waiting time of all participating devices in each device cluster. The contribution is determined based on the local resource status information of the participating devices contained in the device cluster to which the participating device belongs.

[0038] Step 104: For each device cluster, select the target device in the device cluster based on the contribution and average waiting time of each participating device in the cluster;

[0039] Step 105: Assign an update frequency to each target device and send the global parameters of the network to be trained to each target device so that each target device can start local training of the network to be trained based on the global parameters and update frequency.

[0040] Before the training began, each participating device It can autonomously detect its local resource status, including computing power, communication bandwidth, and data characteristics. Computing power is expressed as the number of floating-point operations that can be performed per second, denoted as... The effective uplink bandwidth of the communication is denoted as... To eliminate transient jitter, both bandwidth and computing power are calculated using the median of multiple measurements. Let M be the number of floating-point operations required to complete one local iteration, then the computational latency is: Let the total number of model parameters be... The communication delay is: Meanwhile, each device has a local dataset. Extracting feature vectors. First, the local data is converted into a numerical matrix. Then, truncated singular value decomposition (SVD) is applied to this matrix to obtain a set of feature vectors representing the main directions of data variation. The first feature vector is selected... The most important feature vectors, which make up the data feature vector matrix of this device: Ultimately, each participating device receives three types of information: device computation latency. Device communication delay Data feature vectors And upload it to the server.

[0041] After completing local measurement and feature extraction, each participating device then utilizes its own computing power. ,bandwidth Data scale The data is uploaded to the server. The server then obtains information on the resource and data distribution of all devices.

[0042] The server calculates the cosine similarity between every two participating devices based on the data feature vectors uploaded by the devices. :

[0043]

[0044] in, Let i be the data feature vector of device i. Let be the data feature vector of device j.

[0045] The server constructs an N×N similarity matrix from the cosine similarity results (N is the total number of devices). Then, a hierarchical clustering algorithm is used to group the participating devices. Initially, each device is treated as an independent cluster. The two clusters with the highest cosine similarity among all current clusters are found and merged into a new cluster. This step is repeated until all devices are merged into one large cluster.

[0046] Finally, the optimal number of clusters is selected by calculating the silhouette coefficient. (Equipment) contour coefficient for:

[0047]

[0048] Where a(i) represents the device The average distance to other devices in the same cluster, b(i) represents the device's average distance to other devices in the cluster. The average distance to devices within the nearest cluster. The server will try different cluster number K values ​​(e.g., K=2, 3, 4...), and for each grouping scheme, calculate the average silhouette coefficient for all devices. A high silhouette coefficient indicates high data similarity within a group and good separation between groups. A low silhouette coefficient indicates poor grouping performance. The K value that maximizes the average silhouette coefficient is selected as the final optimal number of clusters.

[0049] After clustering is complete, the server calculates the contribution of each participating device. For devices belonging to a cluster... equipment Its contribution for:

[0050]

[0051] in, For the size of the local dataset, This indicates the weights based on the type of participating device. Different devices, such as desktop computers, mobile phones, and laptops, have different weights. , and This reflects their stability, reliability, and impact on the training process. For example, =1.0 indicates a desktop computer, which is stable and reliable. =0.5 indicates a laptop, which is relatively reliable. =0.2, representing the phone's performance, which may be affected by battery drain or network instability. Contribution level. The calculations are performed within the same device cluster, making the contributions comparable across different clusters. First, the contribution value of each device within the cluster is calculated. Then, the sum of the contributions of all devices within the cluster is calculated, thus yielding the contribution of each individual device. This normalization process allows for the determination of a participating device's contribution within its cluster. It is a value between 0 and 1, and it applies to all devices within the same cluster. The sum is 1.

[0052] Assume there exists a scheduling scheme , Indicates equipment The update frequency.

[0053] Average contribution Used to measure the scheme Below is the overall data value of all selected devices:

[0054]

[0055] in, Belongs to cluster equipment The degree of contribution. For indicator functions, when the device Selected hour, ,So That is, the total number of selected devices. It refers to the contribution of all selected devices. The sum of these. Therefore, the average contribution function represents the average contribution of the selected devices.

[0056] Average waiting time Used to measure the scheme The following results in the overall time efficiency loss of the system:

[0057]

[0058] in, Maximum return time refers to the total time required for the slowest selected device to complete local training and communication under a given scheduling scheme. It determines the total duration of this training round.

[0059]

[0060] in, Indicates equipment Total time to complete local training and communication. Indicates equipment The waiting time, that is, the waiting time that the fastest-completing device must wait for the slowest-completing device. Therefore, It is the average waiting time of all selected devices.

[0061] Based on average contribution and average waiting time Construct the objective function, objective function The goal is to create a unified scoring standard to comprehensively evaluate scheduling schemes. Advantages and disadvantages:

[0062]

[0063] in, Weighting coefficients for average contribution:

[0064]

[0065] In the early stages of training, The value is relatively large, and the objective function focuses more on maximizing the average contribution. To achieve rapid convergence. In the later stages of training, The decay objective function focuses more on minimizing the average waiting time. To improve efficiency, the frequency range constraint must first be satisfied when solving the objective function.

[0066] This constraint ensures that the update frequency is within a reasonable range. The second constraint is the cluster cover constraint. This constraint ensures that at least one device in each cluster is selected, and is a key constraint for guaranteeing data diversity and preventing model bias. Ultimately, a solution was found. As a result of the solution, The value is the largest, this It can directly specify which devices are selected and how often they participate in this round of training.

[0067] The server obtains the solution results. In the process, obtain the update frequency of each target device. and the latest global parameters This is sent to each selected target device, ensuring that all devices begin this round of training and learning based on the same model.

[0068] After receiving instructions from the server, the device will base its actions on global parameters. and update frequency Perform local training. The device uses its own local dataset. The model is iteratively optimized multiple times. The training algorithm uses standard optimization methods, such as stochastic gradient descent (SGD). The gradient descent formula is:

[0069]

[0070] in, This represents the model parameters of the device during the e-th local iteration. The learning rate controls the step size for each parameter update. To obtain from local dataset A batch of data extracted from the data. This is the loss function. This formula means that the device calculates the loss gradient using a batch of data, and then fine-tunes the parameters along the inverse direction of the gradient to reduce the model's prediction error on that batch of data. After completing the local update, the device uploads the parameters. The server uses a federated average algorithm to perform weighted aggregation, and the aggregation formula is as follows:

[0071]

[0072] in, For the first The equipment collection for the round of training. For equipment The size of the local dataset is considered, and the aggregation weight of each device is proportional to its data volume. Devices with more data have a greater impact on the next-generation global model. Through multiple iterations, the server can continuously update the global model, achieving efficient and stable convergence. If a device does not return a result within the time limit, its update will be discarded; devices that fail for multiple consecutive rounds will be temporarily excluded from the candidate set to ensure system robustness. The final output is the optimized model parameters. This method obtains the computational and communication latency of devices through local resource detection, and combines this with feature extraction from the local dataset, uploading it to the server for clustering to group heterogeneous devices. An objective function is constructed that comprehensively considers device contribution and average waiting time, with the function decreasing with each training epoch to ensure rapid convergence in the early stages and improve robustness and generalization in later stages. Update frequencies are dynamically assigned to selected devices, and a weighted average method is used for global parameter aggregation after local training. The weights are determined by the size of the local dataset, thus achieving efficient convergence and fair participation in heterogeneous environments.

[0073] In one embodiment, based on contribution In each device cluster, the device with the highest contribution is selected and called the optimal device. An initial subset of devices, called the first device subset, is formed from the optimal devices of all clusters. Then, this first device subset and its update frequency (which can be initialized to 1) are substituted into the objective function. The calculation is performed, and the result is called the first function value. From all device clusters (including selected and unselected clusters), N (e.g., 1 or 2) participating devices not yet in the first device subset are randomly selected and added to the first device subset to form a new, larger device subset, called the second device subset. The purpose is to verify whether adding devices to the existing set can improve overall utility. Case A: Second function value > First function value, then accept this better second device subset. Update the second device subset to the new first device subset. Case B: Second function value ≤ First function value, then reject this second device subset. The current first device subset remains unchanged. The above process of generating new candidate solutions, evaluating, and deciding is repeated until all participating devices have been tried to be added to the first device subset. This means that the algorithm has fully explored all possible directions for improvement. When the loop ends, the devices in the current first device subset are determined as the target devices finally selected in this round of training. This scheme starts from an initial solution that satisfies the constraints and has high quality, which is much faster than a completely random search. By exploring randomly to avoid getting stuck in local optima, it is possible to discover non-intuitive but better-performing combinations of devices.

[0074] In one embodiment, the target devices in the first subset have been determined through the aforementioned steps. An initial update frequency (which can be initialized to 1) is first assigned to all devices in this subset. This initial frequency allocation scheme is then... Substitute into the objective function The algorithm calculates a baseline objective function value. It then sequentially adjusts the initial update frequency of at least one target device in the target device subset. After each adjustment, it immediately recalculates the new objective function value for the entire device subset. It then determines whether the adjustment improved or decreased the overall utility value. Throughout the iteration process, the algorithm continuously tracks and records the maximum objective function value reached so far, along with the specific frequency allocation scheme corresponding to that maximum value. Finally, the algorithm outputs the frequency allocation scheme corresponding to the maximum value. At this point, the frequency assigned to each target device is its final target update frequency. This step is performed after the target devices are selected. It ensures that the system is optimal not only in device selection but also in the allocation of computational tasks. In this way, the system can maximize the capacity of the selected devices, finding the most precise balance between data contribution and system waiting time, thereby maximizing global training efficiency.

[0075] In one embodiment, the objective function The expression is as follows:

[0076]

[0077] in, The weighting coefficient for the average contribution. The average contribution of the device subset. Let be the average latency of a subset of devices. Objective function As a unified scoring standard, it is used to evaluate any equipment scheduling scheme. The overall utility of this function is maximized.

[0078] In one embodiment, the average contribution is determined according to the following formula. :

[0079]

[0080] in, Belongs to cluster equipment Contribution For indicator functions. Average contribution. This represents the overall data value of the selected subset of devices. In the objective function, a higher average contribution is better, because a higher contribution means that a group of devices with high data value has been selected, which can effectively improve the accuracy and convergence speed of the global model.

[0081] In one embodiment, the average waiting time is determined according to the following formulas. Weighting coefficients for average contribution :

[0082]

[0083]

[0084] in, For the maximum return time, Calculate the latency for the device. For device communication delay, For indicator functions, As the initial value, This refers to the decay rate. Average waiting time. This represents the challenges in adopting scheduling schemes due to equipment heterogeneity. Finally, the overall time efficiency cost of the system is calculated. It measures the average idle waiting time required by all selected devices under the synchronous aggregation mechanism. Its calculation depends on the completion time of the slowest device. In the objective function, the smaller the average waiting time, the better, because a smaller average waiting time means that the system waits for slow devices for a short time, training rounds are completed quickly, and overall efficiency is high. One with training rounds A coefficient that increases and then gradually decreases. In the early stages of training, Small, The objective function is larger and focuses more on maximizing the average contribution. Prioritize using high-contribution devices for faster convergence. In the later stages of training, big, The objective function is smaller and focuses more on minimizing the average waiting time. In order to improve efficiency.

[0085] In one embodiment, the maximum return time is determined according to the following formula. :

[0086]

[0087] in, For equipment Update frequency Calculate the latency for the device. This is due to device communication delay. Predicted equipment Total time from the start of local training to successful upload and update.

[0088] In one embodiment, the method further includes: receiving local training results returned by the target device; weighting and aggregating the local training results based on the size of the local dataset of the target device, and updating the global parameters of the network to be trained. The server receives local training results uploaded from each target device. Local training results typically refer to model weights, i.e., the new model parameters obtained by the device after local training, and model gradients, i.e., the amount of model update calculated by the device during local training. The server weights the local training results based on the size of the local dataset of the target device. The global model is updated by taking a weighted average of all received local training results.

[0089] In one embodiment, the server collects the device computing latency for each device. Device communication delay And through local datasets Data feature vectors obtained through feature extraction Devices are clustered based on the cosine similarity of their data feature vectors, grouping devices with similar data distributions into the same cluster. Within each cluster, the size of the device data is then considered. And device type weight Calculate the contribution of each device Construct the objective function As the core of scheduling decisions. This represents the average contribution of a subset of devices, indicating the value of the data. This represents the average waiting time of a subset of devices, and is a representation of the system efficiency cost. The weights are dynamically decaying to allow for a switch between prioritizing convergence in the early stages of training and prioritizing efficiency in the later stages. A greedy random search strategy is employed, starting with high-quality initial solutions (the best devices in each cluster) that satisfy the cluster coverage constraint. Iteratively, random devices are added to find a subset of devices that maximize the objective function value. Within this selected subset, the update frequency of each device is fine-tuned to find the optimal frequency allocation scheme that maximizes the objective function value. The target devices are updated according to their assigned frequencies. Local training is performed, with high-intensity devices executing more local iterations. The server adjusts its processing based on the amount of data generated by the device. The returned model updates are weighted and averaged to generate a new generation of global model. Devices that time out or experience consecutive disconnections are discarded or excluded to ensure the system's robustness in dynamic environments. Through the above systematic design, this solution can automatically and dynamically select the optimal combination of devices and their best participation intensity in each training round, thereby significantly improving model accuracy, greatly reducing training latency, and enhancing system robustness.

[0090] In one embodiment, to verify the effectiveness of the adaptive device training method proposed in this invention, an experiment was conducted on the publicly available image classification dataset CIFAR-10. The method was compared with three baseline methods. The experiment involved 20 users, and the data distribution was a scenario with 20% uneven label distribution. The global model used a convolutional neural network, with an initial contribution weight of 1.0, a decay rate of 0.02, and device type weights of 1.0 for desktops, 0.5 for laptops, and 0.2 for mobile devices. The maximum update frequency was limited to 6. The experimental results are shown in Table 1. On the CIFAR-10 dataset, the maximum accuracy of the proposed method was 0.72, an improvement of 5.88% compared to the best comparison method of 0.68 (maximum accuracy improvement of 0.04); the average waiting time was 4.20 s, a reduction of 65.38% compared to the best comparison method of 12.13 s (average waiting time reduction of 7.93 s). These results verify the effectiveness of this invention in significantly reducing training waiting time while improving model accuracy.

[0091] Table 1: Comparison of algorithm performance between the present invention and baseline methods

[0092]

[0093] Figure 1 This is a flowchart illustrating a heterogeneous federated learning training method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0094] In one embodiment, such as Figure 2The diagram illustrates the overall flowchart of a heterogeneous federated learning method. This diagram showcases the complete workflow of the proposed heterogeneous federated learning method, comprising the following seven core steps: Start: Initiating the federated learning training task. Local Information Awareness: Each participating device autonomously detects its computing power and network bandwidth, and extracts local data feature vectors. Information Upload: Devices upload resource status information and data features to the server. Server-Side Clustering: The server groups and clusters devices based on data feature similarity. User Selection and Scheduling: The server dynamically selects a subset of devices to participate in training based on contribution and waiting time metrics, using an objective function. Update Frequency Allocation: Differentiated update frequencies are assigned to selected devices to determine their participation intensity. Federated Training Execution: Selected devices perform local training according to the assigned frequencies, and the server aggregates and updates to generate a new generation of global models. This process, through a closed-loop mechanism of awareness-clustering-optimization-execution, achieves intelligent scheduling of heterogeneous devices while protecting data privacy, effectively balancing model accuracy and training efficiency.

[0095] In one embodiment, a heterogeneous federated learning training apparatus (not shown in the figure) is provided, comprising:

[0096] The memory is configured to store instructions;

[0097] The processor is configured to retrieve instructions from memory and, when executing instructions, to implement any of the above-mentioned methods for heterogeneous federated learning training.

[0098] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and heterogeneous federated learning training methods can be implemented by adjusting kernel parameters.

[0099] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0100] This application provides a storage medium storing a program that, when executed by a processor, implements the above-described method for training heterogeneous federated learning.

[0101] This application provides a processor for running a program, wherein the program executes the above-described method for training heterogeneous federated learning during runtime.

[0102] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computational and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a training method for heterogeneous federated learning.

[0103] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0104] This application provides a computer (electronic) device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above methods for heterogeneous federated learning training.

[0105] This application also provides a computer program product that, when executed on a data processing device, is adapted to execute a program that initializes steps for a heterogeneous federated learning training method.

[0106] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0112] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0114] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A heterogeneous federated learning training method, characterized in that, The method includes: Acquire local resource status information of multiple participating devices. The local resource status information includes at least the computational latency, communication latency, and data feature vector of each participating device. The data feature vector is obtained by feature extraction from a local dataset. The local dataset is used for local model training of the multiple participating devices. Based on the similarity of the data feature vectors, the multiple participating devices are clustered into multiple device clusters; The contribution of each participating device in each device cluster and the average waiting time of all participating devices in each device cluster are determined, wherein the contribution is determined based on the local resource status information of the participating devices contained in the device cluster to which the participating device belongs; For each device cluster, the target device in the device cluster is selected based on the contribution and average waiting time of each participating device in the cluster. An update frequency is assigned to each target device, and global parameters of the network to be trained are sent to each target device so that each target device can start local training of the network to be trained based on the global parameters and the update frequency. The step of selecting the target device in the device cluster based on the contribution and average waiting time of each participating device in the device cluster includes: The optimal device in each device cluster is determined based on the contribution of each participating device in the device cluster. Construct a first subset of devices based on the optimal devices; The objective function is solved based on the average contribution and average waiting time of all the best devices in the first device subset to obtain the first function value of the first device subset. The first function value represents the utility score of the device subset and its update frequency. The update frequency refers to the participation intensity of the target device in a training round. N participating devices are randomly selected from the multiple device clusters and added to the first device subset to obtain the second device subset; The objective function is solved based on the average contribution and average waiting time of all devices in the second device subset to obtain the second function value of the second device subset; If the second function value is less than or equal to the first function value, the step of randomly selecting N participating devices from the plurality of device clusters and adding them to the first device subset to obtain the second device subset is executed again until all participating devices are added to the first device subset to solve the objective function and calculate the corresponding objective function value. If the second function value is greater than the first function value, the second device subset is used as the new first device subset, and the step of randomly selecting N participating devices from the plurality of device clusters and adding them to the first device subset is executed again to obtain the second device subset, until all participating devices are added to the first device subset to solve the objective function and calculate the corresponding objective function value; The devices included in the first subset of devices obtained are identified as the target devices.

2. The method according to claim 1, characterized in that, The method of allocating update frequencies for each target device includes: Determine the objective function value corresponding to the subset of target devices containing all target devices; Determine the initial update frequency for each target device; The initial update frequency of at least one target device in the target device subset is adjusted sequentially, and the objective function value of the target device subset after each adjustment is determined, until the maximum value of the objective function value is determined; The update frequency corresponding to each target device at the maximum value is determined as the target update frequency for each target device.

3. The method according to claim 1, characterized in that, The objective function The expression is as follows: in, The weighting coefficient for the average contribution. The average contribution of the device subset. The average waiting time for a subset of devices. For the scheduling scheme, It is a time variable.

4. The method according to claim 3, characterized in that, The average contribution is determined according to the following formula. : in, Belongs to cluster equipment Contribution For indicator functions, For the total number of devices, For equipment The update frequency, indicating the function For indicating device Whether it is selected.

5. The method according to claim 3, characterized in that, The average waiting time is determined according to the following formulas. and the weighting coefficient of the average contribution : in, For the maximum return time, Calculate the latency for the device. For device communication delay, For indicator functions, As the initial value, This refers to the attenuation rate. For the total number of devices, For equipment The update frequency, indicating the function For indicating device Whether it is selected.

6. The method according to claim 5, characterized in that, The maximum return time is determined according to the following formula. : in, For equipment Update frequency Calculate the latency for the device. This refers to device communication delay.

7. The method according to claim 1, characterized in that, The method further includes: Receive the local training results returned by the target device; The local training results are weighted and aggregated based on the size of the local dataset of the target device, and the global parameters of the network to be trained are updated.

8. A heterogeneous federated learning training device, characterized in that, include: The memory is configured to store instructions; A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the heterogeneous federated learning training method according to any one of claims 1 to 7.

9. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the heterogeneous federated learning training method according to any one of claims 1 to 7.

Images