Federal training scheduling method based on hierarchical dynamic selection and diversity constraint

Abstract

This invention discloses a federated training scheduling method based on hierarchical dynamic selection and diversity constraints, belonging to the field of distributed machine learning. This invention aims to address the problems of low training efficiency and unstable convergence in existing federated learning methods under environments with both statistical and system heterogeneity. This invention obtains a set of available clients, maintains latency estimates for each client and hierarchically divides them, determines the sampling quota for each layer based on the average latency, historical participation, and fairness constraints, constructs a client data difference graph within each layer, and uses a greedy decentralized selection strategy with diversity constraints to determine the set of participating clients. Local updates for selected clients are aggregated and the client states are updated. This invention is applicable to distributed training scenarios such as mobile terminal collaborative learning, vehicle networking, and edge intelligent analysis.

Description

Technical Field

[0001] This invention belongs to the field of distributed machine learning technology, and in particular relates to a federated training scheduling method based on hierarchical dynamic selection and diversity constraints. Background Technology

[0002] With the rapid development of mobile internet, edge computing, and IoT technologies, massive amounts of data are increasingly being generated in distributed terminal devices, sensor nodes, and edge computing units. Federated learning, as a distributed training method, involves each client using local data to train the model, uploading only parameter update results to the server without directly transmitting the original data, thus achieving multi-terminal joint modeling while protecting privacy. However, in actual deployments, federated learning is usually not in an ideal homogeneous environment. On the one hand, different clients often have different data sources, sample sizes, label ratios, and sampling methods, leading to significant differences in local update directions. This statistical heterogeneity makes the updates received by the server more discrete, thus affecting the convergence process of the global model. On the other hand, different clients have different computing capabilities, network quality, remaining battery power, and communication conditions. Under synchronous training mechanisms, the server often needs to wait for the selected client to return updates before performing aggregation. A few slow-responding terminals can prolong the entire training round, forming a typical system heterogeneity problem.

[0003] In existing solutions, some methods prioritize training efficiency, favoring clients with faster response times. For example, in mobile terminal collaborative learning scenarios, while these methods can shorten single-round processing time, they can also lead to a long-term concentration of participating clients on a few high-performance devices. If the data distribution of these devices is relatively similar, it weakens the data coverage of each training round, thus affecting the model's adaptability to long-tail samples or edge scenarios. Another type of method focuses on selecting clients with more diverse data to improve model generalization ability and global training quality. For example, in vehicle-to-everything (V2X) joint perception scenarios, if this type of method does not simultaneously consider client latency, it may frequently select high-latency clients, leading to communication congestion, increased synchronization round duration, and decreased overall training efficiency. Therefore, in federated learning scenarios where statistical and system heterogeneity coexist, how to improve the diversity of data distribution among participating clients in each round while ensuring training efficiency, and how to reduce the impact of slow node tailing while ensuring long-term participation fairness, have become core technical issues in current federated learning scheduling optimization. While existing publicly available solutions have explored aspects such as device selection, bandwidth allocation, and participation rate control in hierarchical federated learning, they still lack a unified scheduling mechanism for environments with both statistical and system heterogeneity. This makes it difficult to effectively balance multiple objectives such as latency-aware hierarchical management, diverse constraint selection, dynamic quota allocation, and adaptive state updates. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a federated training scheduling method based on hierarchical dynamic selection and diversity constraints, thereby resolving the issues present in the prior art.

[0005] Firstly, to achieve the above objectives, this invention provides a federated training scheduling method based on hierarchical dynamic selection and diversity constraints, comprising the following steps:

[0006] S1. The server obtains the clients that are available in the current training round and maintains a latency state for each client that reflects its overall response capability.

[0007] S2. The server divides the clients into multiple layers based on the latency status, where clients within the same layer have similar response capabilities;

[0008] S3. The server dynamically allocates a sampling quota to each layer based on the average response capability of clients at each layer and the historical participation of each layer. The sampling quota determines the number of clients selected from that layer.

[0009] S4. For each layer, the server selects clients that meet its sampling quota based on the degree of difference between the local data distribution of clients within the layer, so that the selected client set has high discreteness in data distribution.

[0010] S5. The server merges the selected clients from each layer to form a set of clients participating in the current round of training, and sends global model parameters to the clients in this set for local training and update aggregation.

[0011] S6. The server updates the latency status, data distribution information, and historical participation status based on the actual response of the client in this round of training, for use in scheduling decisions for subsequent rounds.

[0012] Secondly, the present invention also provides a federated training scheduling system based on hierarchical dynamic selection and diversity constraints, for implementing a federated training scheduling method based on hierarchical dynamic selection and diversity constraints, the system comprising:

[0013] The client state maintenance module is used to obtain the clients that are available in the current training round and maintain a latency state for each client that reflects its overall response capability.

[0014] A client-side layering module is used to divide the client into multiple layers based on the latency status, wherein clients within the same layer have similar response capabilities;

[0015] The dynamic quota allocation module is used to dynamically allocate a sampling quota to each layer based on the average response capability of clients at each layer and the historical participation of each layer. The sampling quota determines the number of clients selected from that layer.

[0016] The layer diversity selection module is used to select clients that meet the sampling quota of each layer based on the degree of difference between the local data distribution of clients within the layer, so that the selected client set has high discreteness in data distribution.

[0017] The training participation aggregation module is used to merge the selected clients from each layer to form a set of clients participating in the current round of training, and to send global model parameters to the clients in this set for local training and updating aggregation;

[0018] The state update module is used to update the latency state, data distribution information, and historical participation status based on the actual response of the client in this round of training, for use in scheduling decisions for subsequent rounds.

[0019] Thirdly, the present invention also provides a computer terminal device, comprising:

[0020] One or more processors;

[0021] A memory, coupled to the processor, for storing one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the federated training scheduling method based on hierarchical dynamic selection and diversity constraints in the first aspect described above.

[0023] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the federated training scheduling method based on hierarchical dynamic selection and diversity constraints in the first aspect described above.

[0024] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the federated training scheduling method based on hierarchical dynamic selection and diversity constraints described in the first aspect above.

[0025] Compared with the prior art, the present invention has the following advantages and technical effects:

[0026] This invention provides a federated training scheduling method based on hierarchical dynamic selection and diversity constraints. The method uses a latency-aware hierarchical mechanism to group clients with different response performances, allowing clients with similar response capabilities to compete within the same scheduling layer. This reduces the tail effect of slow nodes on the duration of synchronous training rounds and improves overall training efficiency. The method employs an intra-layer diversity constraint selection mechanism to increase the data distribution diversity of the participating client sets in each round while ensuring system efficiency. This allows the global model to be updated on richer data patterns, contributing to improved model convergence stability, global accuracy, and generalization ability. The method uses a dynamic sampling quota mechanism to incorporate the layer's average latency and historical participation into the sampling decision, achieving a balance between efficiency and fairness. This avoids data coverage bias and training unfairness caused by a few high-performance clients being repeatedly selected over a long period. Finally, the method uses a state adaptive update mechanism to dynamically adjust subsequent scheduling behavior based on the actual operating conditions of the clients, enhancing the system's robustness and long-term stability in real-world complex network environments. Attached Figure Description

[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0028] Figure 1 This is a schematic diagram of the overall architecture of the federated training scheduling system according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the delay-aware hierarchical scheduling process according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram illustrating the construction of an intra-layer data difference relationship graph according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the model aggregation and state update process in an embodiment of the present invention;

[0032] Figure 5 The graph shows the changes in accuracy and loss of various methods with communication rounds and cumulative time under different degrees of data heterogeneity on Fashion-MNIST according to the embodiments of the present invention.

[0033] Figure 6 The graph shows the changes in accuracy and loss of each method with the number of communication rounds and cumulative time under different levels of data heterogeneity on CIFAR-10 in this embodiment of the invention.

[0034] Figure 7 The graph shows the changes in accuracy and loss of the heterogeneous friendly method for different systems on Fashion-MNIST as a function of communication rounds and cumulative time in an embodiment of the present invention.

[0035] Figure 8 The graph shows the accuracy and loss of the heterogeneous friendly method for different systems on CIFAR-10 in this embodiment of the invention as a function of communication rounds and cumulative time. Detailed Implementation

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0038] Example 1

[0039] This embodiment provides a federated training scheduling method based on hierarchical dynamic selection and diversity constraints, including:

[0040] S1. The server obtains the clients that are available in the current training round and maintains a latency state for each client that reflects its overall response capability.

[0041] S2. The server divides the clients into multiple layers based on the latency status, where clients within the same layer have similar response capabilities;

[0042] S3. The server dynamically allocates a sampling quota to each layer based on the average response capability of clients at each layer and the historical participation of each layer. The sampling quota determines the number of clients selected from that layer.

[0043] S4. For each layer, the server selects clients that meet its sampling quota based on the degree of difference between the local data distribution of clients within the layer, so that the selected client set has high discreteness in data distribution.

[0044] S5. The server merges the selected clients from each layer to form a set of clients participating in the current round of training, and sends global model parameters to the clients in this set for local training and update aggregation.

[0045] S6. The server updates the latency status, data distribution information, and historical participation status based on the actual response of the client in this round of training, for use in scheduling decisions for subsequent rounds.

[0046] As one implementation method in this embodiment, the process of dividing the client into multiple layers in S2 includes: the server sorts the client latency values ​​from smallest to largest, and according to the preset number of layers or quantile threshold, the sorted client sequence is sequentially divided into different layers, so that the client with the smaller latency value is divided into the front layer, and the client with the larger latency value is divided into the back layer.

[0047] As one implementation method in this embodiment, the process of dynamically allocating a sampling quota for each layer in S3 includes: the server calculates the sampling score of each layer based on the average latency status value of each layer, the cumulative situation of the sampling quota allocated to each layer in historical rounds, and a preset fairness adjustment coefficient; then the sampling score of each layer is normalized; and the normalized ratio is multiplied by the target total number of samples in this round to obtain the sampling quota of each layer.

[0048] As one implementation method in this embodiment, the process of selecting based on the degree of difference between the local data distribution of clients within the layer in S4 includes: the server constructs a representation vector for each client within the layer to characterize its local data distribution characteristics, calculates the degree of difference between any two client representation vectors within the layer, constructs a data difference relationship graph with clients as nodes and the degree of difference as edge weights, and then uses a greedy algorithm to iteratively select clients from the data difference relationship graph, each time selecting the client with the largest average degree of difference with all clients in the current selected set to add, until the sampling quota of the layer is reached.

[0049] As one implementation method in this embodiment, the local training and update aggregation process in S5 includes: the server only aggregates the set of clients that successfully return local model updates within a preset time window; for clients that fail to return successfully within the time window, the server records the abnormal events in this training and corrects the delay status of the corresponding clients in subsequent rounds based on the recorded abnormal events.

[0050] As one implementation method in this embodiment, the process of updating the historical participation status in S6 includes: the server maintaining the historical selection count of each client, and incrementing the historical selection count of a client selected in this round of training; the server also maintaining the historical usage count of each layer, and incrementing the historical usage count of a layer that is allocated a sampling quota in this round of training. The historical selection count and layer usage count are used to apply fairness constraints when calculating the sampling quota of each layer in subsequent rounds S3.

[0051] In one embodiment of the present invention, the federated learning system includes a server and multiple clients. The server can be a cloud server, an edge master node, or a central node with global scheduling capabilities; the clients can be mobile phones, tablets, vehicle terminals, edge cameras, smart sensors, industrial equipment, wearable devices, or other terminal devices with local training capabilities.

[0052] Suppose the system has M candidate clients. At the start of the t-th round of training, the server can obtain the set of currently online and available clients that meet the participation criteria. Meeting the participation conditions may include, but is not limited to: battery level exceeding a threshold, device being idle, network connectivity being available, having basic computing resources, and policy permission to participate in training.

[0053] The server maintains latency status for each client k. The latency status is used to reflect the client's overall response capability in historical training rounds, including local training time, model upload time, network round-trip latency, edge forwarding time, or other indicators that can characterize response speed.

[0054] After each round of training, for clients that successfully participate and return, the server updates the latency status according to the exponential moving average formula; for clients that fail to return, the server can update the latency status according to the timeout limit, penalty factor or default latency value to prevent their historical status from remaining at outdated values ​​for a long time.

[0055] After obtaining the current latency estimates for each client, the server analyzes the available client set. The clients in the system are sorted in ascending order of latency estimates and divided into layer sets according to a preset number of layers T. , , ..., Normally, The client in the middle has a relatively low response latency, while The client in the middle has a relatively high response latency.

[0056] By using layered processing, clients with similar response capabilities can be grouped into the same layer, thereby avoiding the simultaneous selection of clients with significantly different performance in the same round and reducing the problem of increased synchronization waiting time caused by a very small number of high-latency clients.

[0057] In one embodiment, the server further uses a count for each maintenance layer. and layer average delay Layer usage count is used to represent the cumulative number of sampling quotas or selected clients allocated to the corresponding layer in the previous rounds, while layer average latency is used to reflect the overall response speed of the layer.

[0058] To strike a balance between efficiency and fairness, the server considers factors such as average tier latency, tier usage count, and fairness factor. Calculate the sampling score for each layer. A lower average layer latency generally means that the layer is more suitable for priority sampling; however, if a layer has been frequently used historically, its score can be appropriately reduced through a penalty term to avoid over-concentration of selection.

[0059] In a preferred embodiment, the sampling scores of each layer are normalized using softmax to obtain the sampling ratio of each layer, and then the total number of samples in this round is used as the target. Determine the sampling quota for each layer For deviations caused by rounding down to the nearest integer, a margin compensation strategy can be used to correct them, ensuring that the sum of quotas across all tiers equals the total quota. .

[0060] After completing the tier quota allocation, the server enters the client selection phase within the tier. At each layer, the server constructs a data distribution representation for the clients within that layer. The data distribution representation can be composed of privacy-preserving statistical summaries, tag frequency histograms, prototype vectors, feature mean and variance, embedding distribution summaries, or other lightweight statistical descriptions uploaded by the client.

[0061] It should be noted that, to avoid compromising the privacy of raw data, the data distribution does not require uploading the original training samples; only the distribution level information that has been anonymized, compressed, or transformed is required. The server uses this information to estimate the data differences between clients, rather than directly accessing any user's raw data.

[0062] For any two clients p and q in the same layer, the server calculates their difference. The degree of difference can be expressed as Euclidean distance, cosine distance, KL divergence, Wasserstein distance, or prototype vector distance, etc. This allows for the construction of a graph showing the data difference within a layer. In this context, vertices represent clients, and edge weights represent the degree of data difference between clients.

[0063] In a preferred embodiment, the server employs a greedy distributed selection algorithm from the graph. Selected from The specific process is as follows: First, select a starting client from the candidate clients. This starting client can be randomly generated or selected from clients with high representativeness, low latency, or a large sample size. Then, in each iteration, calculate the average difference between the remaining candidate clients and the currently selected set, and select the client with the largest average difference to add to the set. Repeat the above process until the quota for that layer is filled.

[0064] By adopting the above strategy, the selected client set has a higher degree of dispersion in data distribution, which enables the coverage of richer data patterns in each round of training and avoids training being concentrated on a few clients with highly similar data distributions.

[0065] In some embodiments, the server can also incorporate sample size weights, category coverage rewards, recent non-selection rewards, or task relevance constraints into the greedy selection of targets to adapt to different business scenarios. For example, in tasks with extremely imbalanced categories, the probability of selecting clients with scarce category samples can be prioritized.

[0066] After the client selection at each layer is completed, the server merges the selected clients from all layers to obtain the current round's participant set. It then broadcasts the global model parameters to the client. The client performs several rounds of local training based on local data to obtain local model updates. .

[0067] In a synchronous federated learning scenario, the server collects the set of clients that have successfully returned results after the waiting time window has expired. and only for Updates are aggregated within the edge nodes; in semi-synchronous or hierarchical edge aggregation scenarios, local aggregation can be completed first within the edge nodes, and then global aggregation can be completed by the central server.

[0068] Aggregation methods can include weighted averages or a combination of factors such as sample size, client credibility, update norm, historical stability, or drift correction. For updates with significant outliers, pruning, robust aggregation, or outlier removal strategies can be introduced to further improve training stability.

[0069] After each round of aggregation, the server updates the client's state. Specifically, this includes updating the latency state based on the time taken to return data in this round. Update the data distribution representation based on the distribution summary uploaded by the client or the most recent statistical information. For selected clients, update their historical selection count. Update the layer usage count for the used layer. For disconnections, timeouts, or failures, the client records an exception flag.

[0070] In the long run, the number of historical selections and layer usage counts can serve as important bases for fairness constraints, avoiding the frequent and repeated use of a few fast nodes and ensuring that model training has a more balanced coverage of different terminal groups, different network condition groups, and different data distribution groups.

[0071] In a mathematical implementation example, the server can map latency status, hierarchical information, diversity score, and fairness score together into a comprehensive scheduling objective function, and obtain the participant set under the premise of satisfying the sampling quantity constraint, the intra-layer quota constraint, and the minimum coverage constraint.

[0072] Figure 1 The overall architecture of the invention is illustrated. The server is responsible for global scheduling, model broadcasting, state maintenance, and model aggregation; multiple clients complete training based on their local data and then upload updates; if necessary, an intermediate aggregation layer can be set up between the cloud and edge nodes to adapt to a larger-scale terminal network.

[0073] Figure 2 This illustrates the latency-aware tiering process. The server first collects historical latency and current response information from available clients to form a smoothed latency estimate, and then assigns clients to different tiers based on the ranking results. This process allows for a clear distinction between high-response and low-response clients.

[0074] Figure 3 This illustrates how the data heterogeneity graph within a layer is constructed. For clients at the same layer, the server calculates pairwise dissimilarity based on their distribution representation and uses a graph structure to characterize the data heterogeneity relationships between clients, providing a basis for the subsequent selection process of data distribution diversity.

[0075] Figure 4 The aggregation and state update process of this invention is illustrated. The server aggregates the successfully returned client updates and writes the training feedback from this round back to the state management module, providing a more accurate decision-making basis for the next round of training.

[0076] The following is an application example. In a federated learning scenario for mobile terminal input methods, different users exhibit significant differences in language habits, input times, hot word distribution, and usage behaviors, resulting in statistical heterogeneity. Simultaneously, different mobile phone models, network environments, and power-saving strategies contribute to system heterogeneity. By employing the federated training scheduling method of this invention, terminals with faster response times can be prioritized in low-latency layers, while user terminals with significantly different data distributions are prioritized across all layers. This improves the generalization ability of the hot word prediction and next-word recommendation models without significantly increasing training time.

[0077] For example, in vehicle-to-everything (V2X) joint perception scenarios, different vehicles collect data on road environments, weather conditions, traffic density, and driving behavior patterns that vary significantly, resulting in obvious data heterogeneity. Furthermore, the connection quality, computing resources, and workload between different vehicle terminals and roadside units also differ. By adopting this invention, vehicles with high latency or weak network connectivity can be relegated to a later layer, and their impact on training rounds can be reduced through quota control. Simultaneously, vehicles with scenario-specific characteristics can be selected within each layer for training to improve the perception model's adaptability to complex traffic scenarios.

[0078] In smart healthcare scenarios, training data from different hospitals or medical institutions is often highly heterogeneous due to variations in equipment, patient sources, disease types, and imaging protocols. This invention effectively improves the coverage of different diseases and imaging conditions during federated training without aggregating raw medical data, resulting in better robustness of the model when deployed across institutions.

[0079] In industrial internet scenarios, monitoring data from different production lines, workstations, and factories exhibit significant distribution variations, while the communication stability of terminal devices also differs. The latency hierarchical and diversity selection mechanism of this invention can reduce training waiting time while improving the fault detection model's ability to identify various operating conditions.

[0080] Furthermore, this invention is not limited to a specific model type. Whether for image classification, object detection, speech recognition, time series prediction, anomaly detection, or recommendation ranking tasks, as long as there is a need for multi-client distributed privacy-preserving training, and the clients exhibit both data heterogeneity and system heterogeneity, this invention is applicable.

[0081] This invention is not limited to a specific representation of data differences. Any method that can effectively approximate the differences in client data distribution, without violating privacy principles, can be used as an alternative in the implementation of this invention. For example, category statistical vectors, cluster centers, higher-order moments of the embedding space, hash digests, or distribution descriptions processed with differential privacy can be used.

[0082] This invention is not limited to a fixed number of layers. For scenarios with a small number of clients and weak differences in system performance, fewer layers can be used to reduce scheduling complexity; for scenarios with a large number of terminals, strong network fluctuations, or significant device heterogeneity, the number of layers can be appropriately increased to improve layering accuracy and scheduling flexibility.

[0083] For the intra-layer selection algorithm, this invention is not limited to greedy distributed selection. Any alternative implementation of this invention that can improve the data distribution diversity of the client set while considering the layer quota constraint is also acceptable, such as maximum coverage selection, sub-modulus optimization, cluster representative point selection, deterministic annealing selection, or heuristic multi-objective selection.

[0084] For fairness control, this invention can construct different forms of penalty and reward terms based on the client's historical selection count, layer usage count, average waiting rounds, nearest neighbor similarity exposure degree, or task importance distribution. Any extension of this invention that can prevent long-term training bias towards a few clients and improve overall coverage can be considered a reasonable extension.

[0085] In engineering deployment, the server can encapsulate the scheduling module of this invention as an independent software service, or it can be embedded in existing federated learning platforms, edge scheduling platforms, or cloud-edge collaborative training frameworks. The module can be executed by a general-purpose processor, or by dedicated acceleration hardware, programmable logic devices, or a combination thereof.

[0086] In summary, this invention integrates system heterogeneity and statistical heterogeneity into a federated scheduling framework through delay-aware layering, intra-layer diversity selection, dynamic quota allocation, and adaptive state updates, effectively improving the overall performance of federated learning training in complex heterogeneous environments.

[0087] The above description is only a preferred embodiment of the present invention. For those skilled in the art, several modifications, substitutions and improvements can be made without departing from the spirit and essence of the present invention, and all such modifications, substitutions and improvements should fall within the protection scope of the present invention.

[0088] Experimental verification:

[0089] To verify the effectiveness of the federated training scheduling method based on hierarchical dynamic selection and diversity constraints proposed in this invention (hereinafter referred to as "this method") in a statistically heterogeneous and system heterogeneous environment, a systematic comparative experiment was conducted in a controllable federated learning simulation environment.

[0090] Experimental environment and parameter settings:

[0091] The experiment was conducted on a computer equipped with an Intel Core i5-11400H processor (2.70GHz), a GeForce RTX 5060 GPU, and running Windows 11. A federated learning simulation environment was built using PyTorch 1.13 to simulate a centralized server-multiple client round-robin training paradigm.

[0092] Federated learning parameters: total number of clients K=100, participation ratio per round C=0.2, number of communication rounds T=200, number of local training rounds E=5. The datasets used were Fashion-MNIST and CIFAR-10, employing DNN and CNN as classification models respectively, with a batch size of 64 for both, and initial learning rates of 0.01 and 0.1 respectively (CIFAR-10 decays at 0.99 per round).

[0093] System heterogeneous simulation:

[0094] To approximate the differences in terminals and operational uncertainties in real-world scenarios, a client state machine driven by a virtual clock is used to introduce system heterogeneity, including:

[0095] Client availability: online / offline state transition, simulating user behavior and device sleep;

[0096] Client connectivity: Selecting this option may cause connection interruption, simulating network fluctuations;

[0097] Client integrity: Local training workload is randomly reduced due to insufficient computing power or task preemption.

[0098] Client responsiveness: End-to-end latency Communication latency is related to network speed, while computation latency is related to device computing power and effective workload. Two latency distributions were used in the experiment: uniform distribution (flat heterogeneous) and exponential distribution (long-tailed heterogeneous), the latter being used to highlight the tailing effect of slow nodes.

[0099] All comparison methods share the exact same available set of sequences, latency trajectories, and disconnection events, ensuring that performance differences stem solely from the algorithm design itself.

[0100] Data heterogeneity (Non-IID) settings:

[0101] The client-side local tag distribution is generated using the Dirichlet distribution, through parameters. Controlling heterogeneity strength: The smaller the value, the stronger the data heterogeneity. Experimental setup. (Strong heterogeneity) and (Medium heterogeneity) Two scenarios.

[0102] To comprehensively evaluate this method, representative algorithms for data heterogeneity and system heterogeneity were selected as controls:

[0103] Baseline solution: FedAvg (random selection + mean aggregation);

[0104] Data heterogeneous friendly methods: FedProx, FedNova, SCAFFOLD, Ditto, MOON;

[0105] System heterogeneous friendly methods: TiFL (latency-based hierarchical scheduling), OORT (joint scoring selection), PowerOfChoice (candidate set selection), FedAsync (asynchronous aggregation), and FedBuff (buffered aggregation).

[0106] All methods operate under the same data partition, the same random seed, and completely consistent client latency patterns to ensure fair comparison.

[0107] Analysis of experimental results:

[0108] Comparison with data heterogeneous friendly methods:

[0109] In strong heterogeneity ( ) and moderate heterogeneity ( Under the given conditions, the comparison results between this method and FedAvg, FedProx, FedNova, MOON, Ditto, and SCAFFOLD are shown in Table 1 and... Figure 5 , Figure 6 As shown.

[0110] Table 1. Performance comparison of different methods on CIFAR-10 and Fashion-MNIST.

[0111]

[0112] Results analysis:

[0113] In terms of final accuracy, this method is on par with or even slightly better than various data heterogeneous friendly methods (e.g., it reaches 51.68% in CIFAR-10, which is higher than other methods); although it is slightly lower than SCAFFOLD in Fashion-MNIST, the latter suffers from severe performance degradation due to control variable update noise in heterogeneous system environments (CIFAR-10 is only 40.11%), while this method remains stable.

[0114] In terms of training efficiency, this method has significant advantages: the total time on Fashion-MNIST is approximately 47 hours, compared to approximately 87 hours for other synchronous methods (a reduction of approximately 46%); on CIFAR-10, it takes approximately 70 hours, compared to approximately 130 hours for other methods (a reduction of approximately 46%). This indicates that this method effectively suppresses the slow node tailing effect and significantly reduces time costs through latency-aware layering and dynamic quota allocation.

[0115] Figure 5 and Figure 6 The curves showing the change in accuracy with the number of communication rounds and cumulative time are presented. In the acc-round dimension, the method shows a faster increase in the early stages and a higher plateau in the mid-to-late stages; in the acc-time dimension, the curve of the method shifts to the left overall, indicating that the time required to achieve the same accuracy is significantly shorter.

[0116] Comparison with system heterogeneity-friendly methods:

[0117] The comparison results of this method with TiFL, OORT, PowerOfChoice, FedAsync, and FedBuff are shown in Table 1. Figure 7 , Figure 8 As shown.

[0118] Table 2 Performance comparison of different methods on CIFAR-10 and Fashion-MNIST

[0119]

[0120] Results Analysis: While FedAsync has a very short processing time (2–3 hours), its accuracy drops significantly (only 66.97% for Fashion-MNIST), indicating that purely asynchronous updates cannot guarantee model quality under highly heterogeneous environments. TiFL, employing hierarchical scheduling, outperforms FedAvg in terms of time efficiency (53.96h vs 87.53h), but its accuracy is slightly lower (81.58% vs 82.64%), suggesting that simple hierarchical scheduling may sacrifice data diversity. OORT and PowerOfChoice offer trade-offs in accuracy and time, but neither comprehensively surpasses this method.

[0121] This method maintains accuracy comparable to or even higher than FedAvg, while taking only about 87% of the time of TiFL (47h vs 54h), and significantly outperforms other methods. This indicates that by constraining diversity within layers, this method maintains data coverage while accelerating the process layer by layer, achieving a win-win situation for both efficiency and accuracy.

[0122] Synergistic effect analysis: To further verify the synergistic effect of the mechanism of this invention, comparative reasoning was performed based on the above experimental results:

[0123] System optimization only (analogous to TiFL): By reducing the impact of slow nodes through layering, efficiency was significantly improved (time decreased from 87h to 54h), but accuracy decreased slightly (84.67%→83.98%), indicating that simply pursuing efficiency may result in a loss of data diversity.

[0124] Data optimization only (analogous to FedProx, etc.): It alleviates statistical heterogeneity through regularization or correction, and maintains a high level of accuracy, but takes about 87 hours, which is comparable to FedAvg. It fails to solve the time waste caused by system heterogeneity.

[0125] This invention not only maintains accuracy comparable to (or even slightly higher than) the optimal data optimization method, but also further reduces the time consumption to 47 hours (a 13% reduction compared to TiFL). This indicates that the diversity selection does not increase the time overhead, but rather promotes convergence by optimizing the combination of participating clients, complementing the hierarchical mechanism.

[0126] Specifically, on the highly heterogeneous CIFAR-10, the accuracy of this method (51.68%) is significantly higher than that of TiFL (49.63%) and FedProx (50.36%), while the execution time (70.72 h) is much lower than that of FedAvg (131.29 h) and TiFL (80.92 h). This demonstrates that the two-stage scheduling mechanism of this invention produces a significant synergistic effect.

[0127] Application prospects:

[0128] This invention can be widely applied to scenarios that require distributed model training under privacy protection, such as mobile terminal collaborative learning, smart healthcare, vehicle networking, industrial internet, smart security, edge vision analysis, and IoT collaborative sensing.

[0129] In the above scenarios, there are a large number of terminals, significant differences in device performance, complex network environment, and obvious non-independent and identically distributed data distribution. This invention can improve training efficiency, training stability, and model generalization ability without increasing too much scheduling overhead, and has good application value and industrialization prospects.

[0130] Based on this, the federated training scheduling method based on hierarchical dynamic selection and diversity constraints provided by this invention has at least the following advantages compared with the prior art:

[0131] First, the present invention groups clients with different response performances through a delay-aware hierarchical mechanism, so that clients with similar response capabilities can compete in the same scheduling layer, thereby reducing the trailing effect of slow nodes on the duration of synchronous training rounds and improving overall training efficiency.

[0132] Second, this invention improves the data distribution diversity of the client set participating in each round by using an intra-layer diversity constraint selection mechanism, while ensuring system efficiency. This enables the global model to be updated on richer data patterns, which helps to improve the model's convergence stability, global accuracy, and generalization ability.

[0133] Third, this invention incorporates the average layer latency and historical participation into the sampling decision through a dynamic sampling quota mechanism, achieving a balance between efficiency and fairness, and avoiding data coverage bias and training unfairness caused by a few high-performance clients being repeatedly selected over a long period of time.

[0134] Fourth, through the state adaptive update mechanism, this invention can dynamically correct subsequent scheduling behavior according to the actual operation of the client, thereby enhancing the robustness and long-term stability of the system in real and complex network environments.

[0135] Fifth, this invention has good versatility and is compatible with various federated learning aggregation frameworks and deployment architectures. It can be used in centralized federated learning, hierarchical federated learning, edge collaborative learning, and other privacy-preserving distributed training scenarios.

[0136] Based on the above technical solutions, this invention is not only applicable to application scenarios such as mobile terminal collaborative learning, edge intelligent analysis, vehicle-to-everything (V2X) joint perception, industrial internet fault diagnosis, smart healthcare joint modeling, and IoT distributed training, but also suitable for complex environments with a large number of terminal devices, significant network quality fluctuations, substantial differences in device performance, and non-independent, identically distributed local data. In practical engineering deployments, this invention demonstrates high feasibility, scalability, and application promotion value.

[0137] Example 2

[0138] In this embodiment, a computer terminal device is provided, including:

[0139] One or more processors;

[0140] A memory, coupled to the processor, for storing one or more programs;

[0141] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the federated training scheduling method based on hierarchical dynamic selection and diversity constraints described above.

[0142] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described federated training scheduling method based on hierarchical dynamic selection and diversity constraints.

[0143] In this embodiment, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the above-described federated training scheduling method based on hierarchical dynamic selection and diversity constraints.

[0144] In this embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the above-described federated training scheduling method based on hierarchical dynamic selection and diversity constraints.

[0145] The aforementioned program can run on a processor or be stored in memory (or a computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0146] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented by different modules for different steps.

[0147] This embodiment provides such an apparatus or system. The system, referred to as a federated training scheduling system based on hierarchical dynamic selection and diversity constraints, includes:

[0148] The client state maintenance module is used to obtain the clients that are available in the current training round and maintain a latency state for each client that reflects its overall response capability.

[0149] A client-side layering module is used to divide the client into multiple layers based on the latency status, wherein clients within the same layer have similar response capabilities;

[0150] The dynamic quota allocation module is used to dynamically allocate a sampling quota to each layer based on the average response capability of clients at each layer and the historical participation of each layer. The sampling quota determines the number of clients selected from that layer.

[0151] The layer diversity selection module is used to select clients that meet the sampling quota of each layer based on the degree of difference between the local data distribution of clients within the layer, so that the selected client set has high discreteness in data distribution.

[0152] The training participation aggregation module is used to merge the selected clients from each layer to form a set of clients participating in the current round of training, and to send global model parameters to the clients in this set for local training and updating aggregation;

[0153] The state update module is used to update the latency state, data distribution information, and historical participation status based on the actual response of the client in this round of training, for use in scheduling decisions for subsequent rounds.

[0154] As one implementation method in this embodiment, the client-side hierarchical partitioning module includes:

[0155] The client sorting unit is used to sort clients according to their latency state values ​​from smallest to largest.

[0156] The hierarchical execution unit is used to divide the sorted client sequence into different layers according to a preset number of layers or a quantile threshold, so that clients with smaller latency values ​​are assigned to the front layer and clients with larger latency values ​​are assigned to the back layer.

[0157] As one implementation method in this embodiment, the dynamic quota allocation module includes:

[0158] The layer score calculation unit is used to calculate the sampling score of each layer based on the average delay state value of each layer, the cumulative sampling quota allocated to each layer in the historical rounds, and a preset fairness adjustment coefficient.

[0159] The quota determination unit is used to normalize the sampling scores of each layer, and multiply the normalized ratio by the total number of target samples in this round to obtain the sampling quota of each layer.

[0160] As one implementation method in this embodiment, the intra-layer diversity selection module includes:

[0161] The data difference graph construction unit is used to construct a representation vector for each client in the layer to characterize its local data distribution characteristics, and to calculate the difference degree between any two client representation vectors in the layer, thus constructing a data difference relationship graph with clients as nodes and difference degree as edge weights.

[0162] The greedy selection execution unit is used to iteratively select clients from the graph using a greedy algorithm. Each time, the client with the largest average difference from all clients in the current selected set is selected and added until the sampling quota of this layer is reached.

[0163] As one implementation method in this embodiment, the training participation aggregation module includes:

[0164] The successful return monitoring unit is used to collect the set of clients that successfully return local model updates within a preset time window, and to record the abnormal events of clients that fail to return successfully within the time window.

[0165] An aggregation execution unit is used to perform global model aggregation only on the set of clients that have successfully returned;

[0166] The state correction triggering unit is used to correct the delayed state of the corresponding client in subsequent rounds based on the recorded abnormal events.

[0167] As one implementation method in this embodiment, the state update module includes:

[0168] The client fairness recording unit is used to maintain the historical selection count of each client. For clients selected in this round of training, their historical selection count is incremented.

[0169] The layer fairness record unit is used to maintain the historical usage count of each layer. For the layer that is allocated a sampling quota in the current training round, its historical usage count is incremented.

[0170] The constraint transmission unit is used to transmit the historical selection count and layer usage count to the dynamic quota allocation module so that it can apply fairness constraints when calculating the sampling quota for each layer in subsequent rounds.

[0171] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.

[0172] The above implementation method solves the problem of federated training scheduling based on hierarchical dynamic selection and diversity constraints in related technologies, thereby ensuring that the problems existing in the prior art are resolved.

[0173] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A federated training scheduling method based on hierarchical dynamic selection and diversity constraints, characterized in that, Includes the following steps: S1. The server obtains the clients that are available in the current training round and maintains a latency state for each client that reflects its overall response capability. S2. The server divides the clients into multiple layers based on the latency status, where clients within the same layer have similar response capabilities; S3. The server dynamically allocates a sampling quota to each layer based on the average response capability of clients at each layer and the historical participation of each layer. The sampling quota determines the number of clients selected from that layer. S4. For each layer, the server selects clients that meet its sampling quota based on the degree of difference between the local data distribution of clients within the layer, so that the selected client set has high discreteness in data distribution. S5. The server merges the selected clients from each layer to form a set of clients participating in the current round of training, and sends global model parameters to the clients in this set for local training and update aggregation. S6. The server updates the latency status, data distribution information, and historical participation status based on the actual response of the client in this round of training, for use in scheduling decisions for subsequent rounds.

2. The method according to claim 1, characterized in that, The process of dividing the client into multiple layers as described in S2 includes: the server sorts the client latency values ​​from smallest to largest, and according to the preset number of layers or quantile threshold, the sorted client sequence is sequentially divided into different layers, so that the client with the smaller latency value is assigned to the front layer, and the client with the larger latency value is assigned to the back layer.

3. The method according to claim 1, characterized in that, The process of dynamically allocating a sampling quota for each layer as described in S3 includes: the server calculates the sampling score of each layer based on the average latency status value of each layer, the cumulative sampling quota allocated to each layer in historical rounds, and a preset fairness adjustment coefficient; then, the sampling score of each layer is normalized; and the normalized ratio is multiplied by the target total number of samples in this round to obtain the sampling quota of each layer.

4. The method according to claim 1, characterized in that, The process of selecting clients based on the degree of difference between local data distributions of clients within the layer, as described in S4, includes: the server constructs a representation vector for each client within the layer to characterize its local data distribution features, calculates the degree of difference between any two client representation vectors within the layer, constructs a data difference relationship graph with clients as nodes and the degree of difference as edge weights, and then uses a greedy algorithm to iteratively select clients from the data difference relationship graph, each time selecting the client with the largest average degree of difference with all clients in the current selected set to add, until the sampling quota of the layer is reached.

5. The method according to claim 1, characterized in that, The local training and update aggregation process described in S5 includes: the server only aggregates the set of clients that successfully return local model updates within a preset time window; for clients that fail to return successfully within the time window, the server records the abnormal events in this training and corrects the delay status of the corresponding clients in subsequent rounds based on the recorded abnormal events.

6. The method according to claim 1, characterized in that, The process of updating historical participation in S6 includes: the server maintaining the historical selection count for each client, and incrementing the historical selection count for clients selected in this round of training; the server also maintaining the historical usage count for each layer, and incrementing the historical usage count for layers allocated sampling quotas in this round of training. The historical selection count and layer usage count are used to apply fairness constraints when calculating the sampling quotas for each layer in subsequent rounds S3.

7. A federated training scheduling system based on hierarchical dynamic selection and diversity constraints, characterized in that, The system for implementing the method of any one of claims 1-6 comprises: The client state maintenance module is used to obtain the clients that are available in the current training round and maintain a latency state for each client that reflects its overall response capability. A client-side layering module is used to divide the client into multiple layers based on the latency status, wherein clients within the same layer have similar response capabilities; The dynamic quota allocation module is used to dynamically allocate a sampling quota to each layer based on the average response capability of clients at each layer and the historical participation of each layer. The sampling quota determines the number of clients selected from that layer. The layer diversity selection module is used to select clients that meet the sampling quota of each layer based on the degree of difference between the local data distribution of clients within the layer, so that the selected client set has high discreteness in data distribution. The training participation aggregation module is used to merge the selected clients from each layer to form a set of clients participating in the current round of training, and to send global model parameters to the clients in this set for local training and updating aggregation; The state update module is used to update the latency state, data distribution information, and historical participation status based on the actual response of the client in this round of training, for use in scheduling decisions for subsequent rounds.

8. A computer terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

Images