Wireless federal learning scheduling method based on multi-arm long machine optimization

Abstract

The invention relates to a wireless federated learning scheduling method based on multi-arm long machine optimization, a wireless federated learning model and a multi-arm long machine model, a wireless federated learning algorithm based on a multi-arm long machine can be given, and a super pull rod combination (or each action) is selected in each turn as an integral feedback. The algorithm describes the process of carrying out efficient wireless federal learning training under the assistance of the dobby machine provided by the invention.

Description

Technical Field

[0001] The present invention belongs to the technical field of balanced node scheduling, and in particular relates to a wireless federated learning scheduling method based on multi-armed slot machine optimization. Background Art

[0002] A wireless federated learning model based on a multi-armed bandit, the architecture is as follows Figure 1 As shown, the system consists of a local client, a wireless channel, and a central server (deployment scheduling framework). Edge nodes are IoT devices with computing and communication capabilities. This type of device usually uses limited computing power to train models on local non-independent and identically distributed data sets; the wireless channel serves as an interactive medium between the client and the server; the server is used to carry the multi-arm scheduling framework and aggregate parameters after verification. In the entire structure, the multi-armed slot machine scheduling framework is deployed on the server side, which is transparent to all edge nodes. It is used to count the training information of each round of nodes and use this information to make decisions to determine the node selection for the next round;

[0003] In fact, due to privacy issues or regulatory requirements (for example, private user data in certain regions may not be shared externally), the server cannot directly train models from user behavior data. Instead, user data is strictly stored on the client device or local server (never leaves) to better protect privacy. Therefore, the local view of the project is usually biased and cannot represent the overall distribution, and the global server can only access some aggregated information instead of individual data, which brings serious non-independent and identically distributed (Non-IID) problems. The quality and quantity of data owned by different edge nodes are extremely heterogeneous. The model parameters trained locally often reduce the accuracy of the global model after aggregation. The models of each edge node may have very different contributions to the global model. In the case of limited channels and a large number of edge nodes, the local update parameters sent by the end user to the edge server in the form of broadcast during the wireless federation training process are easily interfered by various types of noise, thereby reducing the effect of the entire federated learning, which brings huge communication load and channel fading risks; more seriously, when the node fails offline or the channel causes the model parameters to be unable to be transmitted to the central server in time, it causes a long wait, which also brings the problem of long convergence delay. How to help the central node make fair decisions on node selection and channel optimization with limited information to improve the training convergence speed and balance the model accuracy leads to the multi-armed bandit method in wireless federated learning. Summary of the invention

[0004] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a wireless federated learning scheduling method based on multi-armed bandit optimization, which uses a multi-armed bandit to help the central server make decisions while adapting to uncertainty and protecting fairness. The multi-armed bandit is maintained by the central server and runs on the edge server side. Using the multi-armed federated learning framework, the node evaluation algorithm is used to further enhance the training effect of higher-quality edge intelligent nodes, and through more rounds of selection, its data potential is further mined to enhance its contribution to the final model. Through the channel evaluation algorithm, the transmission application of reliable channels is amplified, and the access weight parameters of distorted channels are reduced, so that multiple parties in the Internet of Things environment can adjust the central server to schedule the entire training network without making additional architectural changes to the edge nodes, so as to better handle global tasks.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A wireless federated learning scheduling method based on multi-armed bandit optimization, characterized in that it includes the following steps:

[0007] Step 1: Construction of wireless federated learning environment with limited channel resources:

[0008] Set up the following system model: a wireless network consisting of a single-antenna AP and a large number of single-antenna clients, the client set is denoted as κ, where |κ| = K, and each client κ∈K has a local data set W k , the number of sample points is s k , there are k =|W k |, the edge node will perform local training based on the sample point. The purpose of FL training is to minimize the weighted global loss function, which is in the form of:

[0009]

[0010] in is the weight of client k, where And G k (x) is the local experience loss of the local client k, which is defined as follows:

[0011]

[0012] where w ks` Indicates that W kThe loss function of the relevant parameter vector x of the s-th sample point in []. Assume that there are a fixed number of spectrums available and they are equally divided into N orthogonal radio access channels. At most one radio channel is assigned to a client in each round, so there is no interference between clients. Due to the scarcity of spectrum resources, the number of available channels is much less than the number of candidate clients, that is, N < K. The training process of FL is an iterative process consisting of multiple rounds of communication. In each round t, the central server radio access point (AP) selects a subset S(t) ∈ κ of clients, and then assigns the weight vector x(t) of the global model to the selected clients. After receiving the global model weights, each selected client individually updates the global model by calculating the gradient of the local loss function based on its own private data, and then uploads the updated gradient to the AP for model aggregation, that is

[0013] x(t + 1) = x(t) - γv(t) (3)

[0014]

[0015] where the local update gradient is v(t);

[0016] Training delay: Since the central server radio access point (AP) has abundant computing resources compared with clients, the delay caused by global model aggregation can be ignored. According to the above analysis, the time cost of client k in each round t depends on three main components: distribution and download time, local computing time, and upload time, which are denoted as Then, the total time consumed by client k in the t-th round is

[0017]

[0018] where Γ max represents the maximum interval of each round of communication, which is used to avoid endless waiting caused by possible stragglers. The distribution time and upload time of client k in the t-th round depend on the size of the model parameters and the wireless channel quality between the AP and client k. However, since clients can perform multiple processing tasks simultaneously, the available computing resources on each client change over time; due to the heterogeneity of available computing capabilities, the local update times generated between clients are also different. Therefore, in the construction of the multi-armed scheduling model later, the AP needs to obtain the values of the three main components separately in each round, which can be obtained by responding to requests But needs to be obtained and calculated separately;

[0019] Step 2. Slot machine restrictive constraints:

[0020] (1) Availability matrix: During the FL process, some clients may be temporarily unavailable due to poor channel conditions or exhaustion of computing power. Define a binary variable α k (t), if client k is available in round t, then α k (t) = 1; otherwise α k (t) = 0, further define As the set of available clients in round t, where Ρ(κ) is the power set of κ; it is assumed that the set of available clients is independent and identically distributed. As time goes by and the corresponding distribution of available clients changes dynamically, is unknowable in advance. However, at the beginning of each round t, the central node A(t) will actively detect the availability of all nodes and channels. In the network under consideration, the number of clients is much larger than the number of available channels. Therefore, the AP must select a subset S(t) from the available clients, that is,

[0021]

[0022] Where |S(t)| represents the cardinality of S(t), and Q(A(t)) is the power set of A(t);

[0023] (2) Model quality matrix: The local dataset of any specific client cannot represent the overall distribution, and the sizes of local datasets of different clients are different because different clients have different activity characteristics. A joint averaging method is adopted, in which a part of clients participate in the FL process in each round to collaboratively train the shared model. However, the importance of local datasets of different clients is different. Local datasets with larger scale and distribution more similar to the global distribution play a more important role, and the corresponding clients should participate in more communication rounds. Therefore, the model quality matrix is ​​introduced: it tells each client how many communication rounds it should participate in. The model quality matrix can not only avoid the situation of abandoning slow but important clients due to the pursuit of low latency, but also allow important clients to participate in more rounds by setting a larger fairness factor. Define the binary variable b k (t) as an indicator, b k (t) = 1, indicating that customer k is selected for round t, otherwise b k (t) = 0. Then, the model quality matrix is ​​expressed as:

[0024]

[0025] in is the expectation operator, c k∈(0,1) is the minimum number of communication rounds required to select client k. Since the distribution of the local dataset is usually unknown, the size of the local dataset is used as a metric to roughly describe the importance and set fairness constraints. First, find the client And set in and The fairness constraint value for other clients is set to

[0026] These constraints Merge into a vector c = {c1, c2, ..., c k} + ,in{·} + represents the transposition operator. If there exists a strategy that can make a time series {S(t), t≥1} satisfy the constraints in formula (7), then the constraint vector C is called feasible, otherwise it is infeasible. Any strategy that satisfies ∑ k∈κ c k C > N is infeasible. In addition, let C represent the maximum feasible region containing all such feasible C.

[0027] In each round of communication, the AP cannot perform model aggregation until all selected clients have completed data upload, so the time consumption t of each round is determined by the slowest client, that is,

[0028]

[0029] Below, for simplicity, Written as We further consider ideal and non-ideal scenarios. In an ideal situation, all clients are available at any time, and the local data sets of the clients are independent and identically distributed. They are also balanced, so there is no need to consider availability constraints and fairness constraints. In addition, as the number of participating clients in each round increases, the convergence speed is greatly improved. Therefore, the constraint formula (6) can be simplified to

[0030] The goal is to minimize the time cost of the FL training process by finding a decision sequence {S(t), t≥1}, expressed as function ρ1:

[0031]

[0032] η(t)∈(0,1) (11)

[0033] Formula (9), formula (10) and formula (11) are combined to express the target problem constraint ρ1. In the above formula (10), T is a fixed large constant. is the normalized delay, η(t)=1 is an indicator with a value equal to 1; if the FL process does not converge, the value is 0. However, in non-ideal cases, the client may not be available during the learning process, and the local datasets of different clients are non-independent and identically distributed and unbalanced, so fairness constraints and availability constraints should be addressed. In the case where the datasets are not independent and identically distributed, the convergence speed is less dependent on the number of participating clients in each round |S(t)|. As mentioned earlier, the number of clients is much larger than the number of available channels in the considered network. Therefore, for simplicity, it is necessary to make full use of the available channels and select as many clients as possible in each round, that is,

[0034]

[0035] Then, the underlying problem with availability and fairness constraints is formulated as ρ2

[0036]

[0037] η(t)∈(0,1) (14)

[0038] Formula (7), formula (12), formula (13) and formula (14) are combined to express the objective function ρ2. Due to the lack of prior information, it is not easy to solve the scheduling problem (3.13). The AP does not know the dynamics of the client's computing power, nor does it know the wireless channel status CSI between the AP and the client, so it is difficult to make the best decision. In order to solve this problem, some important information is obtained from the feedback of previous decisions, namely the statistical information of the time consumption of each customer. In addition, fairness constraints and availability constraints increase the difficulty of solving problem ρ2. Let's first solve formula ρ1, and then further solve formula ρ2;

[0039] Step 3: Multi-armed slot machine federation node allocation scheduling algorithm framework:

[0040] The MAB problem is one of the sequential decision making problems where the player should make a decision at each time period, deciding which arms of the multi-armed bandit to pull (the levers), where arm pulling is called an action, and a reward (reward) is observed when an action is taken. The goal of the MAB problem is to make sequential decisions to maximize the total reward obtained in a sequence of actions. Due to the limited information available, the player must strike a balance between exploiting actions that performed well in the past and exploring actions that may return higher rewards in the future.

[0041] The scheduling problem ρ1 can be modeled as a MAB problem, where the AP and the client can be regarded as players and slot machine levers (i.e., arms), respectively. The selected subset of arms S(t) is called a super arm (super lever group), and its corresponding normalized reward is Therefore, the goal of problem ρ1 can be interpreted as finding a temporal sequence of actions according to a specific policy to maximize the cumulative reward, i.e.

[0042]

[0043] For the scheduling problem formula (12), regret is used to evaluate the action selection strategy, and regret is defined as the difference between the expected reward of the optimal action and the reward obtained by a given strategy. Among them, minimizing regret is equivalent to maximizing the total reward. represents the normalized reward for pulling lever k The optimal super tie rod set is

[0044]

[0045] Assume that Delta(T) = 1 and Delta(T+1) = 0, which means that FL training converges at the end of T rounds. Then, the cumulative regret for a given policy π1 is

[0046]

[0047] In the above formula, μ(S) = min k∈S μ k is the expected reward of the super lever group S. Then, the problem ρ1 can be restated as the following formula ρ3:

[0048]

[0049] Formula (12) and formula (18) are combined to express the constrained objective function ρ3.

[0050] Step 4: Multi-armed slot machine UCB algorithm combined with wireless federated learning:

[0051] As shown in WFMA, the algorithm proposed for the objective function ρ3 is based on the UCB strategy, that is, the basic principle of the multi-armed bandit UCB algorithm, and is further analyzed and improved by combining the statistical information in wireless federated learning. The key idea behind this algorithm is to observe the feedback of each lever. The reward changes of the same lever can be observed in different actions, and the reward information about a certain arm can be collected and utilized from the operation of different actions to help the system make better decisions in the future. The following defines two vectors and y k (t) is the kth element of y(t), which represents the sample mean of the reward values ​​of the kth arm observed up to the current round t, and is updated according to the following rules:

[0052]

[0053] where zk (t) is the kth element of z(t), representing the call of the kth lever at the end of round t, and is updated as follows:

[0054]

[0055] When t = 0, z and y are both initialized to 0, that is, z(0) = y(0) = 0. In the multi-arm scheduling algorithm, The round is initialized to ensure that each lever is selected at least once. After the initialization is completed, the algorithm enters the main loop and selects a super arm in each round (t represents the current round) according to the following formula:

[0056]

[0057] where y k (t-1) and Corresponding to development and exploration, respectively. Then, the size of P(κ) is exponential in K. Therefore, the complexity of the combinatorial problem formula (21) may be very high, and some effective algorithms need to be further developed. Due to the special structure of linear rewards, where the rewards of different arms in formula (21) are independent of each other, the best super arm S(t) can be found by iteratively selecting the best single arm. More specifically, the customers can be sorted in descending order of reward and the top N customers can be selected.

[0058] For non-ideal scenarios (Non-IID data): availability constraints and fairness constraints need to be further considered. Based on formula ρ1, ρ2 is reformulated as a multi-armed bandit (MAB) problem, but due to the model quality matrix, the above super lever set formula (21) cannot be directly applied to problem ρ2. The availability constraint can be solved by checking the availability of the client before selecting an operation in the algorithm, but other methods must be resorted to to satisfy the fairness constraint. Finally, the virtual queue technology is applied to deal with the fairness constraint.

[0059] The beneficial effects of the present invention are:

[0060] A balanced node scheduling problem in wireless federated learning aims to minimize the time consumption of the entire wireless federated training and maximize the model training accuracy; time consumption refers to the sum of transmission time and local computing time in non-ideal scenarios; and try to avoid the large amount of noise generated by unreliable channels interfering with the training parameters of some nodes, which ultimately leads to poor accuracy and difficulty in convergence of the overall federated learning model. Then, a wireless federated learning scheduling framework based on multi-armed bandit (MAB) is designed and improved to dynamically and balancedly evaluate and select online clients in FL training. Before the decision-making process of node scheduling, the model does not need to know the state of the wireless channel (including channel fading, channel noise, bandwidth limitation, link reliability, etc.) and the dynamic usage of client computing resources (including client data distribution, client data quality, client data volume, node computing power, node data sending power, etc.) in advance. For non-ideal scenarios with non-independent and identically distributed, a multi-arm node scheduling algorithm based on the upper confidence limit (UCB) strategy, namely the UCB-E algorithm, is proposed. The algorithm strikes a balance between taking advantage of actions that performed well in the past and actions that perform well at present. Explore actions that may bring higher returns in the future. The upper bound of regret is provided, and it is shown that the expected performance regret grows logarithmically in the communication rounds. Considering the imbalanced characteristics of the local dataset and the dynamic availability of the client, a fairness constraint is introduced to ensure that each client can participate in a certain proportion of the communication rounds during training, strengthening the "exploration" characteristics of the multi-armed bandit. The UCB-E strategy is further improved by adding a scheduling algorithm with parameter approximate fitting technology, namely the UCB-E algorithm. The upper bound of regret is also derived, and it is shown that the expected performance regret of the proposed UCB-E algorithm can grow sublinearly in the communication rounds under certain conditions. The convergence performance in ideal and non-ideal scenarios is analyzed. The study found that in ideal scenarios, increasing the number of participating clients per round can improve the convergence speed, but in non-ideal scenarios, the convergence performance is less dependent on the number of participating clients per round. The simulation results verify the efficiency of the proposed UCB-E algorithm. In addition, through the ablation experiment of the hyperparameter beta, the optimal parameter configuration that can make a trade-off between performance indicators, regret changes, and node delay is found. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is the architecture diagram of the wireless federated learning model based on the multi-armed bandit;

[0062] Figure 2 This is a comparison chart of the multi-arm scheduling algorithm in this article, the R-Robin algorithm, and the Random algorithm on the same dataset, including (a) test accuracy on the MNIST dataset, and (b) training accuracy comparison on the MNIST dataset.

[0063] Figure 3 The following is a comparison chart of the multi-arm scheduling algorithm, R-Robin algorithm and Random algorithm on the same data set, where (a) is the test accuracy on the MNIST data set, and (b) is the training accuracy comparison on the MNIST data set. DETAILED DESCRIPTION

[0064] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0065] Example:

[0066] Experimental setup: In this section, the following experimental setup is used to perform numerical analysis on the optimization model in the previous section of this chapter through Pytorch simulation. Unless otherwise stated, the system setup is summarized as follows: We consider a network with a single-antenna AP located at the center of the network (connected to the central server), a disk length of 500m, and clients randomly distributed within the coverage of the AP. The channel gain of both the uplink and downlink links is composed of full-scale fading and large-scale fading, where the full-scale fading is set to a Rayleigh distribution with uniform variance, and the large-scale fading is generated by the path-loss model: PL[dB]=128.1+37.6log 10 (d), where d represents the distance in km. Noise power σ 2 The bandwidth of each orthogonal channel is 15KHz. The uplink and downlink transmission power are both 23dBm, that is, pUk = pDk = 23dBm. The computing power φ of each client k k (t) is evenly distributed in in and Here, I assume that and In addition, for simplicity, I assume that m0 = m k =5×10 3 bits, maximum interval Γ max = 5s. I use multinomial logistic regression to study the classification of images of handwritten digits 0-9 in the famous MNIST dataset. The training set and test set consist of 60,000 and 10,000 samples respectively. The SGD method is used for each edge node, with a mini-batch size of 2 and a learning rate of 0.001.

[0067] Experimental verification on real data sets:

[0068] This section will verify the performance and effect of the proposed wireless federated learning algorithm based on multi-armed bandit. It mainly adds controllable noise and data volume restrictions in the process of federated learning, and compares the three federated learning channel scheduling algorithms without and with the method in this paper after availability restrictions. The experimental results show that when the channel of some nodes is severely attenuated, the multi-armed bandit in the algorithm proposed in this chapter scores the node + channel in total based on the feedback index of this round and the historical evaluation index of each node at the center. The test accuracy under different data sets has improved, which proves the effectiveness of the proposed multi-armed bandit wireless federated learning algorithm in resisting channel interference and optimizing node problems.

[0069] In the experimental setting of this section, the performance of the attacked federated learning after adding the multi-armed bandit proposed in this paper is tested through the classification task of supervised learning. The classification task uses a multinomial logistic regression model with a cross entropy error loss function on a real federated dataset (MNIST

[62] , FEMNIST

[63] ). The experimental environment is a host equipped with an AMD Ryzen93900X 12-Core processor and 32GB of memory. Its operating system is Windows 10. The Python version used for federated learning model training is 3.6.10, and the PyTorch version is 1.9.0. In the experiment, the total number of devices with training capabilities is set to 8. Since the federated learning algorithm is allowed to use the SGD algorithm at the edge node, the local data set is randomly sampled. The number of user devices participating in each round is set to 10, and the number of local and global rounds is 15 and 150, respectively. For the data set, in order to show that federated learning can handle non-independent and identically distributed data, all data sets have different sample sizes. In the MNIST data set, each user contains three constant labels, while the FEMNIST data set is obtained by scaling MNIST.

[64] All data sets are randomly split, with the training set accounting for 75% and the test set accounting for 25%.

[0070] In this experiment, we compared the wireless federated learning algorithm FedAvg optimized by a multi-armed bandit with two other federated learning algorithms optimized by node scheduling under different circumstances: (1) R-Robin algorithm: The system maintains a list of nodes participating in federated learning and selects nodes in a pre-set order. In each communication cycle, the currently selected node is allowed to exchange information with the central server according to the preset time slice strategy or number of times, such as sending local model updates or receiving new versions of the global model. In a complete scheduling cycle, when a node completes its task, the scheduler will move to the next node in sequence to ensure that all nodes have the opportunity to participate in the federated learning process within a certain period of time. The main advantages of using the Round Robin scheduling algorithm are its simplicity and fairness. It ensures that all available nodes can participate in the training process in turn at a fixed frequency, avoiding the long-term neglect of certain nodes. It is a typical fair scheduling algorithm. (2) Random scheduling algorithm: This algorithm is also applicable to unreliable and resource-constrained wireless networks. Unlike the Round-Robin scheduling algorithm, it does not select nodes in a fixed order, but adopts a random sampling method. Each node selection process is independent and identically distributed, that is, each node has the same probability of being selected, and does not set different selection probabilities based on node characteristics such as computing power and network conditions. The main advantage is its flexibility and adaptability to system heterogeneity. It can avoid potential problems caused by fixed polling to a certain extent. For example, the data quality or network conditions of some nodes may be better at certain times, and random scheduling may more efficiently utilize these advantages. At the same time, randomness also helps to enhance the robustness of the system and the generalization performance of the model, which is the target of most comparative experiments in scheduling algorithm research.

[0071] Evaluation indicators of wireless federated learning usually include training time and energy consumption, global model accuracy, and global model loss. In wireless federated learning, the computing resources and network bandwidth of different participants may be different. The time and energy consumption of model training is an important indicator. Generally, the lower the training time and energy consumption, the higher the model training efficiency. The training time and energy consumption of the wireless federated learning model proposed in this paper have been evaluated in the numerical experimental analysis in the previous section. The accuracy of the global model is one of the most basic indicators for evaluating the performance of wireless federated learning. Usually, the accuracy refers to the classification accuracy or prediction accuracy on the test data set. The higher the accuracy, the better the performance of the model. Global loss is another basic indicator for evaluating the performance of the federated learning model. It is the weighted average of the loss functions of the local models of all participants. It can usually be used to evaluate the overall performance of the federated learning model. It integrates the data and models of all participants and can reflect the overall fitting ability of the federated learning model. If the global loss is smaller, it means that the overall fitting ability of the federated learning model is better, and the data and models of each participant can work together effectively. Therefore, in the experiments in this section, global model accuracy and global model loss are used as evaluation indicators, such as Figure 2 described.

[0072] The wireless federated learning algorithm UCB-E, which integrates the multi-arm scheduling used in this paper, is compared with the wireless federated learning framework that combines R-Robin and the classic Random scheduling algorithm. The comparative experiment is conducted on the MNIST dataset. The training set results are shown in the figure below. Figure 2 As shown, it can be seen that under normal circumstances, the UCB-E algorithm has a higher accuracy advantage than the R-Robin and Random algorithms. This is the experimental result on the training set. The above figures show the changes in the training loss and training set accuracy of the four strategies at different numbers of iterations. The colors from left to right and top to bottom represent the following: the green dotted line corresponds to the baseline method, the purple solid line represents the UCB-E algorithm of this article, the blue dotted line represents the R-Robin algorithm, and the orange dotted line represents the Random algorithm. Figure 3 shown.

[0073] First, let’s analyze Figure (a): Based on the given data set, I drew a line graph of the training loss versus the number of iterations. In this graph, the horizontal axis represents the cycle count after each parameter update, and the vertical axis records the error calculated by the network in the corresponding state.

[0074] First, observe the overall shape of the whole set of images. You can see that all the curves are constantly decreasing until they tend to be flat, which means that our neural network is gradually approaching the optimal solution space. Secondly, note that the difference between the branches is not very large, especially the evolution trajectories in the first half of the cycle almost overlap to form a straight line structure, because the loss drops rapidly in the early stage. The baseline represents the change in loss value in an ideal state, which is the expected reference. The loss value of the UCB-E algorithm in this paper drops very quickly, and at 50 rounds, it is significantly different from other comparison algorithms. Its loss curve trend shows more stability, less oscillation, and smaller amplitude; its loss difference with the baseline maintains a stable range until convergence, which brings more stable and faster model performance than other methods. The R-Robin algorithm has occasional declines and a certain degree of oscillation; its loss value is generally similar to the random scheduling algorithm, sometimes close to the baseline and sometimes far away, but it has stronger stability than the random scheduling algorithm. For the random algorithm, observing its curve trend, the oscillation amplitude is the largest. This may be due to random uncertainty, which leads to the selection of multiple strong fading channels at one time, resulting in greater losses. Its stability is significantly lower than other algorithms, its scheduling ability is insufficient, and the uncertainty of training effects is also greater. Based on the comprehensive considerations of the above aspects, I believe that the UCB-E algorithm has obvious advantages in terms of rapid and stable loss reduction. On the one hand, it has a low starting point height and continues to decline at a faster rate. On the other hand, and most importantly, it is closest to the ideal loss decline trend curve.

[0075] Figure (b) is analyzed as follows: This figure is a line graph of the training accuracy changing with the number of iterations. In this figure, the horizontal axis represents the loop count value after each parameter update, and the vertical axis records the accuracy calculated by the network in the corresponding state. Baseline method: As a reference benchmark for the ideal situation, its performance is extremely stable at the beginning, and gradually improves with the increase of the number of iterations, and the curve is always smooth. Next is the UCB-E algorithm for multi-arm scheduling, which maintains a high accuracy and steadily increases throughout the process. Especially in the early stage, it performs well, indicating that the algorithm can quickly adapt to the current channel environment and make effective decisions. However, it should be noted that although the overall trend is upward, there are fluctuations at some points, which is due to normal parameter oscillations caused by channel noise interference and node unavailability. In contrast, R-Robin's performance is relatively poor, especially in the early stage. However, over time, its accuracy rate increases significantly and is generally better than the random scheduling algorithm. This trend shows that the algorithm has extremely strong fairness, uniformly absorbs noise interference, and has a certain degree of robustness. At the same time, we can also see that although the overall trend is positive, there are still certain fluctuations in accuracy, reflecting the insufficient restrictions on its scheduling capabilities. Finally, let’s look at the random scheduling method. As expected, it cannot guarantee stable performance at any time and anywhere. Since nodes are selected randomly without evaluation, the channel fading problem is more prominent. It is possible to select multiple channels in one round that are severely interfered by noise, resulting in severe data distortion; therefore, its curve fluctuates significantly, and it is always unable to stabilize and get rid of the trough state. Therefore, it can be said that this is a relatively simple, but also the most unreliable node scheduling method.

[0076] The comparative experiment was conducted on the MNIST dataset, and the test set results are as follows: Figure 3 As shown in the figure, they together constitute the comparison and analysis of the four dimensions of this node scheduling utility comparison experiment. Figure 3 As shown, after Figure 2 The comparison of the results shows that the trend is basically the same. Therefore, the analysis conclusion is the same as above: By comparing the change rules of the four curves, it can be concluded that for this problem, the use of UCB-E, a reinforcement learning technology based on context information, can more effectively solve the needs of node scheduling tasks in wireless federated learning than simply relying on probability statistics or blind random selection attempts.

[0077] In summary, combined with the previous analysis in this article, this shows that in the presence of certain and unequal noise, strong interference has been caused to some channels, resulting in the distortion of the parameters transmitted on these channels. With the help of the multi-armed slot machine scheduling framework, the evaluation of the feedback level of different node parameters is quickly completed within a limited number of rounds, the situation of all nodes is explored, and in subsequent steps, the known node information is concentrated on using the potential of reliable channels to ensure the high quality of the global model update parameters in subsequent rounds. This proves that the introduction of the multi-armed slot machine scheduling algorithm proposed in this article to schedule various node indicators (such as parameter verification, channel quality assessment, response speed feedback) to prevent channel fading and node instability model problems is very effective for improving the accuracy and fast convergence of the wireless federated learning algorithm.

Claims

1. A wireless federated learning scheduling method based on multi-armed bandit optimization, It is characterized in that The following steps are included: Set up the following system model: a wireless network consisting of a single-antenna AP and a large number of single-antenna clients, the client set is denoted as κ, where |κ|=K, and each client κ∈K has a local data set W k , the number of sample points is s k , there are k =|W k |, the edge node will perform local training based on the sample point. The purpose of FL training is to minimize the weighted global loss function, which is in the form of: in is the weight of client k, where And G k (x) is the local experience loss of the local client k, which is defined as follows: where, w ks` represents the loss function of the relevant parameter vector x of the s-th sample point in W k ; assuming that there are a fixed number of spectrums available and are equally divided into N orthogonal radio access channels, at most one radio channel is allocated to a client in each round, so there is no interference between clients. Due to the scarcity of spectrum resources, the number of available channels is much less than the number of candidate clients, that is, N < K. The training process of FL is an iterative process consisting of multiple rounds of communication. In each round t, the central server radio access point (AP) selects a subset S(t) ∈ κ of clients, and then allocates the weight vector x(t) of the global model to the selected clients. After receiving the global model weights, each selected client individually updates the global model by calculating the gradient of the local loss function based on its own private data, and then uploads the updated gradient to the AP for model aggregation, that is x(t+1)=x(t)-γv(t) (3) The local update gradient is v(t); Training delay: Since the central server wireless access point (AP) has abundant computing resources compared with the client, the delay caused by global model aggregation can be ignored. According to the above analysis, the time cost of each round t of client k depends on three main components: distribution download time, local computing time and upload time, which are recorded as Then, the total time consumed by client k in round t is: Among them, Γ max represents the maximum interval of each round of communication, which is used to avoid endless waiting caused by possible stragglers; the distribution time of client k in round t and upload time depends on the size of the model parameters and the quality of the wireless channel between the AP and the client k; however, since the client can perform multiple processing tasks simultaneously, the available computing resources on each client vary over time; due to the heterogeneity of the available computing power, the local update time between clients Therefore, in the subsequent multi-arm scheduling model construction, the AP needs to obtain the values ​​of the three main components separately in each round, which can be obtained by responding to the request but Need to be obtained and calculated separately; Step 2: Slot machine restrictive constraints: (1) Availability matrix: During the FL process, some clients may be temporarily unavailable due to poor channel conditions or exhaustion of computing power. Define a binary variable α k (t), if client k is available in round t, then α k (t) = 1; otherwise α k (t) = 0, further define is the set of available clients in round t, where P(κ) is the power set of κ; it is assumed that the set of available clients is independent and identically distributed; as time goes by and the corresponding distribution of available clients changes dynamically, is unknowable in advance. However, at the beginning of each round t, the central node A(t) will actively detect the availability of all nodes and channels. In the network under consideration, the number of clients is much larger than the number of available channels. Therefore, the AP must select a subset S(t) from the available clients, that is, Where |S(t)| represents the cardinality of S(t), and Q(A(t)) is the power set of A(t); (2) Model quality matrix: The model quality matrix can not only avoid abandoning slow but important clients due to the pursuit of low latency, but also allow important clients to participate in more rounds by setting a larger fairness factor. Define the binary variable b k (t) as an indicator, b k (t) = 1, indicating that customer k is selected for round t, otherwise b k (t) = 0; then, the model quality matrix is ​​expressed as: in is the expectation operator, c k ∈(0,1) is the minimum number of communication rounds required to select client k; since the distribution of the local dataset is usually unknown, the size of the local dataset is used as a metric to roughly describe the importance and set fairness constraints; find the client And set in and The fairness constraint value for other clients is set to These constraints Merge into a vector c = {c 1 ,c 2 ,......,c k } + ,in{·} + represents the transposition operator; if there exists a strategy that can make a time series {S(t), t≥1} satisfy the constraints in formula (7), then the constraint vector C is called feasible, otherwise it is infeasible; any strategy that satisfies ∑ k∈κ c k > N c is infeasible; in addition, let C represent the maximum feasible region containing all such feasible C; In each round of communication, the AP cannot perform model aggregation until all selected clients have completed data upload, so the time consumption t of each round is determined by the slowest client, that is, Will Written as We further consider ideal and non-ideal scenarios. In an ideal situation, all clients are available at any time, and the local data sets of the clients are independent and identically distributed. They are also balanced, so there is no need to consider availability constraints and fairness constraints. In addition, as the number of participating clients in each round increases, the convergence speed is greatly improved. Therefore, the constraint formula (6) can be simplified to: By finding a decision sequence {S(t), t≥1}, the time cost of the FL training process is minimized, represented by the function ρ 1 : η(t)∈(0,1) (11) Formula (9), Formula (10) and Formula (11) are combined to express the target problem constraint ρ 1 In the above formula (10), T is a fixed large constant. is the normalized delay, η(t)=1 is an indicator with a value equal to 1; if the FL process does not converge, the value is 0; however, in non-ideal cases, the client may not be available during the learning process, and the local datasets of different clients are non-IID and unbalanced, so fairness constraints and availability constraints should be addressed; in the case of non-IID datasets, the convergence speed is less dependent on the number of participating clients in each round |S(t)|; as mentioned earlier, the number of clients is much larger than the number of available channels in the considered network; therefore, for simplicity, it is necessary to make full use of the available channels and select as many clients as possible in each round, i.e. Then, the underlying problem with availability and fairness constraints is formulated as ρ 2 η(t)∈(0,1) (14) Formula (7), formula (12), formula (13) and formula (14) are combined to express the objective function ρ 2 , due to the lack of prior information, it is not easy to solve the scheduling problem (3.13); AP does not know the dynamics of the client's computing power, nor does it know the wireless channel status CSI between AP and the client, so it is difficult to make the best decision; to solve this problem, some important information is learned from the feedback of previous decisions, namely the statistical information of the time consumption of each client; in addition, fairness constraints and availability constraints increase the solution to the problem ρ 2 The difficulty of this method is: 1 Then further solve the formula ρ 2 ; Step 3: Multi-armed slot machine federation node allocation scheduling algorithm framework: Scheduling Problem 1 It can be modeled as a MAB problem, where the AP and the client can be regarded as the player and the slot machine lever (i.e., the arm) respectively; the selected subset armsS(t) is called the super arm (super lever group), and its corresponding normalized reward is Therefore, the problem ρ 1 The goal of can be interpreted as finding a temporal sequence of actions according to a particular strategy to maximize the cumulative reward, i.e. For the scheduling problem formula (12), regret is used to evaluate the action selection strategy. Regret is defined as the difference between the expected reward of the optimal action and the reward obtained by a given strategy. Among them, minimizing regret is equivalent to maximizing the total reward. represents the normalized reward for pulling lever k The optimal super tie rod set is Assume that Delta(T) = 1 and Delta(T+1) = 0, which means that FL training converges at the end of T rounds; then, given the policy π 1 The accumulated regret is In the above formula, μ(S) = min k∈S μ k is the expected reward of the super lever group S; then, the problem ρ 1 It can be restated as 3 : Formula (12) and formula (18) are combined to express the constrained objective function ρ 3 ; Step 4: Multi-armed slot machine UCB algorithm combined with wireless federated learning: As shown in WFMA, for the objective function ρ 3 The proposed algorithm is based on the UCB strategy and defines two vectors and y k (t) is the kth element of y(t), which represents the sample mean of the reward values ​​of the kth arm observed up to the current round t, and is updated according to the following rules: where z k (t) is the kth element of z(t), representing the call of the kth lever at the end of round t, and is updated as follows: When t = 0, z and y are both initialized to 0, that is, z(0) = y(0) = 0; in the multi-arm scheduling algorithm, use The round is initialized to ensure that each lever is selected at least once; after the initialization is completed, the algorithm enters the main loop and selects a super arm in each round (t represents the current round) according to the following formula: where y k (t-1) and Corresponding to development and exploration respectively; then, the size of Ρ(κ) is exponential of K; therefore, the complexity of the combinatorial problem formula (21) may be very high, and some effective algorithms need to be further developed; due to the special structure of linear rewards, where the rewards of different arms in formula (21) are independent of each other, the best super arm S(t) is found by iteratively selecting the best single arm; more specifically, the customers can be sorted in descending order of rewards, and the top N customers can be selected; For non-ideal scenarios (Non-IID data): availability constraints and fairness constraints need to be further considered; in formula ρ 1 Based on 2 Re-formulated as a multi-armed bandit (MAB) problem, but due to the model mass matrix, the above super-tie set formula (21) cannot be directly applied to the problem ρ 2 ; Availability constraints can be addressed by checking the availability of the client before selecting an operation in the algorithm, and applying virtual queue techniques to handle fairness constraints.

Images