Layered federal learning method with high generalization ability

Abstract

The invention discloses a hierarchical federal learning method with high generalization ability, and relates to the technical field of electric digital data processing. The hierarchical federal learning method with the high generalization ability comprises the steps of local model training judgment, Q value acquisition and global model generation. According to the method, the training sample data provided by the client within the preset time period is acquired, whether the client participates in local model training is judged according to the training sample data, and meanwhile, the loss expected value in the local model training process is acquired and the edge server is selected; inputting training sample data participating in local model training into the DQN network to obtain a predicted Q value and a target Q value, and finally uploading the trained local model to the selected edge server to generate a global model, so as to improve the balance between the overhead of the hierarchical federated learning framework and the model accuracy. The problem that in the prior art, balance between layered federated learning framework overhead and model accuracy is not fully considered is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a hierarchical federated learning method with high generalization capability. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the continuous breakthroughs in deep learning and reinforcement learning, machine learning models are increasingly capable of solving complex problems. However, in traditional federated learning, whether it is synchronous update and its variants or asynchronous update, a large number of model parameters need to be exchanged in multiple updates. In order to overcome these challenges, researchers proposed the concepts of federated learning (FL) and hierarchical learning (HL), and combined with the advantages of DQN (Deep Q-Network) in deep reinforcement learning (DRL), they constructed a new learning framework, namely a hierarchical federated learning framework based on deep reinforcement learning and with generalization ability, which consists of 1 cloud, L edge servers, N clients, and N reinforcement learning networks.

[0003] Existing technologies divide training tasks into multiple subtasks through the perception capabilities of deep neural networks and the decision-making capabilities of deep reinforcement learning, which are executed by corresponding low-level strategies. Incentive mechanisms are then designed through model aggregation algorithms, while the federated learning process of participants is monitored in real time. Finally, the federated learning process is coordinated and interacted according to high-level strategies to achieve intelligent decision-making in complex environments.

[0004] For example, the invention patent announcement with announcement number: CN112668877B announces a method and system for allocating resource information of things that combines federated learning and reinforcement learning, including: the management and allocation of things to be allocated are uniformly allocated by the manager, and the number of things to be allocated cannot satisfy all individuals at the same time; the manager designs a deep reinforcement learning model and distributes it to each individual, and the individual uses the deep reinforcement learning model to select the things to be allocated; the manager generates a new global model through federated learning.

[0005] For example, the client selection and personalized privacy protection method in asynchronous federated edge learning disclosed in the patent application with publication number: CN117252253A includes: using a private data set to locally train and locally update the client's local learning model; using an asynchronous aggregation scheme to generate an outdated model for clients that did not participate in the aggregation in time; when t=0, the edge server initializes global parameters and an outdated model staleness list, broadcasts the global parameters to all clients for initial synchronization, and notifies the client to start local training; after receiving the global model, the client uses the SGD algorithm to perform local updates to obtain local model parameters; when the client completes the above calculation process locally, it is immediately uploaded to the edge server via a wireless link.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems: In existing technologies, incentive mechanism design mainly focuses on the heterogeneity of participants. This single-dimensional focus is often insufficient to fully address the challenges of efficiency and stability in federated learning in the long term. There is also a problem of insufficient consideration of the balance between the overhead of the hierarchical federated learning framework and the accuracy of the model. Summary of the invention

[0007] The embodiments of the present application solve the problem of insufficient consideration of the balance between the overhead of the hierarchical federated learning framework and the model accuracy in the prior art by providing a hierarchical federated learning method with high generalization capability, thereby achieving an improvement in the balance between the overhead of the hierarchical federated learning framework and the model accuracy.

[0008] The embodiment of the present application provides a hierarchical federated learning method with high generalization ability, comprising the following steps: S1, obtaining training sample data provided by a client within a preset time period, judging whether the client participates in local model training according to the training sample data, and if so, executing S2, wherein the training sample data is stored in a federated learning environment, wherein the federated learning environment includes a state space and an action space, and the local model is trained by the training sample data in the federated learning environment; S2, inputting the training sample data participating in the local model training into a DQN network to obtain a predicted Q value and a target Q value, and at the same time training the Q network according to the built-in parameters of the DQN network to obtain a balanced state value, wherein the predicted Q value is used In order to predict the output value of the local model after training in the current state, the target Q value is used to reflect the expected value of the local model after training in a given state; S3, obtain the sample pairs of the client according to the training sample data and obtain the expected value of the loss according to the obtained sample pairs, and select the edge server according to the obtained expected value of the loss, upload the trained local model in the DQN network to the selected edge server for model aggregation to generate an edge model, and upload the generated edge model to the cloud server to generate a global model, the edge server has generalization capability, the edge model is used to process the training sample data in real time and improve scheduling decisions, and the global model is used to evaluate and improve the local model and edge model according to the scheduling decision.

[0009] Furthermore, the specific process of judging whether the client participates in the local model training based on the training sample data is as follows: judging whether the training sample data is equal to 0: if the training sample data is equal to 0, it indicates that the corresponding client does not participate in this round of local model training, and at the same time, the changes in the training sample data within a preset time period are monitored in real time; if the training sample data is not equal to 0, it indicates that the corresponding client participates in this round of local model training, and at the same time, the corresponding training sample data is input into a preset database for storage through a multi-layer perceptron, and the multi-layer perceptron is used to convert the training sample data provided by the client that is not equal to 0 into a processable state for Q-learning.

[0010] Furthermore, the specific steps of obtaining the expected value of loss based on the acquired sample pairs are: obtaining a loss function based on the predicted value and sample label value in the local model training process, the predicted value is a function corresponding to the local model parameters, and the loss function is used to quantify the degree of difference between the predicted value and the sample label value; obtaining the loss value of the sample pair corresponding to the client according to the acquired loss function, and obtaining the expected value of loss in combination with the sampling result corresponding to the loss value, the loss value is used to reflect the prediction accuracy of the local model on a single sample pair, the sampling result is the sample pair selected by the client, and the expected value of loss is used to reflect the average performance of the local model on the sample pairs of the client.

[0011] Furthermore, the training sample data participating in the local model training is input into the DQN network to obtain the predicted Q value and the target Q value, and then the Q network is trained according to the built-in parameters of the DQN network to obtain the equilibrium state value; the built-in parameters include the Q network parameters and the Target network update frequency; the Q network parameters include the learning rate, the decay rate, and the discount factor; the equilibrium state value is the deviation Q value when the predicted Q value and the target Q value of the Q network reach an equilibrium state during the training process; the equilibrium state is the stable state corresponding to the deviation Q value equal to the reference deviation Q value; the deviation Q value is the difference between the target Q value and the predicted Q value; the reference deviation Q value is represented by the sum and average of the minimum values ​​that are not 0 in the historical deviation Q values ​​within the historical time period in the preset database.

[0012] Furthermore, the specific process of training the Q network according to the built-in parameters includes: step 1, taking the training sample data participating in the local model training as input, and obtaining the training limit data for training the Q network, wherein the training limit data includes the current state, the given state and the action task; step 2, inputting the current state into the Q network, obtaining the predicted Q value of the action task corresponding to the current state through the forward propagation process of the Q network, and obtaining the target Q value of the action task corresponding to the given state through the Target network; step 3, inputting the Q network parameters into the Target network according to the update frequency of the Target network through the backward propagation process of the Q network to ensure the stability of the Q network during the training process; step 4, judging whether the deviation Q value is equal to the reference deviation Q value, if so, completing the Q network training, otherwise returning to step 1, and stopping the Q network training after the difference between the predicted Q value and the target Q value is equal to the reference deviation Q value.

[0013] Furthermore, the specific limiting expression of the deviation Q value is: ; ; ; Where t is the number of the current time step, , T is the total number of current time steps, represents the deviation Q value of the DQN network at the current time step t, represents the target Q value of the Target network at the current time step t, represents the predicted Q value of the Q network at the current time step t, represents the state space of the DQN network at the current time step t, represents the state value selected by the client in the state space, represents the action space of the DQN network at the current time step t, represents the action chosen by the client in the state space, The action target value representing the action selected by the client, represents the future coefficient, represents the cumulative discount from the current time step t to the future time step k, represents the instantaneous reward of the local model during the training process at the current time step t, represents the discount factor, Represents the scheduling decision of the action task into the future state space corresponding to the current time step t in the federated learning environment, It represents the maximum target Q value of all actions performed by the Target network in the future state space.

[0014] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By obtaining the training sample data provided by the client within a preset time period, judging whether the client participates in the local model training according to the training sample data, obtaining the expected value of the loss in the local model training process and selecting the edge server, and then inputting the training sample data participating in the local model training into the DQN network to obtain the predicted Q value and the target Q value, and finally uploading the trained local model to the selected edge server to generate a global model, thereby achieving an improvement in the balance between the accuracy of the DQN network and the local training model, and then achieving an improvement in the balance between the overhead of the hierarchical federated learning framework and the model accuracy, effectively solving the problem of insufficient consideration of the balance between the overhead of the hierarchical federated learning framework and the model accuracy in the prior art; 2. By judging whether the training sample data is equal to 0: If the training sample data is equal to 0, the changes of the training sample data within the preset time period are monitored in real time. If the training sample data is not equal to 0, the corresponding training sample data is input into the preset database through the multi-layer perceptron for storage, thereby achieving more accurate judgment of the training sample data, and further improving the judgment accuracy of the local model training client; 3. The loss function is obtained through the predicted value and sample label value in the local model training process, and then the loss value of the corresponding sample pair of the client is obtained according to the obtained loss function. Finally, the expected value of the loss is obtained by combining the sampling result corresponding to the loss value, thereby achieving more accurate acquisition of the loss value, and then achieving improved accuracy and reliability in obtaining the expected value of the loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a hierarchical federated learning method with high generalization capability provided in an embodiment of the present application; Figure 2 A flowchart of hierarchical federated learning provided for an embodiment of the present application; Figure 3 A schematic diagram of a deep reinforcement learning network provided in an embodiment of the present application; Figure 4 A schematic diagram of overlapping edge base station ranges provided in an embodiment of the present application; Figure 5 A schematic diagram of the results of running different methods on non-independent, equally divided data sets provided in an embodiment of the present application; Figure 6 A schematic diagram of the effect of different methods provided in the embodiments of the present application on the delay for achieving the same accuracy. DETAILED DESCRIPTION

[0016] The embodiment of the present application solves the problem of insufficient consideration of the balance between the overhead and model accuracy of the hierarchical federated learning framework in the prior art by providing a hierarchical federated learning method with high generalization capability. The embodiment of the present application obtains training sample data provided by the client within a preset time period, determines whether the client participates in local model training based on the training sample data, and inputs the training sample data participating in the local model training into the DQN network to obtain predicted Q values ​​and target Q values. Then, the sample pairs of the client are obtained based on the training sample data and the expected loss values ​​are obtained based on the obtained sample pairs. At the same time, an edge server is selected based on the obtained expected loss values. Finally, the trained local model in the DQN network is uploaded to the selected edge server for model aggregation to generate an edge model, and the generated edge model is uploaded to the cloud server to generate a global model, thereby achieving an improvement in the balance between the overhead and model accuracy of the hierarchical federated learning framework.

[0017] The technical solution in the embodiment of the present application is to solve the problem of insufficient consideration of the balance between the overhead and model accuracy of the above-mentioned hierarchical federated learning framework. The overall idea is as follows: By obtaining the training sample data provided by the client within a preset time period, judging whether the client participates in the local model training based on the training sample data, obtaining the expected value of the loss during the local model training process and selecting the edge server, and then inputting the training sample data participating in the local model training into the DQN network to obtain the predicted Q value and target Q value, and finally uploading the trained local model to the selected edge server to generate a global model, achieving the effect of improving the balance between the overhead and model accuracy of the hierarchical federated learning framework.

[0018] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] like Figure 1As shown, it is a flowchart of a hierarchical federated learning method with high generalization ability provided by an embodiment of the present application. A hierarchical federated learning method with high generalization ability provided by an embodiment of the present application includes the following steps: S1, obtaining training sample data provided by the client within a preset time period, judging whether the client participates in local model training based on the training sample data, and if so, executing S2, a scheduling decision of only one action task is included in the preset time period, the client accesses the DQN network, the DQN network includes a Q network and a Target network, the training sample data is stored in a federated learning environment, the federated learning environment includes a state space and an action space, and the local model is trained through the training sample data in the federated learning environment; S2, inputting the training sample data participating in the local model training into the DQN network to obtain a predicted Q value and a target Q value, and at the same time according to the DQN network The built-in parameters of the Q network are trained to obtain the equilibrium state value. The predicted Q value is used to predict the output value of the local model after training in the current state, and the target Q value is used to reflect the expected value of the local model after training in a given state; S3, obtains the sample pairs of the client according to the training sample data and obtains the loss expectation value according to the obtained sample pairs, and selects the edge server according to the obtained loss expectation value, uploads the trained local model in the DQN network to the selected edge server for model aggregation to generate an edge model, and uploads the generated edge model to the cloud server to generate a global model. The edge server has generalization capability. Model aggregation means reducing the communication bandwidth of the edge server in the process of receiving the local model through federal averaging. The edge model is used to process the training sample data in real time and improve the scheduling decision. The global model is used to evaluate and improve the local model and edge model according to the scheduling decision.

[0020] It should be added that the built-in parameters include Q network parameters and Target network update frequency; Q network parameters include learning rate, decay rate, and discount factor; the equilibrium state value is the deviation Q value when the predicted Q value and the target Q value of the Q network reach an equilibrium state during training; the equilibrium state is the stable state corresponding to the deviation Q value equal to the reference deviation Q value; the deviation Q value is the difference between the target Q value and the predicted Q value, and the deviation Q value is not equal to 0; the reference deviation Q value is represented by the sum and average of the minimum non-zero historical deviation Q values ​​in the historical time period in the preset database.

[0021] In this embodiment, the learning rate is used to control the step size of the parameter change of the Q network at each update. Increasing the learning rate can speed up the training.

[0022] Decay Rate: The learning rate will gradually decrease as the training progresses. This rate of decrease is the decay rate, which helps to learn quickly in the early stages of training.

[0023] The Discount Factor ranges from 0 to 1 and is used to weigh the importance of current rewards and future rewards. The closer the discount factor is to 1, the more important the future rewards are, and the closer it is to 0, the more important the current rewards are. This helps the model find a balance between pursuing immediate rewards and long-term returns.

[0024] The Target network update frequency is used to control the frequency of updating the Target network parameters. A lower update frequency helps stabilize the training process, but may lead to a slower learning speed. An increase in the update frequency indicates that the training process is unstable, but it may also speed up the learning speed. Therefore, in actual applications, it is necessary to adjust and control it in combination with specific application scenarios to achieve improved accuracy and stability of Q network training.

[0025] like Figure 2 As shown, it is a flowchart of hierarchical federated learning provided in an embodiment of the present application; Figure 3 As shown, it is a schematic diagram of the deep reinforcement learning network provided in an embodiment of the present application; the Q network is the neural network architecture in the current state, the output layer of the Q network contains the number of neurons corresponding to the action space, and the output of the neurons corresponds to the predicted Q value of executing the corresponding action task in the current state; the Target network and the Q network structure: are the same target networks; the state space contains the training sample data input by the client, and the action space is used to reflect the increase or decrease of the corresponding training sample data of the client.

[0026] Specifically, since the DQN network needs to take into account both local model performance and system overhead, the generalization of the global model and the time required for federated learning to train an epoch need to be used as the reward function in the DQN network incentive mechanism. The specific restricted expression of the reward function is: ; Where r is the number of local model training rounds, , U is the total number of local model training rounds, Indicates the generalization of the global model corresponding to the local model in the rth round of training on the corresponding client. Indicates the generalization of the global model corresponding to the local model in the r-1th round of training on the corresponding client. It represents the initial generalization of the global model corresponding to the client's local model before training. Indicates the training sample data corresponding to the client participating in the local model training. Indicates the initial time of the corresponding frame before local model training, Indicates the running time of the framework during the r-1th round of training of the local model, Indicates the running time of the framework during the r-1th round of training of the local model, and Represents two hyperparameters in the reward function. It should be noted that one client corresponds to one training sample data. In practical applications, Usually set to 0.95, Typically set to 0.05.

[0027] In summary, by simultaneously considering the generalization of the global model (i.e., the performance of the local model on unseen training sample data) and the local model training time, the reward function can guide the DQN network to improve the local model performance while also paying attention to the effective use of framework resources. This helps the DQN network to better understand the comprehensive impact of different training strategies on framework resource overhead and model performance. In addition, by considering the local model training time, in the context of hierarchical federated learning, this helps to reduce the waiting time for each round of training, improves the reliability and stability of the hierarchical federated learning framework, makes it more suitable for actual application scenarios, and achieves an improvement in the balance between the hierarchical federated learning framework overhead and model accuracy.

[0028] Furthermore, the specific process of judging whether the client participates in the local model training according to the training sample data is as follows: judging whether the training sample data is equal to 0: if the training sample data is equal to 0, it indicates that the corresponding client does not participate in this round of local model training, and at the same time, the changes of the training sample data within the preset time period are monitored in real time; if the training sample data is not equal to 0, it indicates that the corresponding client participates in this round of local model training, and at the same time, the corresponding training sample data is input into the preset database for storage through the multi-layer perceptron, and the multi-layer perceptron is used to convert the training sample data provided by the client that is not equal to 0 into a processable state for Q-learning.

[0029] In this embodiment, whether the client participates in the local model training is determined by judging whether the training sample data is equal to 0, which can effectively avoid invalid training sample data from participating in the training process of the local model, thereby saving framework resources and time. Secondly, the training sample data is preprocessed by a multilayer perceptron (MLP) to convert it into a state that can be processed by the Q-learning algorithm (such as the encoding state of the training sample data), which helps to achieve data standardization and feature extraction. This preprocessing step can improve the generalization ability and training speed of the local model. In addition, by comprehensively considering the feature vectors and corresponding label values ​​of the training sample data within a preset time period, the local model can learn more comprehensive and accurate data distribution patterns. This learning method helps to enhance the generalization ability of the model, so that it can make more accurate predictions and decisions when facing new data or unknown situations, thereby achieving more effective training of the local model and improving the generalization ability of the local model.

[0030] Furthermore, the specific steps for obtaining the expected value of loss based on the obtained sample pairs are as follows: obtaining the loss function based on the predicted value and sample label value in the local model training process, the predicted value is a function corresponding to the local model parameters, and the loss function is used to quantify the degree of difference between the predicted value and the sample label value; obtaining the loss value of the sample pair corresponding to the client according to the obtained loss function, and obtaining the expected value of loss in combination with the sampling result corresponding to the loss value, the loss value is used to reflect the prediction accuracy of the local model on a single sample pair, the sampling result is the sample pair selected by the client, and the expected value of loss is used to reflect the average performance of the local model on the sample pairs of the client.

[0031] Among them, the sample pair consists of a sample feature and a sample label value; the sample feature is the feature vector of the training sample data within a preset time period; and the sample label value is the label value corresponding to the feature vector.

[0032] In this embodiment, the specific limiting expression of the loss value is: ; The specific restriction expression of the sampling result is: ; The specific limiting expression of the expected value of loss is: ; In the formula, represents the parameters of the loss function, Represents the loss value of the corresponding sample pair of the client, represents the characteristics of the training sample data, represents the sample label value, represents a sample pair, Indicates the client ID. , Indicates the training sample data corresponding to the client participating in the local model training. , M represents the number of clients selected to participate in local model training, Indicates the predicted value during the local model training process, j is the model number corresponding to the client participating in the local model training, , G is the total number of models corresponding to the clients participating in local model training, Represents the loss value of the corresponding sample pair of the client, represents the expected value of the loss of the corresponding sample pair of the client, Indicates the loss value of the corresponding sample pair of the client during the local model training process.

[0033] It should be noted that the loss function in this embodiment is the minimization objective function of the client corresponding sample pair. In summary, the specific restriction expression of the minimization objective function is: ; By calculating the expected value of loss, the average performance of the local model on client data can be accurately evaluated, which helps to understand the generalization ability of the model on different clients. The specific calculation of the loss value (as shown in the formula) directly reflects the prediction accuracy of the local model on a single sample pair. This quantification method makes the evaluation of model performance more objective and specific. Secondly, the calculation of the expected value of loss combines the loss value and the sampling result, which means that it takes into account the randomness and bias of data selection. In addition, the specific restricted expression of the minimization objective function provides a clear direction for the optimization process. By minimizing this objective function, it can be better adapted to the data distribution of the client, thereby improving the generalization ability and prediction accuracy of the model.

[0034] like Figure 4 As shown, it is a schematic diagram of the overlapping range of edge base stations provided in an embodiment of the present application. For client A within the communication range of edge servers 1, 2, and 3, since different edge servers may contain different built-in parameters of the DQN network, the parameters of the local model are processed differently. The generalization of different edge servers on client A can be calculated, and the edge server with the largest generalization can be selected to upload its own local parameters.

[0035] In order to highlight the effectiveness of generalization on client A, the following four image classification datasets are usually selected as representative AI applications of LE-HFL: (1) MNIST (Modified National Institute of Standards and Technology database) is a handwritten digit dataset in which each sample is Grayscale images; (2) Fashion-MNIST is a fabric dataset with the same sample format as MNIST; (3) CIFAR-10 is a general object dataset where each sample is RGB images; (4) GTSRB (German Traffic Sign Recognition Benchmark) is a commonly used dataset in traffic sign classification tasks, in which each sample is RGB (Red Green Blue) image.

[0036] In order to prove that the LE-HFL proposed in this paper can show better performance when processing non-independent and identically distributed data, the three methods of LE-HFL, FedAvg and HierFL are tested on four non-independent and identically distributed data sets. In order to compare the superiority of the three methods, the accuracy value, average loss and AUC value of the model are recorded.

[0037] like Figure 5 As shown, it is a schematic diagram of the results of running different methods on non-independent and identically distributed datasets provided by the embodiment of the present application, showing the number of rounds required for different methods to achieve the same accuracy on the dataset, wherein the first column represents the number of rounds required for the three methods to achieve a model accuracy of 0.9 on the non-independent and identically distributed MNIST dataset, the second column represents the number of rounds required for the three methods to achieve a model accuracy of 0.7 on the non-independent and identically distributed Fashion-MNIST dataset, and the third and fourth columns represent the number of rounds required for the three methods to achieve a model accuracy of 0.5 on the non-independent and identically distributed Cifar10 and GTSRB datasets. Compared with the other two methods, LE-HFL uses the amount of data trained based on deep reinforcement learning for training. Compared with always using the same data for training, clients with different resources should use training sample data with a faster DQN network convergence speed for training. However, on the non-independent and identically distributed GTSRB dataset, the convergence speed of the DQN network is reduced, resulting in the convergence speed of LE-HFL being slower than the other two methods.

[0038] like Figure 6 As shown, a schematic diagram of the impact of different methods provided in an embodiment of the present application on the delay to achieve the same accuracy is shown, which shows the results of the three methods on four non-IID data sets. Compared with the performance on the IID data set, the performance of HierFL and FedAvg on non-IID data has declined. Although the result of LE-HFL on the GTSRB data set is not as good as the other two methods, it maintains high accuracy, low loss and AUC values ​​closer to 1 on the other three data sets, and achieves faster convergence in all indicators, that is, the global model achieves higher generalization performance.

[0039] Furthermore, the specific process of training the Q network according to the built-in parameters includes: step 1, taking the training sample data participating in the local model training as input, and obtaining the training limited data for training the Q network, the training limited data includes the current state, the given state and the action task; step 2, inputting the current state into the Q network, obtaining the predicted Q value of the action task corresponding to the current state through the forward propagation process of the Q network, and obtaining the target Q value of the action task corresponding to the given state through the Target network; step 3, inputting the Q network parameters into the Target network according to the update frequency of the Target network through the back propagation process of the Q network to ensure the stability of the Q network during the training process; step 4, judging whether the deviation Q value is equal to the reference deviation Q value, if so, completing the Q network training, otherwise returning to step 1, and stopping the Q network training after the difference between the predicted Q value and the target Q value is equal to the reference deviation Q value.

[0040] In this embodiment, forward propagation (Forward Propagation) and backpropagation (Backpropagation) of the Q network are two working principles of the multilayer perceptron. The current state is the training state of the Q network at this time, and the given state is the next state predicted according to the predicted Q value after the Q network is trained in the current state. It should be noted that in order to ensure the stable operation of the Q network and the Target network, the parameters in the Target network are copied from the Q network. This embodiment optimizes its built-in parameters by continuously iteratively training the Q network to minimize the deviation between the predicted Q value and the target Q value, thereby gradually learning the strategy of selecting the optimal action corresponding to the action task under a given state, thereby achieving improved accuracy and efficiency of the Q value predicted by the Q network output.

[0041] Furthermore, the specific limiting expression of the deviation Q value is: ; ; ; Where t is the number of the current time step, , T is the total number of current time steps, represents the deviation Q value of the DQN network at the current time step t, represents the target Q value of the Target network at the current time step t, represents the predicted Q value of the Q network at the current time step t, represents the state space of the DQN network at the current time step t, represents the state value selected by the client in the state space, represents the action space of the DQN network at the current time step t, represents the action chosen by the client in the state space, The action target value representing the action selected by the client, represents the future coefficient, represents the cumulative discount from the current time step t to the future time step k, represents the instantaneous reward of the local model during the training process at the current time step t, represents the discount factor, Represents the scheduling decision of the action task into the future state space corresponding to the current time step t in the federated learning environment, It represents the maximum target Q value of all actions performed by the Target network in the future state space.

[0042] In this embodiment, in order to minimize the loss of the deviation Q value, the mean square error loss is calculated according to the obtained deviation Q value to generate the action target value of the optimal action of the action task under a given state. The specific restriction expression of the mean square error loss is: ; Where r is the number of local model training rounds, , U is the total number of local model training rounds, Represents a built-in parameter, represents the mean square error loss, represents the instantaneous reward of the local model at the current time step t corresponding to the rth round of training, It represents the maximum action target value of all actions performed by the Target network in the future state space during the rth round of local model training. represents the action prediction value of the Q network corresponding to the current time step t during the rth round of training of the local model; in, From Q Network, According to the current training status of the local model And the local action model parameters established by the action, which are usually used in deep reinforcement learning networks to balance the trade-off between minimizing the deviation Q value; From the Target network, used to generate The action target value for performing the best action under the state.

[0043] This embodiment can stabilize the training process and reduce the fluctuation of the Q network training process by using the Target network. This is because the Target network provides a stable target Q value, so that the Q network can gradually approach this stable target. Secondly, by considering the mean square error between the predicted value and the target value, it helps to reduce the occurrence of overfitting, so that the Q network can learn the strategy of taking the best action in the current state and the given state, thereby improving the accuracy and reliability of obtaining the deviation Q value.

[0044] Furthermore, the generated edge model is uploaded to the cloud server to generate a global model, and then the edge model loss expectation value and the loss expectation value are differenced to obtain the generalized gap between the global model and the local model; the edge model loss expectation value represents the optimal loss expectation of the edge model after weighted average aggregation; the generalized gap is used to reflect the generalization ability of the global model on the client.

[0045] In this embodiment, an edge model is generated, and then a global vision of the edge model is formed through a cloud server. The global vision is a field of view framework of the global model. This framework not only includes knowledge and features learned from multiple clients, but also can provide a deeper understanding of the overall data distribution and characteristics. The expected value of the edge model loss reflects the performance of the edge model on each client. After the cloud server collects edge models from multiple clients, it aggregates these models in a weighted average manner to generate a global model. This method can integrate the learning results of different clients and improve the generalization ability and performance of the global model.

[0046] Specifically, the specific restriction expression of the generalized gap is: ; In the formula, represents the generalized gap between the global model and the local model, represents the expected value of the marginal model loss, Represents the expected value of loss.

[0047] In the application scenario of mobile devices, when the mobile device is within the radiation range of two or more edge servers, compared with the prior art, the present invention proposes a personalized edge server decision mechanism (to optimize the generalization ability of the edge model), by aggregating local updates from different clients in a distributed environment, when the client is in the overlapping area of ​​the communication range of different edge servers, by optimizing the generalization of the edge model on the client, the optimal edge server is selected for the client to upload the local model, which can effectively cope with various client data, improve the generalization of the global model, and thus improve the accuracy of the global model.

[0048] To summarize, the embodiment of the present application obtains the training sample data provided by the client within a preset time period, determines whether the client participates in the local model training based on the training sample data, obtains the expected value of the loss in the local model training process and selects an edge server, and then inputs the training sample data participating in the local model training into the DQN network to obtain the predicted Q value and the target Q value, and finally uploads the trained local model to the selected edge server to generate a global model, thereby achieving an improvement in the balance between the DQN network and the accuracy of the local training model, and then achieving an improvement in the balance between the overhead of the hierarchical federated learning framework and the model accuracy, effectively solving the problem of insufficient consideration of the balance between the overhead of the hierarchical federated learning framework and the model accuracy in the prior art.

[0049] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0053] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0054] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A hierarchical federated learning method with high generalization capability, characterized in that: The following steps are involved: S1, obtaining training sample data provided by the client within a preset time period, judging whether the client participates in local model training according to the training sample data, and if so, executing S2, wherein the training sample data is stored in a federated learning environment, wherein the federated learning environment includes a state space and an action space, and the local model is trained by the training sample data in the federated learning environment; S2, input the training sample data participating in the local model training into the DQN network to obtain the predicted Q value and the target Q value, and train the Q network according to the built-in parameters of the DQN network to obtain the equilibrium state value, the predicted Q value is used to predict the output value of the local model after training in the current state, and the target Q value is used to reflect the expected value of the local model after training in a given state; S3, obtains sample pairs of the client according to the training sample data and obtains the expected loss value according to the obtained sample pairs, and selects the edge server according to the obtained expected loss value, uploads the trained local model in the DQN network to the selected edge server for model aggregation to generate an edge model, and uploads the generated edge model to the cloud server to generate a global model, the edge server has generalization capability, the edge model is used to process the training sample data in real time and improve scheduling decisions, and the global model is used to evaluate and improve the local model and edge model according to the scheduling decision.

2. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The DQN network includes a Q network and a Target network; The Q network is the neural network architecture in the current state; The output layer of the Q network contains a number of neurons corresponding to the action space; The output of the neuron corresponds to the predicted Q value of executing the corresponding action task in the current state; The Target network and the Q network structure are the same target network; The state space includes training sample data input by the client; The action space is used to reflect the increase or decrease of the training sample data corresponding to the client.

3. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The specific process of judging whether the client participates in local model training based on training sample data is as follows: Determine whether the training sample data is equal to 0: If the training sample data is equal to 0, it indicates that the corresponding client does not participate in this round of local model training, and monitors the changes of the training sample data in the preset time period in real time; If the training sample data is not equal to 0, it indicates that the corresponding client participates in this round of local model training. At the same time, the corresponding training sample data is input into the preset database through a multi-layer perceptron for storage. The multi-layer perceptron is used to convert the training sample data not equal to 0 provided by the client into a processable state for Q-learning.

4. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The sample pair consists of a sample feature and a sample label value; The sample feature is a feature vector of the training sample data within a preset time period; The sample label value is the label value corresponding to the feature vector.

5. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The specific steps of obtaining the expected value of loss based on the obtained sample pair are: Obtaining a loss function according to the predicted value and the sample label value in the local model training process, wherein the predicted value is a function corresponding to the local model parameter, and the loss function is used to quantify the degree of difference between the predicted value and the sample label value; The loss value of the sample pair corresponding to the client is obtained according to the obtained loss function, and the expected loss value is obtained in combination with the sampling result corresponding to the loss value. The loss value is used to reflect the prediction accuracy of the local model on a single sample pair. The sampling result is the sample pair selected by the client, and the expected loss value is used to reflect the average performance of the local model on the sample pairs of the client.

6. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The equilibrium state value is the deviation Q value when the predicted Q value and the target Q value of the Q network reach an equilibrium state during the training process; The built-in parameters include Q network parameters and Target network update frequency; The Q network parameters include learning rate, decay rate, and discount factor; The equilibrium state is a stable state corresponding to when the deviation Q value is equal to the reference deviation Q value; The deviation Q value is the difference between the target Q value and the predicted Q value; The reference deviation Q value is represented by the sum and average of the minimum values ​​of the historical deviation Q values ​​that are not 0 in the historical time period in the preset database.

7. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The specific process of training the Q network according to the built-in parameters of the DQN network includes: Step 1: taking the training sample data participating in the local model training as input, obtaining the training limited data for training the Q network, wherein the training limited data includes the current state, the given state and the action task; Step 2: Input the current state into the Q network, obtain the predicted Q value of the corresponding action task in the current state through the forward propagation process of the Q network, and obtain the target Q value of the corresponding action task in the given state through the Target network; Step 3: Input the Q network parameters into the Target network according to the Target network update frequency through the back propagation process of the Q network to ensure the stability of the Q network during the training process; Step 4: Determine whether the deviation Q value is equal to the reference deviation Q value. If so, complete the Q network training. Otherwise, return to step 1 and stop the Q network training until the difference between the predicted Q value and the target Q value is equal to the reference deviation Q value.

8. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The specific limiting expression of the deviation Q value is: ； ； ； Where t is the number of the current time step, , T is the total number of current time steps, represents the deviation Q value of the DQN network at the current time step t, represents the target Q value of the Target network at the current time step t, represents the predicted Q value of the Q network at the current time step t, represents the state space of the DQN network at the current time step t, represents the state value selected by the client in the state space, represents the action space of the DQN network at the current time step t, represents the action chosen by the client in the state space, The action target value representing the action selected by the client, represents the future coefficient, represents the cumulative discount from the current time step t to the future time step k, represents the instantaneous reward of the local model during the training process at the current time step t, represents the discount factor, Represents the scheduling decision of the action task into the future state space corresponding to the current time step t in the federated learning environment, It represents the maximum target Q value of all actions performed by the Target network in the future state space.

9. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The generating of the edge model further includes forming a global vision of the edge model through a cloud server; The global vision is the visual field framework of the global model.

10. A hierarchical federated learning method with high generalization capability as claimed in claim 1, characterized in that: The generated edge model is uploaded to the cloud server to generate a global model, and then the difference operation between the expected loss value of the edge model and the expected loss value is performed to obtain the generalized gap between the global model and the local model; The expected loss value of the edge model represents the optimal loss expectation of the edge model after weighted average aggregation; The generalized gap is used to reflect the generalization ability of the global model on the client.

Images