A hierarchical federated learning method with high generalization ability

Abstract

This invention discloses a hierarchical federated learning method with high generalization ability, relating to the field of electronic digital data processing technology. This hierarchical federated learning method with high generalization ability includes local model training judgment, Q-value acquisition, and global model generation. This invention acquires training sample data provided by the client within a preset time period, determines whether the client participates in local model training based on the training sample data, obtains the expected loss value during the local model training process, selects an edge server, and then inputs the training sample data participating in local model training into a DQN network to obtain the predicted Q-value and target Q-value. Finally, the trained local model is uploaded to the selected edge server to generate a global model, thereby improving the balance between the overhead and model accuracy of the hierarchical federated learning framework and solving the problem of insufficient consideration of the balance between overhead and model accuracy in existing hierarchical federated learning frameworks.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a hierarchical federated learning method with high generalization ability. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the continuous breakthroughs in deep learning and reinforcement learning, machine learning models are increasingly capable of solving complex problems. However, in traditional federated learning, whether it is synchronous updates and their variants or asynchronous updates, a large number of model parameters need to be exchanged in multiple updates. To overcome these challenges, researchers have proposed the concepts of federated learning (FL) and hierarchical learning (HL), and combined with the advantages of the DQN (Deep Q-Network) network in deep reinforcement learning (DRL), they have constructed a new learning framework, namely a hierarchical federated learning framework based on deep reinforcement learning with generalization capabilities. It consists of 1 cloud, L edge servers, N clients, and N reinforcement learning networks.

[0003] Existing technologies divide training tasks into multiple sub-tasks by leveraging the perception capabilities of deep neural networks and the decision-making capabilities of deep reinforcement learning. These sub-tasks are then executed by corresponding low-level policies. An incentive mechanism is designed using a model aggregation algorithm, while the federated learning process of participants is monitored in real time. Finally, the federated learning process is coordinated and interacted with according to high-level policies, thereby achieving intelligent decision-making in complex environments.

[0004] For example, the invention patent announcement CN112668877B, which describes a method and system for allocating resource information by combining federated learning and reinforcement learning, includes: the management and allocation of items to be allocated are uniformly allocated by a manager, and the number of items to be allocated cannot simultaneously satisfy all individuals; the manager designs a deep reinforcement learning model and distributes it to each individual, and the individual uses the deep reinforcement learning model to select items to be allocated; the manager generates a new global model through federated learning.

[0005] For example, the client selection and personalized privacy protection method in asynchronous federated edge learning disclosed in patent application CN117252253A includes: using a private dataset to train and update the client's local learning model locally; using an asynchronous aggregation scheme to generate stale models from clients that did not participate in the aggregation in time; when t=0, the edge server initializes global parameters and a list of stale models, broadcasts the global parameters to all clients for initial synchronization, and notifies the clients to start local training; after receiving the global model, the client uses the SGD algorithm to update locally and obtain local model parameters; after the client completes the above calculation process locally, it immediately uploads the data to the edge server via a wireless link.

[0006] However, in the process of implementing the inventive technical solution in the embodiments of this application, it was found that the above-mentioned technology has at least the following technical problems:

[0007] In existing technologies, incentive mechanism design mainly focuses on the heterogeneity of participants. This single-dimensional focus is often insufficient to fully address the challenges of long-term efficiency and stability in federated learning, and there is a problem of insufficient consideration of the balance between the overhead of hierarchical federated learning frameworks and model accuracy. Summary of the Invention

[0008] This application provides a hierarchical federated learning method with high generalization ability, which solves the problem of insufficient consideration of the balance between the cost and model accuracy in the existing hierarchical federated learning framework, and improves the balance between the cost and model accuracy of the hierarchical federated learning framework.

[0009] This application provides a hierarchical federated learning method with high generalization ability, including the following steps: S1, acquiring training sample data provided by the client within a preset time period, determining whether the client participates in local model training based on the training sample data, and if so, proceeding to S2, where the training sample data is stored in a federated learning environment, the federated learning environment including a state space and an action space, and the local model is trained using the training sample data in the federated learning environment; S2, inputting the training sample data participating in local model training into a DQN network to obtain the predicted Q-value and the target Q-value, and simultaneously training the Q-network according to the built-in parameters of the DQN network to obtain equilibrium state values, wherein the predicted Q-value is used... To predict the output value of the local model after training in the current state, the target Q value is used to reflect the expected value of the local model after training in a given state; S3, obtain the client's sample pairs based on the training sample data and obtain the expected loss value based on the obtained sample pairs. At the same time, select an edge server based on the obtained expected loss value, upload the local model trained in the DQN network to the selected edge server for model aggregation to generate an edge model, and upload the generated edge model to the cloud server to generate a global model. The edge server has generalization ability. The edge model is used to process training sample data in real time and improve scheduling decisions. The global model is used to evaluate and improve the local model and the edge model based on the scheduling decisions.

[0010] Furthermore, the specific process for determining whether a client participates in local model training based on training sample data is as follows: Determine if the training sample data is equal to 0: If the training sample data is equal to 0, it indicates that the corresponding client does not participate in this round of local model training, and the changes in the training sample data within a preset time period are monitored in real time; If the training sample data is not equal to 0, it indicates that the corresponding client participates in this round of local model training, and the corresponding training sample data is input into a preset database for storage via a multilayer perceptron. The multilayer perceptron is used to convert the non-zero training sample data provided by the client into a Q-learning processable state.

[0011] Furthermore, the specific steps for obtaining the expected loss value based on the acquired sample pairs are as follows: A loss function is obtained based on the predicted values ​​and sample label values ​​during the local model training process. The predicted values ​​are functions corresponding to the local model parameters, and the loss function is used to quantify the degree of difference between the predicted values ​​and the sample label values. The loss value for the corresponding sample pair on the client side is obtained based on the acquired loss function, and the expected loss value is obtained by combining the sampling results corresponding to the loss value. The loss value reflects the prediction accuracy of the local model on a single sample pair, and the sampling results are the sample pairs selected by the client. The expected loss value reflects the average performance of the local model on the client's sample pairs.

[0012] Furthermore, the step of inputting the training sample data participating in the local model training into the DQN network to obtain the predicted Q value and the target Q value, and then further includes training the Q network according to the built-in parameters of the DQN network to obtain the equilibrium state value; the built-in parameters include the Q network parameters and the update frequency of the target network; the Q network parameters include the learning rate, decay rate, and discount factor; the equilibrium state value is the deviation Q value when the predicted Q value and the target Q value reach an equilibrium state during the training process; the equilibrium state is the stable state corresponding to the deviation Q value equal to the reference deviation Q value; the deviation Q value is the difference between the target Q value and the predicted Q value; the reference deviation Q value is represented by the sum and average of the minimum non-zero deviation Q values ​​in the historical deviation Q values ​​within a preset database.

[0013] Furthermore, the specific process for training the Q-network based on the built-in parameters includes: Step 1, using the training sample data participating in the local model training as input to obtain training constraint data for training the Q-network, the training constraint data including the current state, the given state, and the action task; Step 2, inputting the current state into the Q-network, obtaining the predicted Q-value of the corresponding action task in the current state through the forward propagation process of the Q-network, and simultaneously obtaining the target Q-value of the corresponding action task in the given state through the Target network; Step 3, inputting the Q-network parameters into the Target network according to the update frequency of the Target network through the backpropagation process of the Q-network to ensure the stability of the Q-network during the training process; Step 4, determining whether the deviation Q-value is equal to the reference deviation Q-value. If so, the Q-network training is completed; otherwise, return to Step 1, and stop the Q-network training when the difference between the predicted Q-value and the target Q-value is equal to the reference deviation Q-value.

[0014] Furthermore, the specific limiting expression for the deviation Q value is as follows:

[0015] ;

[0016] ;

[0017] ;

[0018] In the formula, t is the number of the current time step. T is the total number of steps at the current time. This represents the deviation Q value of the DQN network at the current time step t. This represents the target Q-value of the Target network at the current time step t. This represents the predicted Q-value of the Q-network at the current time step t. This represents the state space of the DQN network at the current time step t. This represents the state value selected by the client in the state space. This represents the action space of the DQN network at the current time step t. This represents the action selected by the client in the state space. This indicates the target value of the action selected by the client. Represents future coefficients. This represents the cumulative discount from the current time step t to the future time step k. This represents the immediate reward of the local model during the training process at the current time step t. Indicates the discount factor. This indicates that the scheduling decision for the action task enters the future state space corresponding to the current time step t in the federated learning environment. This represents the maximum target Q value for the Target network to execute all actions in the future state space.

[0019] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0020] 1. By acquiring training sample data provided by the client within a preset time period, the system determines whether the client participates in local model training based on the training sample data. Simultaneously, it obtains the expected loss value during the local model training process and selects an edge server. Then, the training sample data participating in the local model training is input into the DQN network to obtain the predicted Q value and the target Q value. Finally, the trained local model is uploaded to the selected edge server to generate a global model. This improves the balance between the accuracy of the DQN network and the locally trained model, thereby improving the balance between the overhead of the hierarchical federated learning framework and the model accuracy. This effectively solves the problem of insufficient consideration of the balance between the overhead and the model accuracy in the existing technology.

[0021] 2. By determining whether the training sample data is equal to 0: If the training sample data is equal to 0, the changes in the training sample data within a preset time period are monitored in real time. If the training sample data is not equal to 0, the corresponding training sample data is input into a preset database through a multilayer perceptron for storage. This achieves a more accurate judgment of the training sample data, thereby improving the accuracy of the judgment of the local model training client.

[0022] 3. The loss function is obtained through the predicted values ​​and sample label values ​​during the local model training process. Then, the loss value of the corresponding sample pair on the client is obtained based on the obtained loss function. Finally, the expected loss value is obtained by combining the sampling results corresponding to the loss value, thereby achieving a more accurate acquisition of the loss value and improving the accuracy and reliability of the expected loss value acquisition. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a hierarchical federated learning method with high generalization ability provided for embodiments of this application;

[0024] Figure 2 A flowchart of hierarchical federated learning provided for embodiments of this application;

[0025] Figure 3 A schematic diagram of a deep reinforcement learning network provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram showing the overlapping range of edge base stations provided in an embodiment of this application;

[0027] Figure 5 A schematic diagram illustrating the results of running different methods on a non-independent equally divided dataset, provided as an embodiment of this application.

[0028] Figure 6 This diagram illustrates the time delay impact of different methods provided in the embodiments of this application on achieving the same accuracy. Detailed Implementation

[0029] This application provides a hierarchical federated learning method with high generalization ability, which solves the problem of insufficient consideration of the balance between the overhead and model accuracy in the existing hierarchical federated learning framework. By acquiring training sample data provided by the client within a preset time period, it determines whether the client participates in local model training based on the training sample data. At the same time, the training sample data participating in local model training is input into the DQN network to obtain the predicted Q value and the target Q value. Then, the client's sample pairs are obtained based on the training sample data, and the expected loss value is obtained based on the obtained sample pairs. At the same time, the edge server is selected based on the obtained expected loss value. Finally, the local model trained in the DQN network is uploaded to the selected edge server for model aggregation to generate an edge model. The generated edge model is then uploaded to the cloud server to generate a global model, thereby improving the balance between the overhead and model accuracy of the hierarchical federated learning framework.

[0030] The technical solution in this application aims to address the problem of insufficient consideration of the balance between overhead and model accuracy in the aforementioned hierarchical federated learning framework. The overall approach is as follows:

[0031] By acquiring training sample data provided by the client within a preset time period, determining whether the client participates in local model training based on the training sample data, obtaining the expected loss value during the local model training process, selecting an edge server, and then inputting the training sample data participating in local model training into the DQN network to obtain the predicted Q value and the target Q value, the trained local model is finally uploaded to the selected edge server to generate a global model, thus achieving the effect of improving the balance between the overhead and model accuracy of the hierarchical federated learning framework.

[0032] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0033] like Figure 1 The diagram shows a flowchart of a hierarchical federated learning method with high generalization ability provided in this application embodiment. The hierarchical federated learning method with high generalization ability provided in this application embodiment includes the following steps: S1, obtaining training sample data provided by the client within a preset time period, determining whether the client participates in local model training based on the training sample data, and if so, proceeding to S2, scheduling decisions for only one action task within the preset time period, the client accessing the DQN network, which includes a Q network and a Target network, the training sample data being stored in a federated learning environment, which includes a state space and an action space, and the local model being trained using the training sample data in the federated learning environment; S2, inputting the training sample data participating in local model training into the DQN network to obtain the predicted Q value and the target Q value, and simultaneously, according to the DQN network... The built-in parameters are used to train the Q-network to obtain equilibrium state values. The predicted Q-value is used to predict the output value of the local model after training in the current state. The target Q-value is used to reflect the expected value of the local model after training in a given state. S3, the client's sample pairs are obtained based on the training sample data, and the expected loss value is obtained based on the obtained sample pairs. At the same time, the edge server is selected based on the obtained expected loss value. The local model trained in the DQN network is uploaded to the selected edge server for model aggregation to generate an edge model. The generated edge model is then uploaded to the cloud server to generate a global model. The edge server has generalization ability. Model aggregation means reducing the communication bandwidth of the edge server in the process of receiving the local model through federated averaging. The edge model is used to process the training sample data in real time and improve the scheduling decision. The global model is used to evaluate and improve the local model and the edge model based on the scheduling decision.

[0034] It should be added that the built-in parameters include the Q-network parameters and the Target network update frequency; the Q-network parameters include the learning rate, decay rate, and discount factor; the equilibrium state value is the deviation Q-value when the predicted Q-value and the target Q-value reach an equilibrium state during the training process; the equilibrium state is the stable state corresponding to when the deviation Q-value equals the reference deviation Q-value; the deviation Q-value is the difference between the target Q-value and the predicted Q-value, and the deviation Q-value is not equal to 0; the reference deviation Q-value is represented by the sum and average of the minimum non-zero deviation Q-values ​​in the historical deviation Q-values ​​within the historical time period in the preset database.

[0035] In this embodiment, the learning rate is used to control the step size of parameter changes in the Q-network during each update. Increasing the learning rate can speed up the training process.

[0036] Decay Rate: The learning rate gradually decreases as training progresses. This rate of decrease is called the decay rate, which helps in rapid learning during the early stages of training.

[0037] The discount factor ranges from 0 to 1 and is used to weigh the importance of current rewards against future rewards. The closer the discount factor is to 1, the more important the future rewards are, and the closer it is to 0, the more important the current rewards are. This helps the model find a balance between pursuing immediate rewards and long-term returns.

[0038] The Target Network update frequency controls the frequency of Target Network parameter updates. A lower update frequency helps stabilize the training process but may slow down the learning speed. An higher update frequency indicates that the training process is unstable but may also speed up the learning speed. Therefore, in practical applications, it is necessary to adjust the frequency according to the specific application scenario to improve the accuracy and stability of Q-network training.

[0039] like Figure 2 The diagram shown is a flowchart of hierarchical federated learning provided in an embodiment of this application; as shown Figure 3 The diagram shown is a schematic of a deep reinforcement learning network provided in an embodiment of this application. The Q network is the neural network architecture in the current state. The output layer of the Q network contains the number of neurons corresponding to the action space. The output of the neurons corresponds to the predicted Q value of the corresponding action task in the current state. The Target network has the same structure as the Q network. The state space contains the training sample data input by the client, and the action space is used to reflect the increase or decrease of the corresponding training sample data by the client.

[0040] Specifically, because the DQN network needs to balance local model performance and system overhead, the generalization ability of the global model and the training time of one epoch in federated learning need to be used as the reward function in the DQN network's incentive mechanism. The specific constrained expression of the reward function is as follows:

[0041] ;

[0042] In the formula, r is the number of the local model training round. U represents the total number of training rounds for the local model. This indicates the generalization ability of the local model to the corresponding global model trained in the r-th round on the corresponding client. This indicates the generalization ability of the global model corresponding to the local model trained in the (r-1)th round on the corresponding client. This indicates the initial generalization performance of the global model compared to the client's local model before training. This represents the training sample data corresponding to the clients participating in local model training. This indicates the initial time of the corresponding framework before the local model is trained. This represents the runtime of the framework during the (r-1)th round of training of the local model. This represents the runtime of the framework during the (r-1)th round of training of the local model. and These represent two hyperparameters in the reward function. It's important to note that one client corresponds to one training sample data point; in practical applications, It is usually set to 0.95. It is usually set to 0.05.

[0043] In summary, by simultaneously considering both the generalization ability of the global model (i.e., the performance of the local model on unseen training sample data) and the training time of the local model, the reward function can guide the DQN network to improve the performance of the local model while also focusing on the effective utilization of framework resources. This helps the DQN network better understand the combined impact of different training strategies on framework resource overhead and model performance. Furthermore, by considering the training time of the local model, in the context of hierarchical federated learning, this helps to reduce the waiting time for each training round, improves the reliability and stability of the hierarchical federated learning framework, and makes it more suitable for practical application scenarios, thus achieving a balance between the overhead and model accuracy of the hierarchical federated learning framework.

[0044] Furthermore, the specific process for determining whether a client participates in local model training based on training sample data is as follows: Determine if the training sample data is equal to 0: If the training sample data is equal to 0, it indicates that the corresponding client does not participate in this round of local model training, and the changes in the training sample data within a preset time period are monitored in real time; If the training sample data is not equal to 0, it indicates that the corresponding client participates in this round of local model training, and the corresponding training sample data is input into a preset database for storage via a multilayer perceptron. The multilayer perceptron is used to convert the non-zero training sample data provided by the client into a processable state for Q-learning.

[0045] In this embodiment, determining whether the client participates in local model training by judging whether the training sample data is equal to 0 can effectively avoid invalid training sample data from participating in the local model training process, thereby saving framework resources and time. Secondly, the training sample data is preprocessed by a multilayer perceptron (MLP) to convert it into a state that the Q-learning algorithm can handle (such as the encoding state of the training sample data), which helps to achieve data standardization and feature extraction. This preprocessing step can improve the generalization ability and training speed of the local model. In addition, by comprehensively considering the feature vectors and corresponding label values ​​of the training sample data within a preset time period, the local model can learn a more comprehensive and accurate data distribution pattern. This learning method helps to enhance the generalization ability of the model, enabling it to make more accurate predictions and decisions when facing new data or unknown situations, thus achieving more effective training of the local model and improving its generalization ability.

[0046] Furthermore, the specific steps for obtaining the expected loss value based on the acquired sample pairs are as follows: Obtain the loss function based on the predicted values ​​and sample label values ​​during the local model training process. The predicted values ​​are functions corresponding to the local model parameters, and the loss function is used to quantify the degree of difference between the predicted values ​​and the sample label values. Obtain the loss value for the corresponding sample pair on the client side based on the obtained loss function, and simultaneously obtain the expected loss value by combining the sampling results corresponding to the loss value. The loss value is used to reflect the prediction accuracy of the local model on a single sample pair, and the sampling results are the sample pairs selected by the client. The expected loss value is used to reflect the average performance of the local model on the client's sample pairs.

[0047] Each sample pair consists of sample features and sample label values; the sample features are the feature vectors of the training sample data within a preset time period; and the sample label values ​​are the label values ​​corresponding to the feature vectors.

[0048] In this embodiment, the specific constraint expression for the loss value is:

[0049] ;

[0050] The specific constraint expression for the sampling result is:

[0051] ;

[0052] The specific constraint expression for the expected value of loss is:

[0053] ;

[0054] In the formula, The parameters represent the loss function. This represents the loss value for the corresponding sample pair on the client side. Representing the features of the training sample data, Indicates the sample label value. To represent a sample pair, Indicates the client's ID. , This represents the training sample data corresponding to the clients participating in local model training. M represents the number of clients selected to participate in local model training. This represents the predicted value during the local model training process, where j is the model number corresponding to the client participating in the local model training. G represents the total number of models corresponding to clients participating in local model training. This represents the loss value for the corresponding sample pair on the client side. This represents the expected loss value for the corresponding sample pair on the client side. This represents the loss value for the corresponding sample pair on the client side during the local model training process.

[0055] It should be noted that the loss function in this embodiment is the objective function for minimizing the corresponding sample pair on the client side. Therefore, the specific constrained expression for minimizing the objective function is as follows:

[0056] ;

[0057] By calculating the expected loss, the average performance of the local model on client data can be accurately evaluated. This helps to understand the model's generalization ability on different clients. The specific calculation of the loss value (as shown in the formula) directly reflects the prediction accuracy of the local model on a single sample pair. This quantification method makes the evaluation of model performance more objective and specific. Secondly, the calculation of the expected loss value combines the loss value and the sampling results, which means that it takes into account the randomness and bias of data selection. In addition, the specific constraint expression of minimizing the objective function provides a clear direction for the optimization process. By minimizing this objective function, it can better adapt to the data distribution of the client, thereby improving the model's generalization ability and prediction accuracy.

[0058] like Figure 4 The diagram shown is a schematic diagram of the overlapping range of edge base stations provided in the embodiment of this application. For client A, which is within the communication range of edge servers 1, 2, and 3, since different edge servers may contain different built-in parameters of the DQN network, they have different processing methods for the parameters of the local model. The generalization of different edge servers on client A can be calculated, and the edge server with the largest generalization can be selected to upload its own local parameters.

[0059] To highlight the effectiveness of generalization on client A, the following four image classification datasets are typically selected as representative AI applications of LE-HFL:

[0060] (1) MNIST (Modified National Institute of Standards and Technology database) is a handwritten digit dataset in which each sample is a... Grayscale image;

[0061] (2) Fashion-MNIST is a fabric dataset, and its sample format is the same as MNIST;

[0062] (3) CIFAR-10 is a general object dataset in which each sample is RGB images;

[0063] (4) GTSRB (German Traffic Sign Recognition Benchmark) is a commonly used dataset in traffic sign classification tasks, where each sample is... RGB (Red Green Blue) image.

[0064] To demonstrate that the proposed LE-HFL performs better when handling non-independent and identically distributed data, three methods—LE-HFL, FedAvg, and HierFL—were tested on four non-independent and identically distributed datasets. To compare the superiority of the three methods, the accuracy, average loss, and AUC of the models were recorded.

[0065] like Figure 5The diagram illustrates the results of running different methods on a non-independent, identically distributed dataset, as provided in this application embodiment. It shows the number of rounds required for different methods to achieve the same accuracy on the dataset. The first column represents the number of rounds required for the three methods to achieve a model accuracy of 0.9 on the non-independent, identically distributed MNIST dataset; the second column represents the number of rounds required for the three methods to achieve a model accuracy of 0.7 on the non-independent, identically distributed Fashion-MNIST dataset; and the third and fourth columns represent the number of rounds required for the three methods to achieve a model accuracy of 0.5 on the non-independent, identically distributed Cifar10 and GTSRB datasets. Compared to the other two methods, LE-HFL uses a dataset trained based on deep reinforcement learning. Instead of consistently using the same data for training, clients with varying resources should use training samples with faster convergence speeds for DQN networks. However, on the non-independent, identically distributed GTSRB dataset, the convergence speed of the DQN network decreases, causing LE-HFL to converge slower than the other two methods.

[0066] like Figure 6 The diagram illustrates the time delay impact of different methods provided in this application's embodiments on achieving the same accuracy. It shows the results of the three methods on four non-IID datasets. Compared to their performance on the IID dataset, HierFL and FedAvg both show a decrease in performance on non-IID data. Although LE-HFL's results on the GTSRB dataset are not as good as the other two methods, it maintains high accuracy, low loss, and an AUC value closer to 1 on the other three datasets. Furthermore, it achieves faster convergence across all metrics, meaning the global model achieves higher generalization performance.

[0067] Furthermore, the specific process for training the Q-network based on the built-in parameters includes: Step 1, using the training sample data participating in the local model training as input to obtain the training constraint data for training the Q-network, which includes the current state, the given state, and the action task; Step 2, inputting the current state into the Q-network, obtaining the predicted Q-value of the corresponding action task in the current state through the forward propagation process of the Q-network, and simultaneously obtaining the target Q-value of the corresponding action task in the given state through the Target network; Step 3, inputting the Q-network parameters into the Target network according to the update frequency of the Target network through the backpropagation process of the Q-network to ensure the stability of the Q-network during training; Step 4, determining whether the bias Q-value is equal to the reference bias Q-value. If so, the Q-network training is completed; otherwise, return to Step 1, and stop the Q-network training when the difference between the predicted Q-value and the target Q-value equals the reference bias Q-value.

[0068] In this embodiment, the forward propagation and backpropagation of the Q-network are the two working principles of the multilayer perceptron. The current state is the training state of the Q-network at this time, and the given state is the next state predicted by the Q-network based on the predicted Q-value after training in the current state. It should be noted that in order to ensure the stable operation of the Q-network and the Target network, the parameters in the Target network are copied from the Q-network. This embodiment optimizes its built-in parameters by iteratively training the Q-network to minimize the deviation between the predicted Q-value and the target Q-value, thereby gradually learning the strategy of selecting the optimal action corresponding to the action task in a given state, thus improving the accuracy and efficiency of the Q-network's output predicted Q-value.

[0069] Furthermore, the specific constraint expression for the deviation Q value is as follows:

[0070] ;

[0071] ;

[0072] ;

[0073] In the formula, t is the number of the current time step. T is the total number of steps at the current time. This represents the deviation Q value of the DQN network at the current time step t. This represents the target Q-value of the Target network at the current time step t. This represents the predicted Q-value of the Q-network at the current time step t. This represents the state space of the DQN network at the current time step t. This represents the state value selected by the client in the state space. This represents the action space of the DQN network at the current time step t. This represents the action selected by the client in the state space. This indicates the target value of the action selected by the client. Represents future coefficients. This represents the cumulative discount from the current time step t to the future time step k. This represents the immediate reward of the local model during the training process at the current time step t. Indicates the discount factor. This indicates that the scheduling decision for the action task enters the future state space corresponding to the current time step t in the federated learning environment. This represents the maximum target Q value for the Target network to execute all actions in the future state space.

[0074] In this embodiment, to minimize the loss of the deviation Q value, the mean squared error loss is calculated based on the obtained deviation Q value to generate the optimal action target value for the action task under a given state. The specific constraint expression for the mean squared error loss is as follows:

[0075] ;

[0076] In the formula, r is the number of the local model training round. U represents the total number of training rounds for the local model. Indicates built-in parameters. Indicates the mean square error loss. This represents the immediate reward of the local model at the current time step t, corresponding to the r-th round of training. This represents the maximum action target value of the Target network in the future state space during the r-th round of local model training. This represents the action prediction value of the Q-network at the current time step t during the r-th round of training of the local model;

[0077] in, From Q Network It is based on the current training state of the local model. And the local action model parameters for action establishment, which are usually used to balance deep reinforcement learning networks with minimizing the bias Q value; From the Target network, used in the generation The target value for the action to perform the best action under the given state.

[0078] This embodiment uses a Target network to stabilize the training process and reduce fluctuations in the Q-network training process. This is because the Target network provides a stable target Q-value, allowing the Q-network to gradually approach this stable target. Secondly, by considering the mean square error between the predicted value and the target value, it helps to reduce overfitting, enabling the Q-network to learn the strategy of taking the best action in the current state and a given state, thereby improving the accuracy and reliability of obtaining the deviation Q-value.

[0079] Furthermore, the generated edge model is uploaded to the cloud server to generate the global model. Then, the difference between the expected loss of the edge model and the expected loss is calculated to obtain the generalized gap between the global model and the local model. The expected loss of the edge model represents the optimal expected loss of the edge model after weighted average aggregation. The generalized gap is used to reflect the generalization ability of the global model on the client.

[0080] In this embodiment, an edge model is generated, followed by the formation of a global vision of the edge model via a cloud server. The global vision is the visual framework of the global model, which not only includes the knowledge and features learned from multiple clients, but also provides a deeper understanding of the overall data distribution and characteristics. The expected value of the edge model loss reflects the performance of the edge model on its respective client. After the cloud server collects edge models from multiple clients, it aggregates these models using a weighted average to generate a global model. This method can integrate the learning results of different clients and improve the generalization ability and performance of the global model.

[0081] Specifically, the specific constraint expression for the generalized gap is:

[0082] ;

[0083] In the formula, This represents the generalized difference between the global model and the local model. This represents the expected value of the marginal model loss. This represents the expected value of the loss.

[0084] In mobile device application scenarios, when a mobile device is within the coverage area of ​​two or more edge servers, compared with existing technologies, this invention proposes a personalized edge server decision mechanism (to optimize the generalization capability of the edge model). By aggregating local updates from different clients in a distributed environment, when a client is in an area where the communication ranges of different edge servers overlap, the invention optimizes the generalization of the edge model on the client and selects the optimal edge server for the client to upload the local model. This can effectively cope with various client data, improve the generalization capability of the global model, and thus improve the accuracy of the global model.

[0085] In summary, this application embodiment obtains training sample data provided by the client within a preset time period, determines whether the client participates in local model training based on the training sample data, obtains the expected loss value during the local model training process and selects an edge server, then inputs the training sample data participating in local model training into the DQN network to obtain the predicted Q value and the target Q value, and finally uploads the trained local model to the selected edge server to generate a global model. This improves the balance between the accuracy of the DQN network and the locally trained model, thereby improving the balance between the overhead of the hierarchical federated learning framework and the model accuracy, effectively solving the problem of insufficient consideration of the balance between the overhead and the model accuracy in the prior art.

[0086] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0091] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A hierarchical federated learning method with high generalization ability, characterized in that, Includes the following steps: S1, obtain training sample data provided by the client within a preset time period, determine whether the client participates in local model training based on the training sample data, if so, execute S2, the training sample data is stored in a federated learning environment, the federated learning environment includes a state space and an action space, the local model is trained using the training sample data in the federated learning environment; S2, the training sample data participating in the local model training is input into the DQN network to obtain the predicted Q value and the target Q value. At the same time, the Q network is trained according to the built-in parameters of the DQN network to obtain the equilibrium state value. The predicted Q value is used to predict the output value of the local model after training in the current state, and the target Q value is used to reflect the expected value of the local model after training in a given state. S3. Obtain sample pairs from the client based on the training sample data and obtain the expected loss value based on the obtained sample pairs. At the same time, select an edge server based on the obtained expected loss value, upload the local model trained in the DQN network to the edge server for model aggregation to generate an edge model, and upload the generated edge model to the cloud server to generate a global model. The edge server has generalization ability. The edge model is used to process training sample data in real time and improve scheduling decisions. The global model is used to evaluate and improve the local model and the edge model based on the scheduling decisions. The specific process for determining whether the client participates in local model training based on training sample data is as follows: Determine if the training sample data is equal to 0: If the training sample data is equal to 0, the changes in the training sample data within the preset time period will be monitored in real time. If the training sample data is not equal to 0, the corresponding training sample data is input into the preset database through a multilayer perceptron for storage. The multilayer perceptron is used to convert the non-zero training sample data provided by the client into a processable state of Q-learning. The step of determining whether the client participates in local model training based on training sample data also includes obtaining sample pairs based on training sample data. The sample pair consists of sample features and sample label values; The sample features are feature vectors of training sample data within a preset time period; The sample label value is the label value corresponding to the feature vector; The step of obtaining sample pairs based on training sample data further includes obtaining the expected loss value for the corresponding client based on the obtained sample pairs. The expected loss value is used to reflect the average performance of the local model on sample pairs on the client. The steps for obtaining the expected value of the loss are as follows: The loss function is obtained based on the predicted values ​​and sample label values ​​during the local model training process. The predicted values ​​are functions corresponding to the local model parameters, and the loss function is used to quantify the degree of difference between the predicted values ​​and the sample label values. The loss value for the corresponding sample pair on the client is obtained based on the acquired loss function. At the same time, the expected loss value is obtained by combining the sampling result corresponding to the loss value. The loss value is used to reflect the prediction accuracy of the local model on a single sample pair. The sampling result is the sample pair selected by the client.

2. The hierarchical federated learning method with high generalization ability as described in claim 1, characterized in that, The DQN network includes a Q network and a Target network; The Q-network is the neural network architecture in the current state; The output layer of the Q network contains the number of neurons corresponding to the action space; The output of the neuron corresponds to the predicted Q value for performing the corresponding action task in the current state; The Target network and Q network have the same target network structure. The state space contains the training sample data input by the client; The action space is used to reflect the increase or decrease of the corresponding training sample data on the client side.

3. The hierarchical federated learning method with high generalization ability as described in claim 1, characterized in that, The built-in parameters include Q network parameters and the Target network update frequency; The Q-network parameters include learning rate, decay rate, and discount factor; The equilibrium state value is the deviation Q value between the predicted Q value and the target Q value of the Q network during the training process. The equilibrium state is the stable state corresponding to when the deviation Q value is equal to the reference deviation Q value; The deviation Q value is the difference between the target Q value and the predicted Q value; The reference deviation Q value is represented by the sum and average of the minimum non-zero historical deviation Q values ​​within a historical time period in the preset database.

4. The hierarchical federated learning method with high generalization ability as described in claim 1, characterized in that, The specific process for training the Q-network based on the built-in parameters of the DQN network includes: Step 1: Use the training sample data participating in the local model training as input to obtain the training constraint data for training the Q network. The training constraint data includes the current state, the given state, and the action task. Step 2: Input the current state into the Q network, obtain the predicted Q value of the corresponding action task in the current state through the forward propagation process of the Q network, and at the same time obtain the target Q value of the corresponding action task in the given state through the Target network. Step 3: Through the backpropagation process of the Q network, the parameters of the Q network are input into the Target network according to the update frequency of the Target network to ensure the stability of the Q network during training. Step 4: Determine whether the bias Q value is equal to the reference bias Q value. If yes, complete the Q-network training; otherwise, return to step 1. Continue training the Q-network until the difference between the predicted Q value and the target Q value equals the reference bias Q value.

5. The hierarchical federated learning method with high generalization ability as described in claim 3, characterized in that, The specific limiting expression for the deviation Q value is: ； ； ； In the formula, t is the number of the current time step. T is the total number of steps at the current time. This represents the deviation Q value of the DQN network at the current time step t. This represents the target Q-value of the Target network at the current time step t. This represents the predicted Q-value of the Q-network at the current time step t. This represents the state space of the DQN network at the current time step t. This represents the state value selected by the client in the state space. This represents the action space of the DQN network at the current time step t. This indicates the action selected by the client in the action space. This indicates the target value of the action selected by the client. Represents future coefficients. This represents the cumulative discount from the current time step t to the future time step k. This represents the immediate reward of the local model during the training process at the current time step t. Indicates the discount factor. This indicates that the scheduling decision for the action task enters the future state space corresponding to the current time step t in the federated learning environment. This represents the maximum target Q value for the Target network to execute all actions in the future state space.

6. The hierarchical federated learning method with high generalization ability as described in claim 1, characterized in that, The generation of the edge model then includes a global vision of the edge model formed via a cloud server; The global vision refers to the field of view framework of the global model.

7. The hierarchical federated learning method with high generalization ability as described in claim 1, characterized in that, The process of uploading the generated edge model to the cloud server to generate the global model further includes performing a difference operation between the expected loss value of the edge model and the expected loss value to obtain the generalized difference between the global model and the local model. The expected loss value of the edge model represents the optimal expected loss of the edge model after weighted average aggregation. The generalized gap is used to reflect the generalization ability of the global model on the client side.

Images