Federal learning incentive method, device and system based on deep reinforcement learning

Abstract

The application discloses a federated learning incentive method, device and system based on deep reinforcement learning, and the method comprises the following steps: acquiring model parameters uploaded by each participant client user after current round iteration training; performing iteration test according to the model parameters to obtain multi-dimensional test parameters of each participant client user; inputting the multi-dimensional test parameters of the participant client user and total fees required to be proportionally paid in the current round into a pre-constructed incentive distribution model to determine the reward corresponding to the participant client user. According to the application, effective incentive distribution can be realized for the participants of the federated learning, and the participation enthusiasm of high-contribution clients is improved.

Description

Technical Field

[0001] This invention relates to the field of distributed model training, and more specifically to a federated learning incentive method, apparatus, and system based on deep reinforcement learning. Background Technology

[0002] With the rapid development of IoT, edge computing, and mobile internet technologies, massive amounts of data are generated in real time on terminal devices. To protect user data privacy and reduce the communication overhead of centralized data transmission, Federated Learning (FL), a distributed machine learning paradigm, has emerged. This mechanism allows multiple participating clients to collaboratively train a global model without sharing the original data, with a central server aggregating local model updates to achieve the privacy goal of "the model moves while the data remains stationary." However, the practical deployment of federated learning faces core challenges such as insufficient participation, ambiguous contribution evaluation, and a lack of incentive mechanisms. Especially in open, heterogeneous network environments, client computing resources, data quality, and communication capabilities vary significantly. Rational participants tend to "free-ride," submitting low-quality or fake model updates to circumvent costs, severely impairing the global model's convergence efficiency and final performance.

[0003] Existing incentive mechanisms largely follow the framework of traditional game theory or auction theory, evaluating client contributions and allocating rewards based on pre-defined rules. These methods rely on strong assumptions, such as known client types, static environments, and perfect information, making them ill-suited to the complexities of federated learning, including dynamic participation, non-independent and identically distributed (Non-IID) data, and communication delays. Furthermore, contribution evaluation often employs ex-post calculations such as marginal contribution and Shapley values, incurring high computational costs and lacking online optimization capabilities, resulting in delayed incentive responses and hindering the cultivation of long-term, high-quality participation. While some studies have attempted to incorporate reinforcement learning for dynamic incentive design, most remain confined to centralized decision-making, failing to fully consider the decentralized and privacy-sensitive nature of federated systems, and posing risks of model leakage and policy homogenization. Summary of the Invention

[0004] This invention provides a federated learning incentive method, apparatus, and system based on deep reinforcement learning, which enables effective incentive allocation to participants in federated learning and enhances the participation enthusiasm of high-contribution clients.

[0005] Therefore, the present invention provides the following technical solution:

[0006] A federated learning incentive method based on deep reinforcement learning, the method comprising:

[0007] Obtain the model parameters uploaded by each participating client user after the current round of iteration training;

[0008] Iterative testing is performed based on the model parameters to obtain multi-dimensional test parameters for each participating client user.

[0009] Input the multi-dimensional test parameters of all participating client users into the pre-built incentive allocation model to determine the cost ratio required for the current iteration and the incentive allocation ratio for each participating client user;

[0010] The rewards for each participating client user are calculated based on the remaining total funds, the cost ratio required for the current iteration, and the incentive allocation ratio for each participating client user.

[0011] Optionally, iterative testing is performed based on the model parameters to obtain multi-dimensional test parameters for each participating client user, including:

[0012] Aggregate the model parameters of all participating client users to obtain the first aggregated parameter, and generate the first model based on the first aggregated parameter;

[0013] The model parameters after excluding specific client users are aggregated to obtain second aggregated parameters, and a second model based on the second aggregated parameters is generated.

[0014] The first model and the second model are iteratively tested using the set test set to obtain the test results;

[0015] The multi-dimensional test parameters for the specific client user are calculated based on the test results.

[0016] Optionally, the multi-dimensional test parameters of the participating client users include test parameters related to target loss and target accuracy;

[0017] The target loss is the predicted loss value based on the first model;

[0018] The target accuracy is the classification accuracy of the first model or the second model on the test set.

[0019] Optionally, the method further includes: constructing the incentive allocation model based on the DDPG algorithm of deep reinforcement learning.

[0020] Optionally, the method further includes:

[0021] The incentive allocation ratio of each participating party's client users is used as the aggregation weight. The model parameters uploaded by each participating party's client users are weighted and summed to obtain the aggregate model.

[0022] The aggregation model is then distributed to the client users of each participating party for the next round of iterative training.

[0023] Optionally, the method further includes: performing performance testing on the aggregation model and updating the incentive allocation model based on the test results.

[0024] Optionally, the method further includes: the participating client user calculating the profit and loss status of the current round based on the reward, total loss and calculation cost of the current round.

[0025] A federated learning incentive device based on deep reinforcement learning, the device comprising:

[0026] The receiving module is used to obtain the model parameters uploaded by each participating client user after the current round of iteration training;

[0027] The testing module is used to perform iterative testing based on the model parameters to obtain multi-dimensional test parameters for each participating client user.

[0028] The incentive allocation module is used to input the multi-dimensional test parameters of all participating client users into a pre-built incentive allocation model to determine the cost ratio required for the current iteration and the incentive allocation ratio for each participating client user.

[0029] The calculation module is used to calculate the rewards for each participating client user based on the remaining total funds, the cost ratio required for the current round of iteration, and the incentive allocation ratio for each participating client user.

[0030] Optionally, the test module includes:

[0031] The aggregation unit is used to aggregate the model parameters of all participating client users to obtain a first aggregation parameter and generate a first model based on the first aggregation parameter; and to aggregate the model parameters after excluding specific client users to obtain a second aggregation parameter and generate a second model based on the second aggregation parameter.

[0032] The testing unit is used to perform iterative tests on the first model and the second model using a set of test sets, and obtain test results.

[0033] The calculation unit is used to calculate the multi-dimensional test parameters of the specific client user based on the test results.

[0034] Optionally, the device further includes:

[0035] The aggregation module is used to use the incentive allocation ratio of each participating client user as the aggregation weight, and to perform a weighted summation of the model parameters uploaded by each participating client user to obtain the aggregated model.

[0036] The publishing module is used to distribute the aggregation model to the client users of each participating party for the next round of iterative training.

[0037] Optionally, the apparatus further includes a model building module for building the incentive allocation model based on the DDPG algorithm of deep reinforcement learning.

[0038] Optionally, the apparatus further includes: an update module, used to perform performance testing on the aggregation model and update the incentive allocation model based on the test results.

[0039] A federated learning incentive system based on deep reinforcement learning, the system comprising: a central server and at least two clients; the central server comprising the federated learning incentive device based on deep reinforcement learning.

[0040] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to perform the steps of the federated learning incentive method based on deep reinforcement learning.

[0041] The federated learning incentive method, apparatus and system based on deep reinforcement learning provided by this invention organically integrate deep reinforcement learning with the federated incentive mechanism, breaking through the limitations of traditional static rules and centralized learning, and constructing a privacy-secure, dynamically adaptive incentive decision-making mechanism to achieve effective incentive allocation for the participants in federated learning.

[0042] This invention not only significantly enhances the participation enthusiasm of high-contribution participants and suppresses inefficient behavior, but also ensures the sustainability and model performance of the federated learning system from a mechanism perspective, providing key technical support for the construction of a large-scale, open federated ecosystem. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a federated learning incentive method based on deep reinforcement learning provided in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating the determination of multi-dimensional test parameters for each participating party's client user in this embodiment of the invention;

[0046] Figure 3 This is a schematic diagram of the incentive allocation model in one embodiment of the present invention;

[0047] Figure 4This is a schematic diagram of a federated learning incentive device based on deep reinforcement learning provided in an embodiment of the present invention;

[0048] Figure 5 This is another structural schematic diagram of the federated learning incentive device based on deep reinforcement learning provided in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of a federated learning incentive system based on deep reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0050] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] Deep Reinforcement Learning (DRL) offers a novel approach to designing incentive mechanisms in complex and dynamic environments due to its ability to autonomously learn optimal decision-making policies in high-dimensional state spaces. However, directly applying DRL to federated incentives faces two challenges: first, global rewards are sparse and delayed, making it difficult to effectively guide agent policy updates; second, how to handle the dynamic decision-making policies and interactions of clients regarding data contributions to achieve effective incentive allocation policy learning.

[0053] To this end, embodiments of the present invention provide a federated learning incentive method, apparatus, and system based on deep reinforcement learning. Based on the dynamic environment of federated learning, deep reinforcement learning is used to adaptively predict incentive shares for each participating client user, encouraging each participant to actively participate in the federated learning training process and improving the overall performance of the aggregation model.

[0054] like Figure 1 The diagram shown is a flowchart of a federated learning incentive method based on deep reinforcement learning provided in an embodiment of the present invention, which includes the following steps:

[0055] In step 101, obtain the model parameters uploaded by each participating client user after the current round of iteration training.

[0056] In step 102, iterative testing is performed based on the model parameters to obtain multi-dimensional test parameters for each participating client user.

[0057] The multi-dimensional test parameters include test parameters related to target loss and target accuracy. Specifically: the target loss is the predicted loss value based on the first model; the target accuracy is the classification accuracy based on the first model or the second model on the test set.

[0058] like Figure 2 The diagram shown is a flowchart for determining multi-dimensional test parameters for each participating party's client user in an embodiment of the present invention, including the following steps:

[0059] Step 201: Aggregate the model parameters of all participating client users to obtain the first aggregated parameters, and generate the first model based on the first aggregated parameters.

[0060] Assuming the client user k In the t The model parameters uploaded after the next iteration of training are: The number of client users participating in federated learning is K Then the aggregated model parameters on the server side, i.e., the first aggregated parameters, are: , w k t The aggregate weights can be represented as:

[0061] (1)

[0062] Based on the aggregated model parameters, the first model is obtained.

[0063] Step 202: Aggregate the model parameters after excluding specific client users to obtain second aggregated parameters, and generate a second model based on the second aggregated parameters.

[0064] Assuming from K Excluding client users from the client users i In the t The model parameters uploaded after the next iteration of training are: Then the aggregated model parameters on the server side, i.e., the second aggregated parameters, are: , can be represented as:

[0065] (2)

[0066] Based on the aggregated model parameters, a second model is obtained.

[0067] Step 203: Use the set test set to iteratively test the first model and the second model respectively to obtain the test results.

[0068] In the test set D test The performance of the first and second models described above was tested respectively to obtain the target loss. loss k t and target accuracy acc k t The values ​​of these two parameters can be used to determine the contribution of the model uploaded by the client user. ψ k ( K , v loss )and ψ k ( K , v acc ), can be represented as follows:

[0069] (3)

[0070] in, v loss and v acc These correspond to the target losses of the first and second models on the test set, respectively. loss ) and accuracy ( accuracy ). v loss ( K )represent K The target loss value on the test set after aggregating the model contributions from each participant. v loss ( K { k}) represents except for the first k The target loss value on the test set is the aggregated model contribution from participants other than the original participant. v acc ( K )represent K The accuracy of the model aggregated from the contributions of each participant on the test set. v acc ( K { k}) represents except for the first k The accuracy of the model on the test set after aggregating contributions from participants other than the original participant.

[0071] Step 204: Calculate the multi-dimensional test parameters of the specific client user based on the test results.

[0072] Based on the contribution from the two different perspectives mentioned above, multi-dimensional test parameters for reinforcement learning can be determined.

[0073] For example, in a non-limiting embodiment, a 20-dimensional vector can be used. The vector definition for each dimension is shown in Table 1 below for each client user.

[0074] Table 1

[0075]

[0076] It should be noted that the multi-dimensional test parameters shown in Table 1 above are only illustrative examples. In practical applications, some of them can be selected or other test parameters can be added. This embodiment of the invention does not limit this.

[0077] Continue to refer to Figure 1 In step 103, the multi-dimensional test parameters of all participating client users are input into the pre-built incentive allocation model to determine the cost ratio required for the current iteration and the incentive allocation ratio for each participating client user.

[0078] In the t After each round of iteration, it is necessary to predict the incentive allocation for each participating client user.

[0079] In practice, this can be achieved when the number of interactions reaches a preset maximum number of rounds, or when the remaining amount on the server side... Less than the fixed cost of training per iteration K × cost round (The fixed fees may include, for example, fees for model aggregation and model transmission, and it is assumed that these fees are proportional to the number of participating client users.) K When ), the current interaction ends.

[0080] Step 104: Calculate the rewards for each participating client user based on the remaining total funds, the cost ratio required for the current iteration, and the incentive allocation ratio for each participating client user.

[0081] For participating client users k The allocated incentive amount Satisfy the following formula:

[0082] (4)

[0083] in, Indicates the first t The total amount of funds remaining at the next iteration. βk t In the first t The assignment of the next iteration prediction to the user k The reward share, γ t Indicates the first t The cost ratio required for each iteration, i.e., the reward amount to be allocated to all participants and the total funds. The percentage.

[0084] The incentive allocation model can be constructed based on the DDPG algorithm of deep reinforcement learning. DDPG combines the ideas of deterministic policy gradient (DPG) and deep Q network (DQN), and learns an Actor (responsible for performing actions) and a Critic (responsible for evaluating actions) at the same time, and uses the techniques of DQN to stabilize training.

[0085] Reference Figure 3 , Figure 3 This diagram illustrates a structural schematic of the incentive allocation model in an embodiment of the present invention. The Deep Deterministic Policy Gradient (DDPG) model of deep reinforcement learning is an algorithm for continuous action spaces that combines an actor-critic framework with a deep neural network. Figure 3 In the structure shown, the Action Network takes the observed variables as input, i.e., the feature vectors calculated according to Table 1. It is mapped to 200 units through a linear layer, then to 100 units through another linear layer, finally outputting a continuous action vector. The Critic Network also takes the observed variables as input, first mapping them to 200 units, then concatenating the state features with the action vectors output by the Action Network, inputting them together into a 100-unit linear layer, and finally outputting a Q-score to evaluate the quality of the state-action pair. This design enables the Action Network to learn deterministic policies, while the Critic Network guides policy updates through Q-score feedback, thus achieving efficient learning and decision-making. In some embodiments, the server can also use the incentive allocation ratios of each participating client user as aggregation weights to perform a weighted summation of the model parameters uploaded by each participating client user, obtaining an aggregated model; the aggregated model is then distributed to each participating client user for the next round of iterative training.

[0086] In some embodiments, the server may also perform performance testing on the aggregation model and update the incentive allocation model based on the test results.

[0087] Specifically, each iteration can focus on two objectives simultaneously: reducing the loss after model aggregation and increasing the amount paid each time. Minimize, i.e., keep the remaining amount as low as possible. As much as possible. This can be achieved by multiplying the two objectives so that the reward value simultaneously considers both factors, satisfying the following condition:

[0088] (5)

[0089] in, Representing the t The reward value for the incentive allocation model in each iteration. loss G t Representing the t The target loss value of the next iteration aggregation model on the test set. Representing the server t The total amount of funds remaining in the next iteration. ema_loss G t represent loss G t Exponential moving average, T Indicates the size of the movable window.

[0090] Each iteration t In this process, the input feature vector of the incentive allocation model is calculated based on the feature dimensions in Table 1. This feature vector serves as the input to the incentive allocation model, which predicts the proportion of costs to be spent in this iteration and the incentive allocation for each participant. To obtain the reward function value, the incentive allocation model is iteratively trained using the deep reinforcement learning (DDPG) algorithm to update the parameters of the incentive allocation model.

[0091] Correspondingly, each participating client user can also determine the profit or loss status of the current round by calculating the reward, total loss, and computational cost of the current round.

[0092] The federated learning incentive method based on deep reinforcement learning provided by this invention organically integrates deep reinforcement learning with the federated incentive mechanism, and realizes dynamic and adaptive effective incentive allocation for each client user participating in federated learning, which ensures the interests of the participants, can greatly improve the enthusiasm of the participants, and thus improve the overall performance of the aggregation model.

[0093] Accordingly, embodiments of the present invention also provide a federated learning incentive device based on deep reinforcement learning, such as... Figure 4 The diagram shown is a structural schematic of the device.

[0094] The federated learning incentive device 400 based on deep reinforcement learning includes the following modules:

[0095] The receiving module 401 is used to obtain the model parameters uploaded by each participating client user after the current round of iteration training;

[0096] Test module 402 is used to perform iterative testing based on the model parameters to obtain multi-dimensional test parameters of each participating client user;

[0097] The incentive allocation module 403 is used to input the multi-dimensional test parameters of all participating client users into the pre-built incentive allocation model to determine the cost ratio required for the current iteration and the incentive allocation ratio of each participating client user.

[0098] The calculation module 404 is used to calculate the rewards for each participating client user based on the remaining total funds, the cost ratio required for the current round of iteration, and the incentive allocation ratio for each participating client user.

[0099] One non-limiting structure of the test module 402 may include the following units:

[0100] The aggregation unit is used to aggregate the model parameters of all participating client users to obtain a first aggregation parameter and generate a first model based on the first aggregation parameter; and to aggregate the model parameters after excluding specific client users to obtain a second aggregation parameter and generate a second model based on the second aggregation parameter.

[0101] The testing unit is used to perform iterative tests on the first model and the second model using a set of test sets, and obtain test results.

[0102] The calculation unit is used to calculate the multi-dimensional test parameters of the specific client user based on the test results. The calculation of the multi-dimensional test parameters can be referred to the description in the previous embodiments of the method of the present invention, and will not be repeated here.

[0103] The incentive allocation model can be constructed based on the DDPG algorithm of deep reinforcement learning. Specifically, it can be built by a corresponding model building module. The model building module can be part of the federated learning incentive device 400 based on deep reinforcement learning of the present invention, or it can be independent of the incentive device 400. The embodiments of the present invention do not limit this.

[0104] like Figure 5 The diagram shown is another structural schematic of the federated learning incentive device based on deep reinforcement learning provided in an embodiment of the present invention.

[0105] and Figure 4 The difference between the illustrated embodiment and the one shown is that, in Figure 5 In the illustrated embodiment, the excitation device 400 further includes an aggregation module 405 and a publishing module 406. Wherein:

[0106] The aggregation module 405 is used to use the incentive allocation ratio of each participating client user as the aggregation weight, and to perform weighted summation of the model parameters uploaded by each participating client user to obtain the aggregated model.

[0107] The publishing module 406 is used to distribute the aggregation model to the client users of each participating party for the next round of iterative training.

[0108] In some embodiments, the incentive device 400 may further include: an update module (not shown) for performing performance tests on the aggregation model and updating the incentive allocation model based on the test results.

[0109] Accordingly, embodiments of the present invention also provide a federated learning incentive system based on deep reinforcement learning, such as... Figure 6 The diagram shown is a structural schematic of the system.

[0110] The deep reinforcement learning-based federated learning incentive system includes a central server 50 and at least two clients 61. The central server 50 includes the aforementioned deep reinforcement learning-based federated learning incentive device 300.

[0111] In this embodiment, the central server 60 acts as the central node. After each round of federated learning, it receives model parameters uploaded by each client 61. It then performs iterative testing based on these model parameters to obtain multi-dimensional test parameters for each participating client user. During the iterative testing process, the model parameters trained by each participating client user need to be aggregated, the model evaluated, and the multi-dimensional test parameters for each participating client user determined. The specific test parameters used can be determined according to actual application needs; this embodiment of the invention does not limit this.

[0112] In addition, the central server 60 also needs to perform state management based on the model evaluation results of each participant and determine the rewards for each participant's client users.

[0113] The federated learning incentive device 300 based on deep reinforcement learning uses the DDPG algorithm based on deep reinforcement learning to construct an incentive allocation model, and uses this incentive allocation model to determine the rewards for each participating client user.

[0114] The federated learning incentive method, apparatus and system based on deep reinforcement learning provided in this invention organically integrate deep reinforcement learning with the federated incentive mechanism to achieve effective incentive allocation for participants in federated learning. This not only significantly improves the participation enthusiasm of high-contribution participants and suppresses inefficient behavior, but also ensures the sustainability and model performance of the federated learning system from a mechanism perspective, providing key technical support for the construction of a large-scale, open federated ecosystem.

[0115] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0116] The present invention also provides a storage medium, which is a computer-readable storage medium storing a computer program thereon, the computer program being executable when it is run. Figure 1 or Figure 2 The method shown may include some or all of the steps. The storage medium may include read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0117] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data provider to another website, computer, server, or data provider via wired or wireless means.

[0118] The embodiments of the present invention have been described in detail above. Specific implementation methods have been used to illustrate the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and apparatus of the present invention, and are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention, and the content of this specification should not be construed as a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A federated learning incentive method based on deep reinforcement learning, characterized in that, The method comprises: obtaining model parameters uploaded by each participant client user after current round iteration training; performing iterative testing according to the model parameters to obtain multi-dimensional test parameters of each participant client user, specifically comprising: aggregating model parameters of all participant client users to obtain first aggregation parameters and generate a first model based on the first aggregation parameters; aggregating model parameters excluding a specific client user to obtain second aggregation parameters and generate a second model based on the second aggregation parameters; performing iterative testing on the first model and the second model respectively using a set test set to obtain test results; and calculating multi-dimensional test parameters of the specific client user according to the test results; inputting multi-dimensional test parameters of all participant client users into a pre-constructed incentive distribution model to determine a required cost proportion of the current round iteration and an incentive distribution proportion of each participant client user; the incentive distribution model is constructed based on a DDPG deep reinforcement learning algorithm; calculating rewards of each participant client user according to a remaining total amount of funds, the required cost proportion of the current round iteration, and the incentive distribution proportion of each participant client user; At each iteration, two objectives are simultaneously considered, namely the loss of the aggregated model and the remaining amount of money To this end, the reward value is made to simultaneously take into account both factors by multiplying the two objectives, i.e. by satisfying the following condition: （5） wherein, represent the first t iteration reward value of the incentive allocation model, loss G t represent the first t iteration target loss value of the aggregation model on the test set, represent the first t iteration remaining total amount of funds on the server side; ema_loss G t represent the loss G t exponential moving average, T denotes the moving window size; The incentive distribution model is iteratively trained by a DDPG deep reinforcement learning algorithm to update parameters of the incentive distribution model.

2. The federated learning incentive method based on deep reinforcement learning according to claim 1, wherein, The multi-dimensional test parameters of the participant client user include test parameters related to a target loss and a target accuracy; The target loss is a predicted loss value based on the first model; The target accuracy is a classification accuracy of the first model or the second model on the test set.

3. The federated learning incentive method based on deep reinforcement learning according to claim 1, wherein, The method further comprises: weighting and summing model parameters uploaded by each participant client user by taking the incentive distribution proportion of each participant client user as an aggregation weight to obtain an aggregated model; distributing the aggregated model to each participant client user for next round iteration training.

4. The federated learning incentive method based on deep reinforcement learning according to claim 3, characterized in that, The method further comprises: performing performance testing on the aggregated model and updating the incentive distribution model according to a test result.

5. The federated learning incentive method based on deep reinforcement learning according to any one of claims 1 to 4, characterized in that, The method further comprises: The participant client user calculates a profit and loss state of the current round according to a reward of the current round, total loss, and calculation cost.

6. An apparatus for the federated learning incentive method based on deep reinforcement learning of claim 1, characterized in that, The device comprises: a receiving module configured to obtain model parameters uploaded by each participant client user after current round iteration training; a testing module configured to perform iterative testing according to the model parameters to obtain multi-dimensional test parameters of each participant client user; an incentive distribution module configured to input multi-dimensional test parameters of all participant client users into a pre-constructed incentive distribution model to determine a required cost proportion of the current round iteration and an incentive distribution proportion of each participant client user; an accounting module configured to calculate rewards of each participant client user according to a remaining total amount of funds, the required cost proportion of the current round iteration, and the incentive distribution proportion of each participant client user; The testing module comprises: aggregate model parameters of all participant client users to obtain first aggregated parameters, generate a first model based on the first aggregated parameters; aggregate model parameters excluding a specific client user to obtain second aggregated parameters, generate a second model based on the second aggregated parameters; a testing unit configured to perform iterative testing on the first model and the second model based on a set test set respectively, and obtain test results; a calculation unit configured to calculate multi-dimensional test parameters of the specific client user according to the test results.

7. The apparatus of claim 6, wherein, The device further comprises: an aggregation module configured to take the incentive allocation proportion of each participant client user as an aggregation weight, and perform weighted summation on model parameters uploaded by each participant client user to obtain an aggregated model; a publishing module configured to distribute the aggregated model to each participant client user for next round of iterative training.

8. The apparatus of claim 7, wherein, The device further comprises: an updating module configured to perform performance testing on the aggregated model, and update the incentive allocation model according to test results.

9. A federated learning incentive system based on deep reinforcement learning, characterized in that, The system comprises a central server and at least two clients; the central server comprises the device according to any one of claims 6 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, performs the steps of the federated learning incentive method based on deep reinforcement learning in any one of claims 1 to 5.

Images