Federal learning excitation method, device and system based on deep reinforcement learning

Abstract

The invention discloses a federated learning excitation method, device and system based on deep reinforcement learning. The method comprises the following steps: acquiring model parameters uploaded by each participant client user after the current round of iterative training; performing an iterative test according to the model parameters to obtain multi-dimensional test parameters of each participant client user; and inputting the multi-dimensional test parameters of the participant client users and the total cost which needs to be used for payment in proportion in the current round into a pre-constructed incentive distribution model, and determining rewards corresponding to the participant client users. By using the scheme of the invention, effective incentive distribution can be realized for participants of federal learning, and the participation enthusiasm of high-contribution clients is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed model training, in particular to a federated learning incentive method, device and system based on deep reinforcement learning. BACKGROUND

[0002] With the rapid development of Internet of Things, edge computing and mobile Internet technology, massive data is generated in real time on terminal devices. In order to protect user data privacy and reduce the communication overhead of centralized data transmission, federated learning (FL) emerges as the times require as a distributed machine learning paradigm. This mechanism allows multiple participating clients to collaboratively train a global model without sharing raw data, and the central server aggregates local model updates to achieve the privacy protection goal of "data not moving and model moving". However, the actual deployment of federated learning faces core challenges such as insufficient willingness to participate, ambiguous contribution evaluation and lack of incentive mechanism. Especially in an open and heterogeneous network environment, the differences in computing resources, data quality and communication capabilities of clients are significant, and rational participants tend to engage in "free riding" behavior, i.e., submitting low-quality or false model updates to avoid costs, which seriously damages the convergence efficiency and final performance of the global model.

[0003] Existing incentive mechanisms mostly follow the traditional game theory or auction theory framework, evaluating client contributions and allocating rewards based on pre-set rules. Such methods rely on strong assumptions such as known client types, static environment and perfect information, making it difficult to adapt to the complex characteristics of dynamic participation, Non-IID data and communication delays in federated learning. In addition, contribution evaluation often uses post-computation methods such as marginal contribution and Shapley value, which have high computational overhead and cannot be optimized online, resulting in a lag in incentive response and making it difficult to guide long-term high-quality participation. Although some research attempts to introduce reinforcement learning for dynamic incentive design, most of them are limited to centralized decision-making and do not fully consider the decentralized and privacy-sensitive nature of federated systems, which poses risks of model leakage and strategy homogenization. SUMMARY

[0004] The present application provides a federated learning incentive method, device and system based on deep reinforcement learning, which effectively incentivizes participants in federated learning and improves the participation enthusiasm of high-contribution clients.

[0005] To this end, the present application provides the following technical solutions: A federated learning incentive method based on deep reinforcement learning, the method comprising: obtaining model parameters uploaded by each participant client user after current round iteration training; performing iterative testing according to the model parameters to obtain multi-dimensional test parameters of each participant client user; inputting multidimensional test parameters of all participant client users into a pre-constructed incentive distribution model to determine a proportion of fees required for the current round iteration and a proportion of incentives of each participant client user to be distributed; According to the total amount of remaining funds, the proportion of fees required for the current round iteration, and the proportion of incentives of each participant client user to be distributed, the rewards of each participant client user are calculated.

[0006] Optionally, the iteration test according to the model parameters obtains the multidimensional test parameters of each participant client user, including: aggregating the model parameters of all participant client users to obtain first aggregated parameters, and generating a first model based on the first aggregated parameters; aggregating the model parameters excluding a specific client user to obtain second aggregated parameters, and generating a second model based on the second aggregated parameters; using a set test set to iteratively test the first model and the second model respectively to obtain test results; According to the test results, the multidimensional test parameters of the specific client user are calculated.

[0007] Optionally, the multidimensional test parameters of the participant client user include test parameters related to target loss and target accuracy; The target loss is a predicted loss value based on the first model; The target accuracy is the classification accuracy of the first model or the second model on the test set.

[0008] Optionally, the method further comprises: constructing the incentive distribution model based on a DDPG algorithm of deep reinforcement learning.

[0009] Optionally, the method further comprises: The proportion of incentives of each participant client user is used as an aggregation weight to perform weighted summation on the model parameters uploaded by each participant client user to obtain an aggregated model; The aggregated model is distributed to each participant client user for next round iteration training.

[0010] Optionally, the method further comprises: performance testing the aggregated model, and updating the incentive distribution model according to the test results.

[0011] Optionally, the method further comprises: the participant client user calculates the profit and loss state of the current round according to the reward of the current round, total loss, and calculation cost.

[0012] A federated learning incentive device based on deep reinforcement learning, the device comprising: a receiving module configured to acquire model parameters uploaded by each participant client user after the current round of iterative training; a testing module configured to perform iterative testing according to the model parameters to obtain multi-dimensional test parameters of each participant client user; an incentive distribution module configured to input the multi-dimensional test parameters of all participant client users into a pre-constructed incentive distribution model to determine a required cost proportion of the current round of iteration and an incentive distribution proportion of each participant client user; an accounting module configured to calculate rewards of each participant client user according to a total amount of remaining funds, the required cost proportion of the current round of iteration, and the incentive distribution proportion of each participant client user.

[0013] Optionally, the testing module comprises: an aggregation unit configured to aggregate the model parameters of all participant client users to obtain first aggregated parameters, generate a first model based on the first aggregated parameters, and aggregate the model parameters excluding a specific client user to obtain second aggregated parameters, generate a second model based on the second aggregated parameters; a testing unit configured to perform iterative testing on the first model and the second model based on a set test set respectively to obtain test results; a calculation unit configured to calculate multi-dimensional test parameters of the specific client user according to the test results.

[0014] Optionally, the apparatus further comprises: an aggregation module configured to weight-sum the model parameters uploaded by each participant client user by taking the incentive distribution proportion of each participant client user as an aggregation weight to obtain an aggregated model; a publishing module configured to distribute the aggregated model to each participant client user for next round of iterative training.

[0015] Optionally, the apparatus further comprises a model construction module configured to construct the incentive distribution model based on a DDPG algorithm of deep reinforcement learning.

[0016] Optionally, the apparatus further comprises an updating module configured to perform performance testing on the aggregated model, and update the incentive distribution model according to a test result.

[0017] A federated learning incentive system based on deep reinforcement learning, comprising a central server and at least two clients; the central server comprises the federated learning incentive apparatus based on deep reinforcement learning.

[0018] The application further provides a computer readable storage medium, which stores a computer program, and the computer program performs the steps of the federated learning incentive method based on deep reinforcement learning when executed by a processor.

[0019] The federated learning incentive method, device and system based on deep reinforcement learning provided by the application organically integrate deep reinforcement learning and federated incentive mechanism, break through the limitations of traditional static rules and centralized learning, and construct a privacy-safe and dynamically adaptive incentive decision mechanism, thereby achieving effective incentive distribution for participants in federated learning.

[0020] The application scheme can not only significantly improve the participation enthusiasm of high-contribution participants and suppress inefficient behavior, but also guarantee the sustainability and model performance of the federated learning system from the mechanism, thereby providing key technical support for the construction of a large-scale and open federated ecology. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described in the following are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0022] Figure 1 is a flowchart of the federated learning incentive method based on deep reinforcement learning provided by the embodiment of the application; Figure 2 is a flowchart of the determination of multi-dimensional test parameters of the client users of each participant in the embodiment of the application; Figure 3 is a structural schematic diagram of the incentive distribution model in the embodiment of the application; Figure 4 is a structural schematic diagram of the federated learning incentive device based on deep reinforcement learning provided by the embodiment of the application; Figure 5 is another structural schematic diagram of the federated learning incentive device based on deep reinforcement learning provided by the embodiment of the application; Figure 6 is a structural schematic diagram of the federated learning incentive system based on deep reinforcement learning provided by the embodiment of the application. DETAILED DESCRIPTION

[0023] The specific embodiments of the application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0024] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort should fall within the protection scope of the present application.

[0025] Deep Reinforcement Learning (DRL) provides a new idea for incentive mechanism design in complex dynamic environments due to its ability to autonomously learn optimal decision strategies in high-dimensional state spaces. However, directly applying deep reinforcement learning to federated incentives faces two challenges: first, global rewards are sparse and delayed, making it difficult to effectively guide agent policy updates; second, how to deal with the dynamic decision strategies and interactions of clients in terms of data contribution to achieve effective incentive allocation strategy learning.

[0026] To this end, the embodiments of the present application provide a federated learning incentive method, device and system based on deep reinforcement learning, which, based on the dynamic environment of federated learning, uses deep reinforcement learning to achieve adaptive incentive share prediction for each participant client user, encourages each participant to actively participate in the federated learning training process, and improves the overall performance of the aggregated model.

[0027] As shown in Figure 1 , it is a flowchart of a federated learning incentive method based on deep reinforcement learning provided by the embodiments of the present application, which includes the following steps: In step 101, the model parameters uploaded by each participant client user after the current round of iterative training are obtained.

[0028] In step 102, iterative testing is performed based on the model parameters to obtain multi-dimensional test parameters of each participant client user.

[0029] The multi-dimensional test parameters include test parameters related to target loss and target accuracy. The target loss is a predicted loss value based on the first model, and the target accuracy is the classification accuracy of the first model or the second model on the test set.

[0030] As shown in Figure 2 , it is a flowchart of determining the multi-dimensional test parameters of each participant client user in the embodiments of the present application, which includes the following steps: Step 201, aggregate the model parameters of all participant client users to obtain first aggregation parameters, and generate a first model based on the first aggregation parameters.

[0031] Suppose a client userk After the first iteration training, the model parameters uploaded by the client users are t , the number of client users participating in the federated learning is , and the model parameters aggregated at the server side, i.e., the first aggregation parameters, are K , w k t is the aggregation weight, which can be represented as: (1) Based on the aggregated model parameters, the first model is obtained.

[0032] In step 202, the model parameters after excluding the specific client users are aggregated to obtain the second aggregation parameters, and a second model based on the second aggregation parameters is generated.

[0033] Suppose client user K is excluded from i client users t After the first iteration training, the model parameters uploaded by the client users are , and the model parameters aggregated at the server side, i.e., the second aggregation parameters, are , which can be represented as: (2) Based on the aggregated model parameters, the second model is obtained.

[0034] In step 203, the first model and the second model are respectively iteratively tested using a set test set to obtain test results.

[0035] The performance of the first model and the second model is tested on the test set D test to obtain the values of the target loss loss k t and the target accuracy acc k t Based on these two parameters, the contribution of the client users uploading the model can be determined ψ k ( K , v loss ) and ψ k ( K , v acc ), which can be represented as: (3) wherein, v ​loss and v acc These correspond to the target losses of the first and second models on the test set, respectively. loss ) and accuracy ( accuracy ). v loss ( K )represent K The target loss value on the test set after aggregating the model contributions from each participant. v loss ( K { k}) represents except for the first k The target loss value on the test set is the aggregated model contribution from participants other than the original participant. v acc ( K )represent K The accuracy of the model aggregated from the contributions of each participant on the test set. v acc ( K { k}) represents except for the first k The accuracy of the model on the test set after aggregating contributions from participants other than the original participant.

[0036] Step 204: Calculate the multi-dimensional test parameters of the specific client user based on the test results.

[0037] Based on the contribution from the two different perspectives mentioned above, multi-dimensional test parameters for reinforcement learning can be determined.

[0038] For example, in a non-limiting embodiment, a 20-dimensional vector can be used. The vector definition for each dimension is shown in Table 1 below for each client user.

[0039] Table 1

[0040] It should be noted that the multi-dimensional test parameters shown in Table 1 above are only illustrative examples. In practical applications, some of them can be selected or other test parameters can be added. This embodiment of the invention does not limit this.

[0041] Continue to refer to Figure 1 In step 103, the multi-dimensional test parameters of all participating client users are input into the pre-built incentive allocation model to determine the cost ratio required for the current iteration and the incentive allocation ratio for each participating client user.

[0042] In the tAfter the round iteration, the incentive distribution prediction needs to be performed for each participant client user.

[0043] In a specific implementation, when the number of interactions reaches a preset maximum round, or the remaining amount of money on the server side is less than a fixed fee for each iteration training K cost round The current interaction ends when the fixed fee (which may include, for example, model aggregation, model transmission, etc., and is assumed to be proportional to the number of participant client users K ).

[0044] Step 104: Calculate the rewards of each participant client user according to the total remaining amount of money, the cost ratio required for the current round iteration, and the incentive distribution ratio of each participant client user.

[0045] For a participant client user k , the allocated incentive amount satisfies the following formula: (4) Wherein, represents the total remaining amount of money at the t th iteration, β k t is the reward share allocated to the user t predicted at the k th iteration, γ t represents the cost ratio required at the t th iteration, that is, the percentage of the total amount of money to the amount of money allocated as rewards to all participants.

[0046] The incentive distribution model can be constructed based on the DDPG algorithm of deep reinforcement learning. DDPG combines the ideas of deterministic policy gradient (DPG) and deep Q network (DQN), simultaneously learns an Actor (responsible for performing actions) and a Critic (responsible for evaluating actions), and borrows the skills of DQN to stabilize training.

[0047] Referring to Figure 3 , Figure 3 , a structure diagram of an incentive distribution model in an embodiment of the present application is shown. The DDPG (Deep Deterministic Policy Gradient) model of deep reinforcement learning is an algorithm for continuous action space, which combines the actor-critic framework and deep neural network. In Figure 3 ​​In the illustrated structure, the actor network takes the observation variable as input, i.e., the feature vector calculated according to Table 1, maps it to 200 units through a linear layer, and then maps it to 100 units through another linear layer, and finally outputs a continuous action vector; the critic network also takes the observation variable as input, first maps it to 200 units, and then connects the state features and the action vector output by the actor network to a 100-unit linear layer for common input, and finally outputs a Q value to evaluate the state-action pair. This design enables the actor network to learn a deterministic policy, while the critic network guides policy updates through Q value feedback, thereby achieving efficient learning and decision-making. In some embodiments, the server can also assign the incentive distribution ratio of each participant client user as an aggregation weight, weight-sum the model parameters uploaded by each participant client user, and obtain an aggregated model; and distribute the aggregated model to each participant client user for next round of iterative training.

[0048] In some embodiments, the server can also test the performance of the aggregated model and update the incentive distribution model according to the test results.

[0049] Specifically, each iteration can focus on two goals at the same time, i.e., the loss of the aggregated model is reduced, and the amount of money paid each time as little as possible, i.e., the remaining amount as much as possible. To this end, the reward value can be multiplied by two goals to take into account both factors, i.e., to satisfy the following conditions: (5) wherein, represents the reward value of the t th iteration for the incentive distribution model, loss G t represents the target loss value of the aggregated model on the test set in the t th iteration, represents the total amount of funds remaining on the server in the t th iteration. ema_loss G t represents the exponential moving average of loss G t , and T represents the moving window size.

[0050] Each iteration tIn the middle, the input feature vector of the incentive allocation model is calculated according to the characteristic dimensions of Table 1. The feature vector is used as the input of the incentive allocation model, which predicts the cost proportion that should be spent in this iteration, and the incentive allocation of each participant, so as to As the reward function value, the incentive allocation model is iteratively trained by the deep reinforcement learning DDPG algorithm to realize the parameter update of the incentive allocation model.

[0051] Correspondingly, each participant client user can also determine the profit and loss state of the current round by calculating the reward, total loss and calculation cost according to the current round.

[0052] The federated learning incentive method based on deep reinforcement learning provided by the application organically combines deep reinforcement learning with federated incentive mechanism, realizes dynamic and adaptive effective incentive allocation for each client user participating in federated learning, guarantees the interests of the participants, can greatly improve the enthusiasm of the participants, and further improves the comprehensive performance of the aggregated model.

[0053] Correspondingly, the embodiment of the application also provides a federated learning incentive device based on deep reinforcement learning, as shown in Figure 4 Fig. 1 is a structural schematic diagram of the device.

[0054] The federated learning incentive device based on deep reinforcement learning 400 comprises the following modules: The receiving module 401 is used to obtain the model parameters uploaded by each participant client user after the current round of iterative training; The testing module 402 is used to perform iterative testing according to the model parameters to obtain multi-dimensional testing parameters of each participant client user; The incentive allocation module 403 is used to input the multi-dimensional testing parameters of all participant client users into a pre-constructed incentive allocation model to determine the cost proportion required for the current round of iteration and the incentive allocation proportion of each participant client user; The accounting module 404 is used to calculate the rewards of each participant client user according to the total amount of remaining funds, the cost proportion required for the current round of iteration and the incentive allocation proportion of each participant client user.

[0055] Among them, a non-limiting structure of the testing module 402 can include the following units: The aggregation unit is used to aggregate the model parameters of all participant client users to obtain first aggregation parameters, generate a first model based on the first aggregation parameters; and aggregate the model parameters excluding a specific client user to obtain second aggregation parameters, generate a second model based on the second aggregation parameters; a test unit configured to perform iterative tests on the first model and the second model respectively using a set test, and obtain test results; a calculation unit configured to calculate multi-dimensional test parameters of the specific client user according to the test results. The calculation of the multi-dimensional test parameters can refer to the description in the foregoing method embodiments of the present application, and will not be described here again.

[0056] The incentive allocation model can be constructed based on the DDPG algorithm of deep reinforcement learning, and specifically, can be established by a corresponding model construction module. The model construction module can be part of the federated learning incentive device 400 based on deep reinforcement learning, or can be independent of the incentive device 400, and the embodiments of the present application are not limited in this regard.

[0057] As shown in Figure 5 , it is another structural schematic diagram of the federated learning incentive device based on deep reinforcement learning provided by the embodiments of the present application.

[0058] Unlike the embodiments shown in Figure 4 , in the embodiments shown in Figure 5 , the incentive device 400 further includes an aggregation module 405 and a publishing module 406. Wherein: The aggregation module 405 is configured to take the incentive allocation proportions of the client users of the participating parties as aggregation weights, and perform weighted summation on the model parameters uploaded by the client users of the participating parties, to obtain an aggregated model. The publishing module 406 is configured to distribute the aggregated model to the client users of the participating parties, for next round of iterative training.

[0059] In some embodiments, the incentive device 400 can further include an updating module (not shown in the figure), configured to perform performance test on the aggregated model, and update the incentive allocation model according to the test results.

[0060] Correspondingly, the embodiments of the present application further provide a federated learning incentive system based on deep reinforcement learning, as shown in Figure 6 , it is a structural schematic diagram of the system.

[0061] The federated learning incentive system based on deep reinforcement learning includes a central server 60 and at least two clients 61. The central server 50 includes the above-mentioned federated learning incentive device 300 based on deep reinforcement learning.

[0062] In this embodiment, the central server 60 acts as a central node, receives the model parameters uploaded by each client 61 after each round of federated learning is completed, and performs iterative testing according to the model parameters to obtain multi-dimensional test parameters of each participant client user. During the iterative testing process, the model parameters trained by each participant client user need to be aggregated, the model needs to be evaluated, and the multi-dimensional test parameters of each participant client user are determined. The specific test parameters used can be determined according to actual application needs, and the embodiments of the present application do not limit this.

[0063] In addition, the central server 60 also needs to manage the state based on the model evaluation results of each participant to determine the rewards of each participant client user.

[0064] The federated learning incentive device 300 based on deep reinforcement learning adopts a DDPG algorithm based on deep reinforcement learning to construct an incentive distribution model, and uses the incentive distribution model to determine the rewards of each participant client user.

[0065] The federated learning incentive method, device and system based on deep reinforcement learning provided by the embodiments of the present application organically integrate deep reinforcement learning and federated incentive mechanism, realize effective incentive distribution of participants in federated learning, not only can significantly improve the participation enthusiasm of high-contribution participants and suppress low-efficiency behavior, but also guarantee the sustainability and model performance of the federated learning system from the mechanism, and provide key technical support for the construction of large-scale and open federated ecology.

[0066] It should be noted that, for each of the foregoing method embodiments, in order to simply describe, each is described as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0067] The present application also provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium. The computer program can execute part or all steps of the method shown in Figure 1 or Figure 2 when running. The storage medium can include read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. The storage medium can also include non-volatile memory or non-transitory memory, etc.

[0068] The above-described embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the above-described embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data provider to another website, computer, server or data provider through wired or wireless means.

[0069] The above describes the embodiments of the present application in detail, and the present application is described by using specific implementation manners. The above description of the embodiments is only used to help understand the method and device of the present application, and only represents a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application, and the content of the specification should not be understood as the limitation of the present application. Therefore, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A federated learning incentive method based on deep reinforcement learning, characterized in that, The method includes: Obtain the model parameters uploaded by each participating client user after the current round of iteration training; Iterative testing is performed based on the model parameters to obtain multi-dimensional test parameters for each participating client user. Specifically, this includes: aggregating the model parameters of all participating client users to obtain a first aggregated parameter, and generating a first model based on the first aggregated parameter; aggregating the model parameters after excluding a specific client user to obtain a second aggregated parameter, and generating a second model based on the second aggregated parameter; iteratively testing the first model and the second model using a set of test data to obtain test results; and calculating the multi-dimensional test parameters for the specific client user based on the test results. Input the multi-dimensional test parameters of all participating client users into the pre-built incentive allocation model to determine the cost ratio required for the current iteration and the incentive allocation ratio for each participating client user; The rewards for each participating client user are calculated based on the remaining total funds, the cost ratio required for the current iteration, and the incentive allocation ratio for each participating client user.

2. The federated learning incentive method based on deep reinforcement learning according to claim 1, characterized in that, The multi-dimensional test parameters for the participating client users include test parameters related to target loss and target accuracy; The target loss is the predicted loss value based on the first model; The target accuracy is the classification accuracy of the first model or the second model on the test set.

3. The federated learning incentive method based on deep reinforcement learning according to claim 1, characterized in that, The method further includes: The incentive allocation model is constructed based on the DDPG algorithm of deep reinforcement learning.

4. The federated learning incentive method based on deep reinforcement learning according to claim 1, characterized in that, The method further includes: The incentive allocation ratio of each participating party's client users is used as the aggregation weight. The model parameters uploaded by each participating party's client users are weighted and summed to obtain the aggregate model. The aggregation model is then distributed to the client users of each participating party for the next round of iterative training.

5. The federated learning incentive method based on deep reinforcement learning according to claim 4, characterized in that, The method further includes: The aggregation model is subjected to performance testing, and the incentive allocation model is updated based on the test results.

6. The federated learning incentive method based on deep reinforcement learning according to any one of claims 1 to 5, characterized in that, The method further includes: The participating client users calculate the profit and loss status of the current round based on the current round's rewards, total losses, and computational costs.

7. A federated learning incentive device based on deep reinforcement learning, characterized in that, The device includes: The receiving module is used to obtain the model parameters uploaded by each participating client user after the current round of iteration training; The testing module is used to perform iterative testing based on the model parameters to obtain multi-dimensional test parameters for each participating client user. The incentive allocation module is used to input the multi-dimensional test parameters of all participating client users into a pre-built incentive allocation model to determine the cost ratio required for the current iteration and the incentive allocation ratio for each participating client user. The accounting module is used to calculate the reward for each participating client user based on the remaining total funds, the cost ratio required for the current round of iteration, and the incentive allocation ratio for each participating client user. The testing module includes: The aggregation unit is used to aggregate the model parameters of all participating client users to obtain a first aggregation parameter and generate a first model based on the first aggregation parameter; and to aggregate the model parameters after excluding specific client users to obtain a second aggregation parameter and generate a second model based on the second aggregation parameter. The testing unit is used to perform iterative tests on the first model and the second model using a set of test sets, and obtain test results. The calculation unit is used to calculate the multi-dimensional test parameters of the specific client user based on the test results.

8. The federated learning incentive device based on deep reinforcement learning according to claim 7, characterized in that, The device further includes: The aggregation module is used to use the incentive allocation ratio of each participating client user as the aggregation weight, and to perform a weighted summation of the model parameters uploaded by each participating client user to obtain the aggregated model. The publishing module is used to distribute the aggregation model to the client users of each participating party for the next round of iterative training.

9. The federated learning incentive device based on deep reinforcement learning according to claim 8, characterized in that, The device further includes: The model building module is used to build the incentive allocation model based on the DDPG algorithm of deep reinforcement learning.

10. The federated learning incentive device based on deep reinforcement learning according to claim 9, characterized in that, The device further includes: An update module is used to perform performance testing on the aggregation model and update the incentive allocation model based on the test results.

11. A federated learning incentive system based on deep reinforcement learning, characterized in that, The system includes: a central server and at least two clients; the central server includes a federated learning incentive device based on deep reinforcement learning as described in any one of claims 7 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the federated learning incentive method based on deep reinforcement learning as described in any one of claims 1 to 6.

Images