User bidding method and device based on multi-agent reinforcement learning algorithm under federated learning

Abstract

The present invention discloses a user bidding method and device based on a multi-agent reinforcement learning algorithm in federated learning. The method includes: obtaining a learning task issued by a federated learning platform, a sample client uploading bidding information to the federated platform using a reinforcement learning algorithm, the platform selecting a sample client through an algorithm and then issuing a global shared model to the selected sample client, the selected sample client performing local training and uploading updated parameters, the platform aggregating the uploaded updated model parameters according to an aggregation algorithm and updating the model parameters in the global model. In order to complete the learning task issued by the federated learning platform, this method not only realizes dynamic bidding for users participating in federated learning but also alleviates model overfitting, solving the problem of lack of fairness in federated learning and model overfitting caused by the fact that after a user submits a bidding strategy, the user's bidding strategy does not change during the subsequent training process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a user bidding method and device based on a multi-agent reinforcement learning algorithm under federated learning. Background Art

[0002] With increasing user awareness of privacy and the introduction of relevant policies, traditional machine learning has become increasingly difficult to collect and centralize data for training. Federated learning, which protects user privacy by eliminating the need for users to upload raw data, has become a promising deep learning paradigm. However, the training process consumes significant computational and communication resources, meaning that selfish participants cannot fully participate in the learning task without sufficient rewards. Furthermore, due to the complex underlying network structure of federated learning and the limited and heterogeneous nature of node resources, the lack of appropriate incentives for federation initiators can lead to significant communication overhead. These issues not only waste network resources but also hinder the widespread adoption of federated learning.

[0003] Incentive mechanisms in related technologies can employ game theory to select participating users and distribute profits. This can be achieved by incorporating auctions into federated learning. As one approach, lightweight and multi-dimensional incentive schemes can be used to select high-quality participating users. Another approach is to establish an incentive mechanism framework that integrates the learning quality of participating users into federated learning to implement quality-conscious incentive mechanisms and model aggregation. However, existing auction-based incentive mechanisms are almost always static. During the auction process, these methods assume that participating users, once they have determined their strategies, will not change them as platform behavior changes. This approach only maximizes the utility of the platform or social welfare, but fails to maximize the utility of both the platform and participating users. Specifically, during the federated learning auction process, once a participating user confirms their bid, the strategy remains unchanged during subsequent training. Regardless of whether they are selected or not, the participating user simply waits to be selected. In such an auction mechanism, even if a participating user does not win the bid, they cannot change their existing strategy. This results in resource-rich participants being consistently selected, while resource-poor but honest participants are not. This not only leads to a lack of fairness in federated learning and hinders the enthusiasm of participating users, but also reduces data diversity due to the constant selection of specific clients, potentially leading to overfitting of the model. Furthermore, some dynamic bidding methods assume that user information is transparent, meaning that each user knows the private information of other users, which is impossible in practical applications. Summary of the Invention

[0004] This invention provides a user bidding method and device in federated learning based on a multi-agent reinforcement learning algorithm. By incorporating multi-agent reinforcement learning into the incentive mechanism of federated learning, this method solves the problem of the existing auction-based incentive mechanism, which lacks fairness in federated learning due to the fact that the strategy does not change during subsequent training. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a user bidding method based on a multi-agent reinforcement learning algorithm in federated learning, the method comprising:

[0006] Obtaining learning tasks published by the federated learning platform, selecting sample clients from the client set based on the learning tasks and bidding information uploaded by the client set participating in the federated learning, and issuing a global shared model to the sample clients;

[0007] Receive updated model parameters uploaded by each sample client, where the updated model parameters are generated by the sample client outputting the bid information to be submitted in the current round using a multi-agent reinforcement learning algorithm before training begins, and then training the global shared model according to the configuration in the bid information to be submitted after being selected;

[0008] Aggregating the updated model parameters uploaded by each sample client, and using the aggregated updated model parameters to update the model parameters in the global shared model;

[0009] If the updated global shared model reaches the preset model accuracy in the test task, the learning task released by the federated learning platform is considered completed. Otherwise, the steps of updating the model parameters in the global shared model are repeated for multiple rounds so that the updated global shared model reaches the preset model accuracy in the test task.

[0010] Optionally, the process of the sample client using a multi-agent reinforcement learning algorithm to output the bidding information to be submitted by the sample client in the current round includes:

[0011] The sample client is used as an intelligent agent. The intelligent agent observes its own historical status information in the federated learning environment, and uses the historical status information to output the bidding information to be submitted by the sample client in the current round.

[0012] Optionally, the multi-agent reinforcement learning algorithm includes a strategy controller and an experience pool. The sample client is used as an agent, the agent observes its own historical state information in the federated learning environment, and uses the historical state information to output the bidding information to be submitted by the sample client in the current round, including:

[0013] Using the sample client as an agent, the experience pool in the multi-agent reinforcement learning algorithm is used to store historical task status information observed by each agent in the federated learning environment, where the historical task status information includes at least whether the agent was selected in historical rounds, historical resource values, historical data provided amounts, and historical unit resource amounts;

[0014] By inputting the historical task status information observed by the agent in the federated learning environment as the status information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm, the bidding information to be submitted by the agent in the current round is output.

[0015] Optionally, after inputting historical task status information observed by the agent in the federated learning environment as the agent's status information in the current round into the strategy device in the multi-agent reinforcement learning algorithm and outputting the agent's bid information to be submitted in the current round, the method further includes:

[0016] Calculate the revenue resources fed back by the federated learning environment to the agent in the current round, and use the experience pool in the multi-agent reinforcement learning algorithm to store the historical state of the environment observed by the agent in the current round, the bid information to be submitted, the state of the environment after the bid information to be submitted is uploaded, and the revenue resources fed back to the agent by the federated learning environment for the bid information to be submitted uploaded in the current round.

[0017] Optionally, the calculation of the revenue resources fed back by the agent in the current round in the federated learning environment includes:

[0018] Based on the bidding information to be uploaded by the agent in the current round, the resource parameters involved in the bidding process of the agent are obtained respectively;

[0019] The resource parameters involved in the bidding process of the agent are input into the pre-built profit function to obtain the profit resources fed back by the federated learning environment to the agent in the current round.

[0020] Optionally, each sample client is configured with a policy controller, which includes an action network and a value network. The policy controller inputs historical task status information observed in the federated learning environment as the agent's state information in the current round into the multi-agent reinforcement learning algorithm, and outputs the agent's bid information to be submitted in the current round, including:

[0021] By inputting historical task status information observed by the agent in the federated learning environment as the state information of the agent in the current round into the action network in the strategy device, outputting the bidding information to be submitted by the agent in the current round, and obtaining the bidding information to be uploaded by the agent in the current training round;

[0022] By inputting the state information of the agent in the current round and the bidding information to be uploaded by the agent in the current round into the value network in the strategist, the bidding information to be uploaded is evaluated to obtain an evaluation score of the bidding information to be uploaded;

[0023] Among them, the action network is trained using the evaluation score of the bidding information to be uploaded, and the network parameters of the action network are updated by gradient ascent. The value network is trained using the evaluation score of the bidding information to be uploaded and the income resources actually fed back by the intelligent agent, and the network parameters of the value network are updated by the temporal difference method.

[0024] Optionally, aggregating the updated model parameters uploaded by each sample client and using the aggregated updated model parameters to update the model parameters in the global shared model includes:

[0025] Calculate the ratio of the data volume of each sample client to the data volume of all sample clients to obtain the data volume proportion corresponding to each sample client;

[0026] After multiplying the data volume proportion corresponding to each sample client by the updated model parameters uploaded by the corresponding sample client, the updated model parameters corresponding to all sample clients are aggregated, and the model parameters in the global shared model are updated by accumulating the aggregated updated model parameters.

[0027] In a second aspect, an embodiment of the present invention provides a user bidding device based on a multi-agent reinforcement learning algorithm in federated learning, the device comprising:

[0028] An acquisition unit, configured to acquire a learning task published by a federated learning platform, select sample clients from the client set based on the learning task and bidding information uploaded by the client set participating in the federated learning, and issue a global shared model to the sample clients;

[0029] A receiving unit is configured to receive updated model parameters uploaded by each sample client, wherein the updated model parameters are formed by the sample client outputting the bid information to be submitted in the current round using a multi-agent reinforcement learning algorithm before the start of training, and training the global shared model according to the configuration in the bid information to be submitted after the sample client is selected;

[0030] an aggregation unit, configured to aggregate the updated model parameters uploaded by each sample client, and use the aggregated updated model parameters to update the model parameters in the global shared model;

[0031] The selection unit is used to determine that the learning task released by the federated learning platform is completed if the updated global shared model reaches the preset model accuracy in the test task; otherwise, repeatedly execute the steps of updating the model parameters in the global shared model for multiple rounds so that the updated global shared model reaches the preset model accuracy in the test task.

[0032] Optionally, the device further includes:

[0033] The output unit is used for the sample client to output the bidding information to be submitted by the sample client in the current round using the multi-agent reinforcement learning algorithm;

[0034] The output unit is specifically used to use the sample client as an intelligent agent, the intelligent agent observes its own historical status information in the federated learning environment, and uses the historical status information to output the bidding information to be submitted by the sample client in the current round.

[0035] Optionally, the multi-agent reinforcement learning algorithm includes a strategy device and an experience pool, and the output unit includes:

[0036] A storage module is configured to use the sample client as an agent and the experience pool in the multi-agent reinforcement learning algorithm to store historical task status information observed by each agent in the federated learning environment, wherein the historical task status information includes at least whether the agent was selected in historical rounds, historical resource values, historical data provided amounts, and historical unit resource amounts;

[0037] The output module is used to input the historical task status information observed by the agent in the federated learning environment as the status information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm, and output the bidding information to be submitted by the agent in the current round.

[0038] Optionally, the output unit further includes:

[0039] A computing module is configured to calculate the revenue resources fed back by the federated learning environment to the agent in the current round after inputting the historical task status information observed by the agent in the federated learning environment as the status information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm and outputting the bidding information to be submitted by the agent in the current round, and use the experience pool in the multi-agent reinforcement learning algorithm to store the historical status of the environment observed by the agent in the current round, the bidding information to be submitted, the status of the environment after the bidding information to be submitted is uploaded, and the revenue resources fed back to the agent by the federated learning environment for the bidding information to be submitted uploaded in the current round.

[0040] Optionally, the calculation module is specifically configured to obtain resource parameters involved in the bidding process of the agent based on the bidding information to be uploaded by the agent in the current round;

[0041] The calculation module is specifically used to input the resource parameters involved in the bidding process of the agent into a pre-built profit function to obtain the profit resources fed back by the federated learning environment to the agent in the current round.

[0042] Optionally, each sample client is configured with a policy controller, the policy controller including an action network and a value network, and the output module is specifically configured to input historical task status information observed by the agent in the federated learning environment as the state information of the agent in the current round into the action network of the policy controller, output the bidding information to be submitted by the agent in the current round, and obtain the bidding information to be uploaded by the agent in the current training round;

[0043] The output module is further configured to input the state information of the agent in the current round and the bidding information to be uploaded by the agent in the current round into the value network of the strategist, evaluate the bidding information to be uploaded, and obtain an evaluation score for the bidding information to be uploaded;

[0044] Among them, the action network is trained using the evaluation score of the bidding information to be uploaded, and the network parameters of the action network are updated by gradient ascent. The value network is trained using the evaluation score of the bidding information to be uploaded and the income resources actually fed back by the intelligent agent, and the network parameters of the value network are updated by the temporal difference method.

[0045] Optionally, the polymerization unit includes:

[0046] A calculation module is used to calculate the ratio of the data volume of each sample client to the data volume of all sample clients, and obtain the data volume proportion corresponding to each sample client;

[0047] The aggregation module is used to multiply the data volume proportion corresponding to each sample client by the updated model parameters uploaded by the corresponding sample client, aggregate the updated model parameters corresponding to all sample clients, and update the model parameters in the global shared model by accumulating the updated model parameters after aggregation.

[0048] In a third aspect, an embodiment of the present invention provides a storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method described in the first aspect.

[0049] In a fourth aspect, an embodiment of the present invention provides a device for user bidding in federated learning based on a multi-agent reinforcement learning algorithm, including:

[0050] one or more processors;

[0051] a storage device for storing one or more programs,

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0053] As can be seen from the above content, the user bidding method and device based on the multi-agent reinforcement learning algorithm in federated learning provided by the embodiment of the present invention obtains the learning task issued by the federated learning platform, selects sample clients from the client set based on the learning task and the bidding information uploaded by the client set participating in the federated learning, and sends a global shared model to the sample clients. The updated model parameters uploaded by each sample client are received. The updated model parameters are the sample client's bid information to be submitted in the current round output by the sample client using the multi-agent reinforcement learning algorithm before the start of training. After being selected, the global shared model is trained according to the configuration in the bid information to be submitted. The updated model parameters uploaded by each sample client are further aggregated, and the model parameters in the global shared model are updated using the aggregated updated model parameters. If the updated global shared model reaches the preset model accuracy in the test task, it is determined that the learning task issued by the federated learning platform is completed. Otherwise, the step of updating the model parameters in the global shared model is repeated for multiple rounds so that the updated global shared model reaches the preset model accuracy in the test task. It can be seen from this that compared with the auction-based incentive mechanism in the prior art, the embodiment of the present invention can use a multi-agent learning system to adjust the bidding information uploaded by the client, thereby solving the problem of lack of fairness in federated learning caused by the auction-based incentive mechanism in the prior art because the strategy will not change during subsequent training.

[0054] In addition, the technical effects that can be achieved by this embodiment include:

[0055] (1) Based on the multi-agent reinforcement learning algorithm, the bidding information uploaded by the client is adjusted to increase the probability of the client being selected, ensure the fairness of the participating users in federated learning, solve the suboptimal problem caused by fixed strategies, and achieve the goal of maximizing the utility of the federated learning platform and participating users.

[0056] (2) The multi-agent reinforcement learning algorithm adopts centralized training and distributed execution, so that the corresponding clients of participating users can observe more states and improve the stability of the agent training process.

[0057] (3) The multi-agent reinforcement learning algorithm adopts an asynchronous deep reinforcement training method, decoupling the execution of learning tasks and the updating of bidding information, so that the two can work in parallel and accelerate the training of the model.

[0058] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely some embodiments of the present invention. Those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0060] Figure 1 A flowchart of a user bidding method based on a multi-agent reinforcement learning algorithm in federated learning provided by an embodiment of the present invention;

[0061] Figure 2 A flowchart of the multi-agent reinforcement learning algorithm provided in an embodiment of the present invention outputting bidding information to be submitted by a sample client in the current round;

[0062] Figure 3 A block diagram of the composition of a user bidding device based on a multi-agent reinforcement learning algorithm under federated learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0065] The present invention provides a user bidding method and device in federated learning based on a multi-agent reinforcement learning algorithm. By using a multi-agent learning system to adjust the bidding information uploaded by the client, the method solves the problem of the lack of fairness in federated learning caused by the auction-based incentive mechanism in the existing technology, as the strategy does not change during subsequent training. Traditional auction technology requires the private information of participating users during the auction process. The entire auction process is static, that is, the bids of participating users are fixed. Even after the bid fails, the bidding information uploaded by the client is not adjusted. This makes it impossible for participating users to dynamically change their bidding information, resulting in a lack of fairness in federated learning. Participating users with limited resources are difficult to be selected by the federated learning platform, resulting in a significant waste of participating users' resources. These auction mechanisms only maximize the utility of the federated learning platform or social welfare, but fail to maximize the utility of the federated learning platform and participating users together. The embodiment of the present invention introduces a multi-agent reinforcement learning algorithm into the incentive mechanism of federated learning. Based on the multi-agent reinforcement learning algorithm, the bidding information uploaded by the client is adjusted to increase the probability of the participating user's corresponding client being selected, reduce the aggregation time, ensure the fairness of participating users in federated learning, solve the suboptimal problem caused by fixed strategies, and achieve the goal of maximizing the utility of the federated learning platform and participating users.

[0066] The embodiments of the present invention are described in detail below.

[0067] Figure 1 A flowchart of a user bidding method based on a multi-agent reinforcement learning algorithm in federated learning is provided in an embodiment of the present invention. The method may include the following steps:

[0068] S100: Obtain a learning task published by a federated learning platform, select sample clients from the client set based on the learning task and bidding information uploaded by the client set participating in the federated learning, and issue a global shared model to the sample clients.

[0069] Learning tasks published on the federated learning platform are issued by the server corresponding to the federated publisher. These learning tasks are applicable to various application scenarios involving data collection and training, such as target recognition and data classification. Since the federated learning process requires the selection of clients with data to train a global shared model, selecting high-quality clients to update the model parameters of the global shared model will improve the application effect of the learning tasks. To ensure that the server selects the appropriate client for model training, each client will upload bidding information, and the server will further select each client based on the learning task.

[0070] The bidding information here consists of computing resources, data volume, and bidding resources. Specifically, in the process of selecting sample clients from the client set based on the learning task and the bidding information uploaded by the client set participating in federated learning, after the federated learning platform receives the client's bidding information, it can obtain the sample client and the corresponding bidding resources by modeling and solving the learning task. The modeling and solving process is as follows:

[0071]

[0072] s n ∈{0,1} (2)

[0073] t n max ≤T max (3)

[0074] Among them, (1) represents that the total payment of the federated learning platform to the selected clients must not exceed the platform's budget, (2) represents whether each client is selected or not, where selected is 1 and unselected is 0, and (3) represents that the training time of the selected client cannot exceed the maximum time specified by the federated learning platform.

[0075] By solving the above expression, the federated learning platform will select a set of sample clients and the amount of resources purchased by the federated learning platform for each sample client.

[0076] S110: Receive updated model parameters uploaded by each sample client, where the updated model parameters are formed by the sample client using a multi-agent reinforcement learning algorithm to output the bidding information to be submitted in the current round before the training starts, and after being selected, the global shared model is trained according to the configuration in the bidding information to be submitted.

[0077] In an embodiment of the present invention, the sample client is a client that wishes to participate in the learning task. Only the selected sample client can download and train the global model. Each sample client has a multi-agent reinforcement learning algorithm. Specifically, after the selected sample client receives the global shared model, it trains the global model according to the configuration in its bidding information to obtain updated model parameters.

[0078] The specific sample client uses the multi-agent reinforcement learning algorithm to output the bidding information to be submitted by the sample client in the current round. The sample client is used as the agent. The agent observes its own historical status information in the federated learning environment and uses the historical status information to output the bidding information to be submitted by the sample client in the current round.

[0079] The above-mentioned multi-agent reinforcement learning algorithm includes a strategist and an experience pool. Specifically, the sample client is used as the agent. The agent observes its own historical status information in the federated learning environment, and uses the historical status information to output the sample client's bid information to be submitted in the current round. In the process, the sample client can be used as the agent, and the experience pool in the multi-agent reinforcement learning algorithm can be used to store the historical task status information observed by each agent in the federated learning environment. The historical task status information is equivalent to the status of the agent's historical submitted bid information and the feedback of the federated learning on the agent's historical submitted bid information, at least including whether the agent was selected in the historical round, the historical resource value, the historical amount of data provided, and the historical unit resource amount. Further, the historical task status information observed by the agent in the federated learning environment is input as the state information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm, and the bid information to be submitted by the agent in the current round is output.

[0080] It is understandable that the above multi-agent reinforcement learning algorithm can learn how to map the task state in the federated learning environment to the bidding information so that the client and the platform can obtain the maximum resource benefits at the same time. The basic model of this system is the Markov game process. In the Markov game, all agents simultaneously select and execute the bidding information to be submitted by each agent based on the current task state (or observation value) of the federated learning environment. It is defined as a tuple (n, S, A1, ..., A n ,T,γ,R1,...,R n ), where n is the number of agents, S is the task state of the multi-agent reinforcement learning algorithm, which refers to the historical task state information of each agent; A is the set of bidding information to be submitted by each agent; T: S × A1 × A2 × ... × A n ×S→[0,1] is the set of agent state transition functions, that is, the probability distribution of the next task state given the current task state and joint action. i :S×A1×A2×...×A n ×S→[0,1] is the set of agent i’s reward functions, R i (s,a1,...×a n ,s) is the joint action (a1,...a n ) after that in task state s t+1 The expected cumulative reward obtained by an agent i can be expressed as:

[0081]

[0082] The reward function of the above agent can be expressed as:

[0083] grid;d i is the amount of data of agent i; m i is the unit computing power of the agent, c i is the unit cost; is the average profit obtained by agent i based on its own resource requirements, x i The agent uses resources to serve its own needs. Due to the uncertainty of the agent owner's behavior, for example, the agent may use it for other things for a long time, resulting in almost no remaining resources for task training. i Defined as a random variable in a certain interval Among them, x i Following the probability distribution function F(x i ).

[0084] Furthermore, to understand the revenue feedback from sample clients in each round in real time, after outputting the agent's pending bid information for the current round, the federated learning environment's revenue resources for the agent's feedback in the current round can be calculated. The experience pool in the multi-agent reinforcement learning algorithm can be used to store the agent's historical observations of the environment in the current round, the pending bid information, the state of the environment after the pending bid information was uploaded, and the revenue resources fed back to the agent by the federated learning environment for the pending bid information uploaded in the current round. Specifically, based on the agent's pending bid information in the current round, the resource parameters involved in the agent's bidding process can be obtained. These resource parameters involved in the agent's bidding process can then be input into a pre-constructed revenue function to obtain the revenue resources fed back to the agent by the federated learning environment in the current round.

[0085] Each of the above sample clients is configured with a policy controller, which includes an action network and a value network. Specifically, when outputting the agent's pending bid information for the current round, the policy controller can input the agent's historical task status information observed in the federated learning environment as its current round status information into the policy controller's action network, outputting the agent's pending bid information for the current round and obtaining the agent's pending bid information for the current training round. The policy controller then inputs the agent's current round status information and the agent's pending bid information for the current round into the value network, evaluating the pending bid information and obtaining an evaluation score for the pending bid information. The action network is trained using the evaluation score of the pending bid information, and its network parameters are updated using gradient ascent. The value network is trained using the evaluation score of the pending bid information and the actual revenue resources reported by the agent, and its network parameters are updated using the temporal difference method.

[0086] Specifically in the actual application scenario, it can be assumed that at a certain time step t, there are m sample clients and a task initiator in a certain area. Here, a certain time step t is equivalent to a round in the federated learning process in which the client set submits a task bid, the federated learning platform selects a client, and the selected sample client trains locally and uploads the updated model parameters. The task bid includes the bidding information of the sample client (data volume, computing resources) and the payment expected to be obtained. In federated learning, each sample client acts as an agent and has a reinforcement learning policy. The reinforcement learning policy is composed of a multi-layer perceptron in deep learning, which includes an input layer, a hidden layer, and an output layer. The policy is expressed as follows: st is the state of the federated learning environment at time t, including the state of each agent and the state of the federated learning platform. In each time slot t, the observation space of agent i is in, The price provided by the agent in the previous round, is the bidding result of the previous round i∈{0,1}, s=0 means the bidding failed, s=1 means the bidding succeeded; Represents the single computing resource provided by the agent. Since the agent may not allocate all computing resources to the training task during the training time, the unit computing resource of each agent is related to the resource requirements of the agent itself; Indicates the amount of data provided by the agent in the previous round. Before the start of the current training round, agent i observes the previous state information about the current learning task, and then inputs the state learning observed from the federated learning environment into the action network. The action network calculates and outputs the strategy Strategy is the bidding information to be submitted by the agent in the current round. Is an array that contains four attributes in the bidding information. It is the reward fed back by the federated learning environment after the user takes action in the current round. Each agent has its own reward function. In this embodiment of the present invention, the agent's reward function can be used to calculate the agent's income resources in the current round.

[0087] For example, Figure 2 A flowchart of the process of outputting the bidding information to be submitted by the sample client in the current round for the multi-agent reinforcement learning algorithm. Here, each strategy includes an action network and a value network. The action network and the value network respectively include a main network and a target network. Figure 2 The specific algorithm flow is as follows:

[0088] For the number of episodes, episodes = 1 to M, iterate

[0089] Initialize the action space;

[0090] for t=1 to T, iterate

[0091] a) For each agent i, choose an action

[0092] b) Execute actiona=(a1,...,a N ), observe the reward r and the new state s t+1

[0093] c) will (s t ,a t ,r t ,s t+1 ,t) put it into the experience pool D;

[0094] d)s t ←s t+1

[0095] For agent ito N iterations:

[0096] Randomly extract small batches of stored samples from the experience pool (X j ,a j ,r j ,X′ j )

[0097] Mainnet Update

[0098] By minimizing the loss function To update the main network and target network;

[0099] Update the action network using gradient ascent:

[0100]

[0101] After all updates are completed, the target network is updated for each agent i: θ' i ←τθ i +(1-τ)θ' i

[0102] S120: Aggregating the updated model parameters uploaded by each sample client, and using the aggregated updated model parameters to update the model parameters in the global shared model.

[0103] Specifically, the ratio of the data volume of each sample client to the data volume of all sample clients can be calculated separately to obtain the data volume ratio corresponding to each sample client. After multiplying the data volume ratio corresponding to each sample client by the updated model parameters uploaded by the corresponding sample client, the updated model parameters corresponding to all sample clients are aggregated. The model parameters in the global shared model are updated by accumulating the aggregated updated model parameters.

[0104] It is understandable that the model parameters of the global shared model sent to each sample client are the same, while the model parameters corresponding to the global shared model trained by each sample client according to the output bidding information to be submitted are different. Here, after each sample client trains the global shared model, it will obtain local model parameters. By subtracting the local model parameters from the model parameters of the sent global shared model, the updated model parameters can be obtained, and the federated learning platform further updates the model parameters in the global shared model.

[0105] S130: If the updated global shared model reaches the preset model accuracy in the test task, the learning task released by the federated learning platform is determined to be completed. Otherwise, the steps of updating the model parameters in the global shared model are repeated for multiple rounds so that the updated global shared model reaches the preset model accuracy in the test task.

[0106] In the embodiment of the present invention, considering the huge exploration space of the agent, a multi-intelligence reinforcement learning algorithm with centralized training and distributed execution is used as the framework. Each sample client is an agent, and each agent has a strategy controller. The strategy controller consists of an action network and a value network. Each action network and value network is composed of two networks (main network and target network) for training and updating. The agent observes the task status of the current round, such as historical round selection and non-selection, historical prices, historical data volume, and historical unit resource volume, etc., as input to the action network in the strategy controller. The action network gives the action of the current round, that is, the bidding information to be submitted in the current round. The value network of each agent has the local state observed by each agent and the action taken, and is used as input to score the action output by this agent. Specifically, before each federated learning training round, each agent observes its own historical information s (historical bidding results, historical resource computing resources, historical data volume, and historical bidding prices) as state input into the policy controller. The policy controller then outputs the agent's action a, which represents the user's bid information for the current training round. The user submits this bid information to the federated learning platform (or environment). The federated learning platform selects appropriate sample clients to maximize its profit. The federated learning environment then provides each agent with a reward value r and transitions to the next state s'. The experience pool stores the tuple (s, a, s'r). When the experience pool is unable to collect new data, the policy controller begins training. In this embodiment of the present invention, the policy controller is trained using the principles of centralized training and distributed execution. Centralized training can be performed as follows: First, the action network in each agent's policy controller selects an action a based on the current state. Then, the value network calculates a Q-value based on the state-action pair as feedback to the action network for taking action a. The value network is trained based on the estimated and actual Q-values, and the action network updates its policy based on the feedback from the value network. To obtain more accurate Q-values, the policy controller's value network incorporates the actions and states of all agents during training. The value network parameters are updated using temporal difference (TD) and then the action network parameters are updated using gradient ascent. Distributed execution can be demonstrated as follows: after centralized training is complete, each agent executes according to its currently observed state distribution. After sufficient training time, the policy controller begins to converge, ultimately achieving optimal real-time bidding results.

[0107] The user bidding method based on a multi-agent reinforcement learning algorithm in federated learning provided by an embodiment of the present invention obtains a learning task issued by a federated learning platform, selects sample clients from a client set based on the learning task and the bidding information uploaded by the client set participating in the federated learning, and sends a global shared model to the sample clients. The updated model parameters uploaded by each sample client are received. The updated model parameters are formed by training the global shared model according to the configuration in the bidding information before the sample client starts using the multi-agent reinforcement learning algorithm. The updated model parameters uploaded by each sample client are further aggregated and the model parameters in the global shared model are updated using the aggregated updated model parameters. If the updated global shared model achieves a preset model accuracy in a test task, the learning task issued by the federated learning platform is determined to be completed. Otherwise, the step of updating the model parameters in the global shared model is repeated for multiple rounds so that the updated global shared model achieves the preset model accuracy in the test task. It can be seen from this that compared with the auction-based incentive mechanism in the prior art, the embodiment of the present invention can use a multi-agent learning system to adjust the bidding information uploaded by the client to increase the probability of the client being selected, thereby solving the problem of lack of fairness in federated learning caused by the auction-based incentive mechanism in the prior art because the strategy will not change during subsequent training.

[0108] Based on the above embodiment, another embodiment of the present invention provides a user bidding device based on a multi-agent reinforcement learning algorithm under federated learning, such as Figure 3 As shown, the device includes:

[0109] The acquisition unit 20 may be configured to acquire learning tasks published by the federated learning platform, select sample clients from the client set based on the learning tasks and bidding information uploaded by the client set participating in the federated learning, and distribute the global shared model to the sample clients;

[0110] The receiving unit 22 may be configured to receive updated model parameters uploaded by each sample client. The updated model parameters are generated by the sample client outputting the bid information to be submitted in the current round using a multi-agent reinforcement learning algorithm before training begins, and then training the global shared model according to the configuration in the bid information to be submitted after the sample client is selected.

[0111] The aggregation unit 24 may be configured to aggregate the updated model parameters uploaded by each sample client, and use the aggregated updated model parameters to update the model parameters in the global shared model;

[0112] The selection unit 26 can be used to determine that the learning task released by the federated learning platform is completed if the updated global shared model reaches the preset model accuracy in the test task; otherwise, repeat the steps of updating the model parameters in the global shared model for multiple rounds so that the updated global shared model reaches the preset model accuracy in the test task.

[0113] Optionally, the device further includes:

[0114] The output unit can be used by the sample client to output the bidding information to be submitted by the sample client in the current round using the multi-agent reinforcement learning algorithm;

[0115] The output unit is specifically used to use the sample client as an intelligent agent, the intelligent agent observes its own historical status information in the federated learning environment, and uses the historical status information to output the bidding information to be submitted by the sample client in the current round.

[0116] Optionally, the multi-agent reinforcement learning algorithm includes a strategy device and an experience pool, and the output unit includes:

[0117] A storage module may be configured to use the sample client as an agent and the experience pool in the multi-agent reinforcement learning algorithm to store historical task status information observed by each agent in the federated learning environment, wherein the historical task status information includes at least whether the agent was selected in historical rounds, historical resource values, historical data provided amounts, and historical unit resource amounts;

[0118] The output module can be used to input the historical task status information observed by the agent in the federated learning environment as the status information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm, and output the bidding information to be submitted by the agent in the current round.

[0119] Optionally, the output unit further includes:

[0120] The computing module can be used to input the historical task status information observed by the agent in the federated learning environment as the status information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm, output the bidding information to be submitted by the agent in the current round, calculate the benefit resources fed back by the federated learning environment to the agent in the current round, and use the experience pool in the multi-agent reinforcement learning algorithm to store the historical state of the environment observed by the agent in the current round, the bidding information to be submitted, the state of the environment after the bidding information to be submitted is uploaded, and the benefit resources fed back to the agent by the federated learning environment for the bidding information to be submitted uploaded in the current round.

[0121] Optionally, the calculation module may be specifically configured to obtain resource parameters involved in the bidding process of the agent based on the bidding information to be uploaded by the agent in the current round;

[0122] The calculation module can also be used to input the resource parameters involved in the bidding process of the agent into a pre-built profit function to obtain the profit resources fed back by the federated learning environment to the agent in the current round.

[0123] Optionally, each sample client is configured with a policy controller, the policy controller including an action network and a value network. The output module may be configured to input historical task status information observed by the agent in the federated learning environment as the state information of the agent in the current round into the action network of the policy controller, output the bid information to be submitted by the agent in the current round, and obtain the bid information to be uploaded by the agent in the current training round.

[0124] The output module may be further configured to input the state information of the agent in the current round and the bidding information to be uploaded by the agent in the current round into the value network of the strategist, evaluate the bidding information to be uploaded, and obtain an evaluation score for the bidding information to be uploaded;

[0125] Among them, the action network is trained using the evaluation score of the bidding information to be uploaded, and the network parameters of the action network are updated by gradient ascent. The value network is trained using the evaluation score of the bidding information to be uploaded and the income resources actually fed back by the intelligent agent, and the network parameters of the value network are updated by the temporal difference method.

[0126] Optionally, the aggregation unit 24 includes:

[0127] The calculation module can be used to calculate the ratio of the data volume of each sample client to the data volume of all sample clients, and obtain the data volume proportion corresponding to each sample client;

[0128] The aggregation module can be used to multiply the data volume proportion corresponding to each sample client by the updated model parameters uploaded by the corresponding sample client, aggregate the updated model parameters corresponding to all sample clients, and update the model parameters in the global shared model by accumulating the updated model parameters after aggregation.

[0129] Based on the above method embodiment, another embodiment of the present invention provides a storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the above method.

[0130] Based on the above embodiment, another embodiment of the present invention provides a vehicle, comprising:

[0131] one or more processors;

[0132] a storage device for storing one or more programs,

[0133] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method. The vehicle can be a non-autonomous driving vehicle or an autonomous driving vehicle.

[0134] The above-mentioned system and device embodiments correspond to the method embodiments and have the same technical effects as the method embodiments. For detailed descriptions, please refer to the method embodiments. The device embodiments are obtained based on the method embodiments. For detailed descriptions, please refer to the method embodiments section and will not be repeated here. It should be understood by those skilled in the art that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0135] Those skilled in the art will appreciate that the modules in the apparatuses of the embodiments may be distributed in the apparatuses of the embodiments as described in the embodiments, or may be located in one or more apparatuses different from the embodiments with corresponding changes. The modules in the above embodiments may be combined into one module or further divided into multiple sub-modules.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A user bidding method based on a multi-agent reinforcement learning algorithm in federated learning, characterized in that: The method comprises: Obtaining learning tasks published by the federated learning platform, selecting sample clients from the client set based on the learning tasks and bidding information uploaded by the client set participating in the federated learning, and issuing a global shared model to the sample clients; Receive updated model parameters uploaded by each sample client, the updated model parameters being generated by the sample client outputting its bid information for the current round using a multi-agent reinforcement learning algorithm before training begins, and then training a global shared model according to the configuration in the bid information after being selected; wherein the multi-agent reinforcement learning algorithm includes a policy controller and an experience pool; Aggregating the updated model parameters uploaded by each sample client, and using the aggregated updated model parameters to update the model parameters in the global shared model; If the updated global shared model reaches the preset model accuracy in the test task, the learning task released by the federated learning platform is considered completed. Otherwise, the steps of updating the model parameters in the global shared model are repeated for multiple rounds so that the updated global shared model reaches the preset model accuracy in the test task.

2. The method according to claim 1, wherein The process of the sample client using the multi-agent reinforcement learning algorithm to output the bidding information to be submitted by the sample client in the current round includes: The sample client is used as an intelligent agent. The intelligent agent observes its own historical status information in the federated learning environment, and uses the historical status information to output the bidding information to be submitted by the sample client in the current round.

3. The method according to claim 2, wherein The sample client is used as an intelligent agent, the intelligent agent observes its own historical status information in the federated learning environment, and uses the historical status information to output the bidding information to be submitted by the sample client in the current round, including: Using the sample client as an agent, the experience pool in the multi-agent reinforcement learning algorithm is used to store historical task status information observed by each agent in the federated learning environment, where the historical task status information includes at least whether the agent was selected in historical rounds, historical resource values, historical data provided amounts, and historical unit resource amounts; By inputting the historical task status information observed by the agent in the federated learning environment as the status information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm, the bidding information to be submitted by the agent in the current round is output.

4. The method according to claim 3, wherein After inputting the historical task status information observed by the agent in the federated learning environment as the state information of the agent in the current round into the strategist in the multi-agent reinforcement learning algorithm and outputting the bidding information to be submitted by the agent in the current round, the method further includes: Calculate the revenue resources fed back by the federated learning environment to the agent in the current round, and use the experience pool in the multi-agent reinforcement learning algorithm to store the historical state of the environment observed by the agent in the current round, the bid information to be submitted, the state of the environment after the bid information to be submitted is uploaded, and the revenue resources fed back to the agent by the federated learning environment for the bid information to be submitted uploaded in the current round.

5. The method according to claim 4, wherein The computational federated learning environment calculates the revenue resources of the agent in the current round, including: Based on the bidding information to be uploaded by the agent in the current round, the resource parameters involved in the bidding process of the agent are obtained respectively; The resource parameters involved in the bidding process of the agent are input into the pre-built profit function to obtain the profit resources fed back by the federated learning environment to the agent in the current round.

6. The method according to claim 3, wherein Each sample client is configured with a policy controller, which includes an action network and a value network. The policy controller inputs the historical task status information observed in the federated learning environment as the state information of the agent in the current round into the multi-agent reinforcement learning algorithm, and outputs the agent's bid information to be submitted in the current round, including: By inputting historical task status information observed by the agent in the federated learning environment as the state information of the agent in the current round into the action network in the strategy device, outputting the bidding information to be submitted by the agent in the current round, and obtaining the bidding information to be uploaded by the agent in the current training round; By inputting the state information of the agent in the current round and the bidding information to be uploaded by the agent in the current round into the value network in the strategist, the bidding information to be uploaded is evaluated to obtain an evaluation score of the bidding information to be uploaded; Among them, the action network is trained using the evaluation score of the bidding information to be uploaded, and the network parameters of the action network are updated by gradient ascent. The value network is trained using the evaluation score of the bidding information to be uploaded and the income resources actually fed back by the intelligent agent, and the network parameters of the value network are updated by the temporal difference method.

7. The method according to any one of claims 1 to 6, wherein The aggregating the updated model parameters uploaded by each sample client and using the aggregated updated model parameters to update the model parameters in the global shared model includes: Calculate the ratio of the data volume of each sample client to the data volume of all sample clients to obtain the data volume proportion corresponding to each sample client; After multiplying the data volume proportion corresponding to each sample client by the updated model parameters uploaded by the corresponding sample client, the updated model parameters corresponding to all sample clients are aggregated, and the model parameters in the global shared model are updated by accumulating the aggregated updated model parameters.

8. A user bidding device based on a multi-agent reinforcement learning algorithm in federated learning, characterized in that: The device comprises: An acquisition unit, configured to acquire a learning task published by a federated learning platform, select sample clients from the client set based on the learning task and bidding information uploaded by the client set participating in the federated learning, and issue a global shared model to the sample clients; A receiving unit, configured to receive updated model parameters uploaded by each sample client, the updated model parameters being formed by the sample client using a multi-agent reinforcement learning algorithm to output the bid information to be submitted by the sample client in the current round to train a global shared model; wherein the multi-agent reinforcement learning algorithm includes a strategy controller and an experience pool; an aggregation unit, configured to aggregate the updated model parameters uploaded by each sample client, and use the aggregated updated model parameters to update the model parameters in the global shared model; The selection unit is used to determine that the learning task released by the federated learning platform is completed if the updated global shared model reaches the preset model accuracy in the test task; otherwise, repeatedly execute the steps of updating the model parameters in the global shared model for multiple rounds so that the updated global shared model reaches the preset model accuracy in the test task.

9. A storage medium having executable instructions stored thereon, characterized in that: When the instruction is executed by the processor, the processor implements the user bidding method based on the multi-agent reinforcement learning algorithm under federated learning as described in any one of claims 1-7.

10. A device for user bidding based on a multi-agent reinforcement learning algorithm in federated learning, characterized in that: include: one or more processors; a storage device for storing one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the user bidding method based on the multi-agent reinforcement learning algorithm under federated learning as described in any one of claims 1-7.

Images