A model trading method based on hierarchical reinforcement learning in personalized federated learning

Abstract

The present invention discloses a model trading method based on hierarchical reinforcement learning in personalized federated learning. The method is based on personalized federated learning in a multi-user auction model scenario, including the establishment of a bilateral auction trading platform to incentivize model trading between participants, and give priority to protecting the privacy and security of local data sets. Then, a hierarchical multi-agent reinforcement learning algorithm is designed in this environment to optimize the user's bidding strategy and help the user buy a suitable model for aggregation. And a reward redistribution scheme is proposed to balance competition and cooperation. The algorithm designed based on hierarchical reinforcement learning in the present invention can achieve the optimization of overall social welfare and excellent personalized model accuracy while achieving stable returns for participants.

Description

Technical Field

[0001] The present invention belongs to federated learning technology and reinforcement learning technology, and specifically relates to a model auction trading technology in personalized federated learning, and in particular to a model trading method based on hierarchical reinforcement learning in personalized federated learning. Background Art

[0002] With the rapid development of information technology and smart devices, people's privacy data is growing at a high speed every day, and personal data security is increasingly valued. To address this problem, federated learning provides a decentralized model training method that can leverage the collective strengths of multiple devices without centralized data aggregation. In this framework, the central server only receives trained models, not local private data, thereby protecting user privacy. It has been proven that this collaborative learning approach can effectively address the challenges associated with traditional methods, especially when dealing with independent and identically distributed data sets.

[0003] Personalized federated learning is a branch of federated learning. It was proposed to solve the problem that traditional federated learning performs poorly on non-IID datasets and lacks personalization. This method aims to train a personalized model for each user to better adapt to the unique characteristics of their local data. Although personalized federated learning brings new possibilities for improving model performance, in real applications, users may not actively participate in federated learning. Therefore, in the process of personalized federated learning, in order to ensure that each user is actively involved in model training, the following problems need to be solved:

[0004] 1) Users are unwilling to provide local models for free due to privacy, security and training costs. Ensuring privacy and security while improving model performance has become a requirement to be considered. It is crucial to promote user participation and reduce costs while minimizing potential risks. The traditional method of forcibly distributing models by a central server cannot cope with this reality. It is necessary to consider designing an incentive mechanism for users, such as a model auction market.

[0005] 2) The phenomenon described in 1) raises some fundamental questions:

[0006] 2.1) How should the incentive mechanism be designed? If an auction market is adopted, what criteria should users use to select models for bidding? What strategies should users use to bid at appropriate prices? How does the auctioneer conduct the auction process and how does the price clear?

[0007] 2.2) If we want to maximize the overall benefits, how can we balance the competition and cooperation among participants? How can we prevent the monopoly effect caused by competition or the excessive damage to individual interests caused by cooperation?

[0008] To answer these questions, this paper studies how to use hierarchical multi-agent reinforcement learning to optimize users’ bidding strategies and competition and cooperation in bilateral auction markets. However, this problem is not easy for the following reasons:

[0009] a. The auction market in personalized federated learning is not a simple one-round transaction, but a complex process involving multiple rounds and multiple users. Each complete round of the process includes multiple transactions, which is a problem related to timing. In addition, there are multiple users participating in the market, and each user needs to make multiple actions in each transaction, resulting in a very complex environment.

[0010] b. It is very difficult to train each user's own bidding strategy, because it is necessary to know which models in the auction market can improve the accuracy of the local model after aggregation. Due to the different training data sets, the criteria for selecting models are also different. At the same time, each user has his own budget constraints. It is very difficult to use the limited budget to make a reasonable bid in each transaction round.

[0011] c. The user's bidding strategy will directly affect the relationship with other participants. If the user tends to bid high, it will tend to be a competitive relationship. If the user tends to bid low so that the model can be purchased by participants in need, it will tend to be a cooperative relationship. How to optimize the bidding strategy to control the competitive and cooperative relationship within a reasonable range is also a challenge that the present invention needs to overcome. Summary of the invention

[0012] Purpose of the invention: In response to the problems and challenges of personalized federated learning model training and trading in the above-mentioned existing work, the present invention provides a hierarchical multi-agent reinforcement learning trading method for individual strategy optimization in personalized federated learning. The method is also a hierarchical multi-agent reinforcement learning algorithm that helps participants train bidding strategies, which can achieve a bidding algorithm that converges to stability and maximizes overall social welfare while ensuring individual interests.

[0013] Technical solution: A model trading method based on hierarchical reinforcement learning in personalized federated learning. The method is based on a personalized federated learning framework, oriented to multi-user model auction training collaboration, with the goal of optimizing the performance of all individual models. Hierarchical multi-agent reinforcement learning is used to help users train bidding strategies, and model trading is carried out in the bilateral auction market. The reward function of reinforcement learning is designed based on the idea of ​​reward redistribution, and the trading strategy is optimized and improved to maximize the overall benefit while ensuring the accuracy of the individual user model.

[0014] The method comprises the following steps:

[0015] (1) Define the initial model, considering the user set N, each participant i has a local training dataset D i ={(xi ,y i )}, where x i and i Represents the training data and labels in the dataset;

[0016] At the beginning of each round of training, each user gets a local machine learning model w t，i , where t represents the number of transaction rounds in the current training, i represents the user ID, and then the user trains the dataset D locally. i The performance of the model is determined by the dataset D i Loss on To express;

[0017] (2) Build a bilateral auction platform to realize model transactions between users. The central server of federated learning is used as a trusted auctioneer. Users participating in the auction submit their local models w t，i Upload to the central server, which calculates the similarity of each model with other models And the similarity sequence s t，i Send back to the user;

[0018] In the bilateral auction transaction framework, the similarity between models is calculated by cosine similarity:

[0019] s t，ij =1-cosine similarity(w t，i , w t，j );

[0020] (3) As a buyer, the user receives the similarity sequence and uses it as observation information for the hierarchical multi-agent reinforcement learning model to select the model set H that participates in the auction. t，i ={h t，i1 ...h t，in |j≠i}, and then bid for each model to be purchased to obtain the bidding set a t，i ={a t，ij}, and as a seller, you need to submit the selling price of the local model p t，i , submit these to the central server;

[0021] (4) After receiving the bid and selling price, the central server starts the auction transaction and determines the winning bid and clearing price;

[0022] For the model w t，i , sort the bids from high to low, higher than p t，i The first n sell The buyer is the winner, assuming that the nth sellIf the bid of +1 buyer is higher than the selling price, then this price is used as the clearing price, otherwise it is the selling price. After the auction of all models is over, the server counts the auction results of each user and sends the aggregated model and transaction price to protect the privacy of the user model.

[0023] (5) The user updates and trains the received model as a local model, optimizes the local federated learning model, and pays and updates his or her remaining budget.

[0024] Furthermore, in step (2), the auctioneer of the bilateral auction platform, i.e., the central server, performs similarity calculation on the model submitted by the user, and the following relationship exists:

[0025] s t，i ={s t，i1 , ..., s t，ij , ..., s t，in |j≠i}

[0026] s t，ij =1-cosine similarity(w t，i , w t，j )

[0027] In the calculation of similarity, by using cosine similarity and vector angle, the change of parameter amplitude caused by different amounts of training data is ignored.

[0028] In step (3), a hierarchical multi-agent reinforcement learning algorithm is used to enable users participating in the auction to decide the set of models to purchase in the current transaction round, as well as to solve the problem of bidding for each model in the auction. The reinforcement learning agent corresponding to each user is composed of a model selector and a bid generator, wherein the model selector is a low-level agent and the bid generator is a high-level agent.

[0029] Furthermore, step (3) specifically includes the following process:

[0030] a) Model selection problem for bidding: The model selector uses the DDPG algorithm and first observes the low-level state That is, the model similarity sequence returned by the central server is sorted from high to low according to the similarity; then the model selector outputs the low-level action where μ l is the policy network of the model selector, with a similarity greater than or equal to The model is the selected model, and finally the model set to participate in the auction is obtained. where h t，ij ∈{0, 1}, 1 means participating in the auction of this model, otherwise not participating;

[0031] b) Bid price generation problem: The bid generator must not only consider the value of the current auction model to itself, but also the behavior of other participants. Therefore, the MADDPG algorithm is used to handle this problem. Specifically:

[0032] Take the model set output by the model selector as the observation value, and process each model to be bid. First, input where s t，ij is the similarity between the local model of user i and the model to be bid, B t，i is the user’s remaining budget; for each user, the highest bid for each transaction round is initially set, expressed as a fixed value p i , the agent outputs the bidding ratio The final bid price is obtained after combining

[0033] Finally, the user sends the above results to the auction platform, and the bilateral auction platform executes the auction process.

[0034] Furthermore, in step (4), after executing the auction process, the central server aggregates and sends the models purchased by each user. The aggregation process uses the similarity of the models as the weight, which is mathematically expressed as follows:

[0035]

[0036]

[0037] Among them, m t，ij Indicates whether user i successfully purchased model j at time t. If successful, it is 1, otherwise it is 0. It is the weight parameter corresponding to aggregation. If the user has not purchased the model, no aggregation will be performed.

[0038] The above formula uses similarity as weight to reduce the risk of privacy leakage caused by providing data volume privacy information to the server, and reduces the risk of privacy leakage caused by weighting the data volume .... t，i The impact on performance.

[0039] Furthermore, in step (5), the algorithm training process is performed round by round, each round includes multiple transaction rounds, the central server conducts model auction according to the above process, and individual users aggregate the purchased models. This training process continues until the global model converges;

[0040] Furthermore, the mathematical expression of the budget constraint set by this method to maximize the overall benefit while ensuring the accuracy of the single user model is as follows:

[0041]

[0042]

[0043]

[0044]

[0045] s t，ij =1-cosine similarity(w t，i , w t，j )

[0046] In the formula, It represents the sum of the losses of all user models. The smaller the value, the better the overall model performance is, which is also the overall goal to be optimized.

[0047] Furthermore, this method aims at the competitive relationship existing in the auction market where more than one user participates. The mutual cooperation with social welfare as the overall goal can solve the above problem. Specifically, the temperature-based reward redistribution mechanism is used as the reward function of reinforcement learning to guide the optimization direction of its strategy network to balance the competition and cooperation relationship between participants.

[0048] The reward redistribution mechanism is defined as follows:

[0049]

[0050]

[0051] It is a weighted parameter for assigning rewards to each agent, using a bidding method t，i And model accuracy acc t，i To measure the contribution of the agent to the total reward, where bid t，i The higher it is, the more likely the agent is to succeed in the bid, which means it is more likely to get a reward, and has a greater impact on the total reward than agents with lower bids.

[0052] Furthermore, the method includes reflecting the contribution of social welfare through model accuracy, controlling the cooperative and competitive relationship between agents through parameter τ, modifying the relationship between competition and cooperation between agents, avoiding the emergence of monopoly effects or serious damage to individual benefits, and avoiding the behavior of agents colluding with each other to report low prices when cooperating;

[0053] The specific consideration is: the larger the value of τ is, the smaller the change in the bid and model accuracy of agent i is for τ, that is, the change of λ tends to be stable. In order to get a higher reward, the agent will tend to increase the overall reward. Develop a cooperative relationship, no longer blindly increase the bid but allow other agents to buy the model and get rewards; if τ is small, a slight increase in the agent's bid and accuracy will bring about a big change, tending to increase the bidding price to obtain more reward value, which will lead to competition between agents and further reduce social welfare. In order to distribute more of the reduced total reward, the price can only continue to rise to be more competitive.

[0054] Beneficial effects: Compared with the prior art, the substantial progress and significant effects of the present invention are as follows:

[0055] 1) The method of the present invention can realize a personalized federated learning model trading platform based on bilateral auctions, encourage customers to trade models through bidding, realize data sharing between users, and provide an effective interactive platform for personalized federated model training. At the same time, the present invention further improves the privacy security of user data, helps to enhance user trust, and solves the potential privacy leakage problem faced in the auction market.

[0056] 2) Most of the existing data calculation and processing work is aggregated by the central server, or simple strategy learning is performed using reinforcement learning. The design of hierarchical multi-agent reinforcement learning in the present invention enables the platform to make transaction decisions more flexibly and efficiently in a complex multi-user, multi-action environment. The model selector of the low-level agent is responsible for generating a set of purchase models, and the high-level agent is responsible for optimizing the bid price of the bidding model as a bidding generator, thereby optimizing the decision-making at different levels. The total task is divided into small tasks and assigned to the corresponding agents, which reduces the difficulty of agent learning. At the same time, all users share the low-level agent, which speeds up the training efficiency of the overall algorithm. In addition, the observation value of the low-level agent is the similarity between the user's local model and the model in the auction market, and the observation value of the high-level agent is the set of models to be purchased output by the low-level agent. That is to say, these input conditions are independent of the specific model. The reinforcement learning model trained by the algorithm can have good migration. Even if the data set of the federated learning model is changed, it does not affect the bidding decision made by the reinforcement learning model. Experiments have shown that even if the user's personalized federated learning model is migrated to a new data set, the model accuracy after auction aggregation can also achieve good performance.

[0057] 3) In order to balance the competitive and cooperative relationship, the present invention proposes a reward redistribution scheme to encourage competition and cooperation by allocating total rewards to agents. This helps ensure that all participants receive fair returns throughout the process, prompting them to participate in cooperation more actively. It ensures that more users can successfully purchase models that suit them, improving overall social welfare. This is an innovative solution to the problems caused by the competitive relationship in the traditional auction market.

[0058] 4) From the algorithm aspect, it can be proved according to the present invention that the algorithm provided by the present invention can converge and has better performance than the traditional method. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a schematic diagram of the structure of a bilateral auction market constructed by the method of the present invention;

[0060] Figure 2 It is a schematic diagram of the structure of the hierarchical multi-agent reinforcement learning neural network model designed by the present invention;

[0061] Figure 3 is an experimental result diagram of the model accuracy obtained after the method of the present invention performs personalized federated learning on different data sets;

[0062] Figure 4 This is a graph of experimental results of migrating the reinforcement learning model trained on the MNIST dataset to different datasets.

[0063] Figure 5 This is a comparison of experimental results when the reward redistribution mechanism is not adopted and when it is adopted. DETAILED DESCRIPTION

[0064] In order to explain the technical solution disclosed in the present invention in detail, further elaboration is made below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0065] In the traditional federated learning setting, an auction mechanism is introduced to conduct collaborative model training between users. Although it can improve the performance of the model, individual users may not know which models are beneficial to the performance improvement of the local model after aggregation during the personalized federated learning process, and the competitive relationship in the market will lead to the emergence of a monopoly effect, which is not conducive to the overall social welfare. To address this problem, the present invention provides a hierarchical multi-agent reinforcement learning transaction method for individual strategy optimization in personalized federated learning. The method is based on a personalized federated learning framework and aims to optimize the accuracy of each user's local personalized model during the model auction process. First, the central server calculates the similarity between the individual user model and other models, and then uses the similarity sequence results as input. With the help of the hierarchical multi-agent reinforcement learning algorithm designed based on the hierarchical idea of ​​the present invention, users can effectively select the model set to bid, and then propose a corresponding reasonable bid for each model, and can also make the overall model performance tend to the optimal convergent solution.

[0066] After the users participating in the personalized model auction obtain the solution provided by the method of the present invention, for the personalized federated learning model bilateral auction market participated by multiple users, the present invention adopts an aggregation algorithm with model similarity as weight for training and learning. The training process is carried out in rounds. In each round, the central server first calculates the model similarity in the current market and distributes it to each participant. After each participant submits the model set to be purchased, the corresponding bid price set, and the local model selling price, the central server will clear the auction and then aggregate the models successfully purchased by each user. The user uses the aggregated model to update the parameters of the local model. This process continues until the global model converges, and finally the global model accuracy is used as a reference indicator.

[0067] The specific implementation steps of the technical solution provided by the present invention can be described as follows:

[0068] (1) Define the initial model. Consider a user set N, where each participant i has a local training dataset D i ={(x i ,y i )}. i and i Represents the training data and labels in the dataset.

[0069] At the beginning of each round of training, each user gets a local machine learning model w t，i , where t represents the number of transaction rounds in the current training, i represents the user ID, and then the user trains the dataset D locally. i The performance of the model is determined by the dataset D i Loss on To express;

[0070] (2) Build a bilateral auction platform to enable model transactions between users.

[0071] The central server of federated learning acts as a trusted auctioneer, and users participating in the auction submit their local models w t，i Upload to the central server, the server calculates the similarity of each model with other models And the similarity sequence s t，i Send back to the user.

[0072] In the above-mentioned bilateral auction transaction framework, the present invention calculates the similarity between models by cosine similarity. There is the following relationship:

[0073] s t，i ={s t，i1 , ..., s t，ij , ..., s t，in |j≠i}

[0074] s t，ij =1-cosine similarity(w t，i , w t，j )

[0075] Compared with the traditional method of using parameter vectors to calculate model similarity, using cosine similarity can better utilize the vector angle and ignore the changes in parameter amplitude caused by different amounts of training data.

[0076] Furthermore, as a buyer, the user receives the similarity sequence and uses it as the observation information of the hierarchical multi-agent reinforcement learning model to select the model set H that participates in the auction. t，i ={h t，i1 ...h t，in |j≠i}, and then bid for each model to be purchased to obtain the bidding set a t，i ={a t，ij}, and as a seller, you need to submit the selling price of the local model p t，i , submit these to the central server.

[0077] The reinforcement learning agent corresponding to each user consists of a model selector and a bid generator. The model selector is a low-level agent, and the bid generator is a high-level agent. In the implementation process, the following two problems are solved:

[0078] a) Model selection problem for bidding: The model selector uses the DDPG (Deep Deterministic Policy Gradient) algorithm. First, the low-level state is observed. That is, the model similarity sequence returned by the central server is sorted from high to low according to the similarity. Then the model selector outputs the low-level action where μ l is the policy network of the model selector, with a similarity greater than or equal to The model is the selected model, and finally the model set to participate in the auction is obtained. where h t，ij ∈{0,1};

[0079] b) Bid price generation problem: The bid generator must not only consider the value of the current auction model to itself, but also the behavior of other participants. Therefore, the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm is used to deal with this complex competitive environment. The model set output by the model selector is used as the observation value. For each model to be bid, the first input where s t，ijis the similarity between the local model of user i and the model to be bid, B t，i is the user’s remaining budget. For each user, the highest bid for each transaction round is initially set, represented by a fixed value p i , the agent outputs the bidding ratio The final bid price is obtained after combining

[0080] In addition, when the central server distributes the current similarity sequence to each user, the reinforcement learning agents corresponding to these users will be able to use the low-level historical experience buffer pool to and advanced historical experience buffer pool A batch is randomly selected as training data, and then the optimization algorithm is used to optimize the models of the two layers of agents respectively. The corresponding historical buffer experience can help the user's reinforcement learning agent converge to a stable range faster.

[0081] Finally, the user sends the above results to the auction platform, and the bilateral auction platform executes the auction process. After the central server receives the bid and the selling price, it starts the auction transaction and determines the winning bid and the clearing price. t，i , sort the bids from high to low, higher than p t，i The first n sell The buyer is the winner, assuming that the nth sell If the bid of +1 buyer is higher than the selling price, then this price is used as the clearing price, otherwise it is the selling price. After the auction of all models is over, the server counts the auction results of each user. In order to protect the privacy of user models, the central server aggregates and sends the models purchased by each user after executing the auction process. The aggregation process uses the similarity of the model as the weight, which is mathematically expressed as follows:

[0082]

[0083]

[0084] Among them, m t，ij Indicates whether user i successfully purchased model j at time t. If successful, it is 1, otherwise it is 0. It is the weight parameter corresponding to aggregation. If the user has not purchased the model, no aggregation will be performed.

[0085] Compared with the traditional method of using the amount of training set data as weight, data weighting requires providing data privacy information to the server, increasing the risk of privacy leakage, and if a model with large data volume but low similarity appears, it will seriously affect w t，i performance, and using similarity as weight can reduce this impact.

[0086] The user updates and trains the received model as a local model, optimizes the local federated learning model, and pays and updates his or her remaining budget.

[0087] (3) System problem modeling and balance of competitive and cooperative relationships.

[0088] In order to maximize social welfare, that is, maximize the sum of the utilities of all individuals, and satisfy the budget constraints of each user, the constraints are mathematically expressed as follows:

[0089]

[0090]

[0091]

[0092]

[0093] s t，ij =1-cosine similarity(w t，i , w t，j )

[0094] In the formula, It represents the sum of the losses of all user models. The smaller the value, the better the overall model performance is, which is also the overall goal to be optimized.

[0095] Competition is inevitable in an auction market with multiple users. Users with larger budgets and a tendency to bid high may form an oligopoly effect. Competition leads to higher prices, and only a few users will be able to buy models, while other users fail in the competition and can only train locally, contributing less social welfare. In addition, low-quality models will reduce their own model accuracy after aggregation. Mutual cooperation with social welfare as the overall goal can solve the above problems, but it may sacrifice the interests of some individual users. A temperature-based reward redistribution mechanism is designed as the reward function of reinforcement learning to guide the optimization direction of its policy network to balance the relationship between competition and cooperation among participants. The reward redistribution mechanism is defined as follows:

[0096]

[0097]

[0098] is a weighted parameter for allocating rewards to each agent. This invention adopts bidding t，i And model accuracy acc t，i To measure the contribution of the agent to the total reward. t，iThe higher the value, the more likely the agent is to succeed in bidding, which means it is more likely to get rewarded, and has a greater impact on the total reward than agents with lower bids. Since price is only one factor, it is not comprehensive enough to use it to indirectly reflect contribution, so the model accuracy is added in this invention. In addition, The most critical parameter is τ, which is used to control the competition and cooperation between agents. The analysis of τ is as follows:

[0099] 1) If τ takes a larger value, for agent i, the change in its bid and model accuracy is small for τ, that is, the change in λ tends to be stable. In order to get a higher reward, the agent will tend to increase the overall reward. Develop a cooperative relationship, no longer just raise the bid but allow other agents to buy the model and get rewards.

[0100] 2) If τ is small, a slight improvement in the agent's bid and accuracy will lead to larger changes, tending to increase the bidding price to obtain more reward value, which will lead to competition between agents and further reduce social welfare. In order to distribute more of the reduced total reward, the price can only continue to rise to be more competitive.

[0101] By adjusting the value of τ, the relationship between competition and cooperation among agents can be modified, effectively avoiding the emergence of monopoly effects or serious damage to individual benefits, which plays a vital role in complex auction environments involving multiple agents. At the same time, in the auction process, it is stipulated that the bid must be higher than the selling price for the auction to be successful, which avoids the behavior of agents colluding with each other to report low prices when cooperating. In essence, it sets a lower limit to ensure that the bid output after adjusting τ can be within a reasonable range. In general experiments, the value range of parameter τ can be [10, 60].

[0102] Experiments show that in all data sets, the method proposed by the present invention always achieves the best performance. Compared with other existing algorithms, the present invention has a stronger ability to adapt to multi-agent environments. The existing algorithms are not as good as the present invention in terms of final model performance after convergence and stability during the convergence process. The present invention considers model similarity as a weighting coefficient for model aggregation to reduce the impact of low-similarity models on local models. In terms of design, it pays more attention to the effective use of information from other models, which can help to understand the environment more comprehensively and adapt to changes more effectively during the learning process. And it still performs well on other data sets, proving that it has strong generalization and applicability. This is attributed to the fact that the information mainly observed by the agent in the algorithm is independent of the specific model and the distribution of training data, but pays more attention to the model similarity ranking, so that the model can play a role in different environments.

Claims

1. A model trading method based on hierarchical reinforcement learning in personalized federated learning, characterized in that: The method is based on a personalized federated learning framework and is aimed at multi-user model auction training collaboration. It aims to optimize the performance of all individual models. It uses hierarchical multi-agent reinforcement learning to help users train bidding strategies, and uses this to trade models in the bilateral auction market. It designs a reinforcement learning reward function based on the idea of ​​reward redistribution, optimizes and improves the trading strategy, and maximizes the overall benefit while ensuring the accuracy of individual user models. The method comprises the following steps: (1) Define the initial model, considering the user set N, each user i has a local training dataset D i ={(x i ,y i )}, where x i and i Represents the training data and labels in the dataset; At the beginning of each round of training, each user gets a local model w t,i , where t represents the number of transaction rounds in the current training, i represents the user ID, and then the user trains the dataset D locally. i The performance of the model is determined by the dataset D i The loss on l(D i ;w i ) to indicate; (2) Build a bilateral auction platform to realize model transactions between users. The central server of federated learning is used as a trusted auctioneer. Users participating in the auction submit their local models w t,i Upload to the central server, and the server calculates the similarity s between each local model and other models t,ij , and the similarity sequence s t,i Send back to the user; The central server calculates the similarity of the model submitted by the user as follows: s t，i ={s t,i1 ,...,s t,ij ...,s t,in |j≠i} s t,ij =1-cosine similarity(w t，i ,w t，j ) In the calculation of similarity, the cosine similarity and vector angle are used to ignore the parameter amplitude changes caused by the different amount of training data; (3) As a buyer, the user receives the similarity sequence and uses it as observation information for the hierarchical multi-agent reinforcement learning model to select the model set H that participates in the auction. t,i ={h t,i1 ..h t,ij ....h t,in |j≠i}, and then bid for each model to be purchased to obtain the bidding set a t,i ={a t,ij }, and as a seller, you need to submit the selling price of the local model p t,i , submit these to the central server; In this step, a hierarchical multi-agent reinforcement learning algorithm is used to enable users participating in the auction to decide on the model set to purchase in the current transaction round, and to solve the problem of quoting for each model in the auction. The reinforcement learning agent corresponding to each user is composed of a model selector and a bid generator, wherein the model selector is a low-level agent and the bid generator is a high-level agent. The solution to the above problem specifically includes the following process: a) Model selection problem for bidding: The model selector uses the DDPG algorithm and first observes the low-level state That is, the model similarity sequence returned by the central server is sorted from high to low according to the similarity; then the model selector outputs the low-level action Where μl is the policy network of the model selector, and the similarity is greater than or equal to The model is the selected model, and finally the model set to participate in the auction is obtained. where h t,ij ∈{0,1}, 1 means participating in the auction of this model, otherwise not participating; b) Bid price generation problem: The bid generator must not only consider the value of the current auction model to itself, but also the behavior of other participants. Therefore, the MADDPG algorithm is used to handle this problem. Specifically: Take the model set output by the model selector as the observation value, and process each model to be bid. First, input where s t,ij is the similarity between the local model of user i and the model to be bid, B t,i is the user’s remaining budget; for each user, the highest bid for each transaction round is initially set, expressed as a fixed value p i , the agent outputs the bidding ratio The final bid price is obtained after combining Finally, the user sets the model The bid price corresponding to each model in is sent to the auction platform, and the bilateral auction platform executes the auction process; (4) After receiving the bid and selling price, the central server starts the auction transaction and determines the winning bid and clearing price; For the local model w t,i , sort the bids from high to low, higher than p t,i The first n sell The buyer is the winner, assuming that the nth sell If the bid of +1 buyer is higher than the selling price, then this price is used as the clearing price, otherwise it is the selling price. After the auction of all models is over, the server counts the auction results of each user and sends the aggregated model and transaction price to protect the privacy of the user model. After executing the auction process, the central server aggregates and sends the models purchased by each user. The aggregation process uses the similarity of the model as the weight, which is mathematically expressed as follows: Among them, m t,ij Indicates whether user i successfully purchased model j in the transaction round number t in the current training. If successful, it is 1, otherwise it is 0. It is the weight parameter corresponding to aggregation. If the user has not purchased the model, no aggregation will be performed. The above formula can reduce the local model w by using similarity as weight. t,i Impact on performance; (5) After executing the auction process, the central server aggregates and sends the models purchased by each user. The user updates and trains the received model as a local model, optimizes the local federated learning model, and pays and updates its remaining budget.

2. The model trading method based on hierarchical reinforcement learning in personalized federated learning according to claim 1, characterized in that: Step (5) The updating and training process of the local model is carried out in rounds, each of which includes multiple transaction rounds. The central server conducts model auctions according to the above process, and individual users aggregate the purchased models. This training process continues until the global model converges.

3. The model trading method based on hierarchical reinforcement learning in personalized federated learning according to claim 1, characterized in that: This method ensures the accuracy of the single user model and sets the budget constraint to maximize the overall benefit. The mathematical expression is as follows: s t,ij =1-cosine similarity(w t，i ,w t,j ) In the formula, It represents the sum of the losses of all user models. The smaller the value, the better the overall model performance is, which is also the overall goal to be optimized.

4. The model trading method based on hierarchical reinforcement learning in personalized federated learning according to claim 3, characterized in that: This method targets the competitive relationship existing in the auction market where more than one user participates, and takes social welfare as the overall goal of mutual cooperation. Specifically, it adopts a temperature-based reward redistribution mechanism as the reward function of reinforcement learning to guide the optimization direction of its strategy network to balance the relationship between competition and cooperation among participants. The reward redistribution mechanism is defined as follows: It is a weighted parameter for assigning rewards to each agent, using a bidding method t,i And the model accuracy acc t,i To measure the contribution of the agent to the total reward, where bid t,i The higher it is, the more likely the agent is to succeed in the bid, which means it is more likely to get a reward, and has a greater impact on the total reward than agents with lower bids.

5. The model trading method based on hierarchical reinforcement learning in personalized federated learning according to claim 4, characterized in that: The method includes reflecting the contribution of social welfare through model accuracy, controlling the cooperative and competitive relationship between agents through parameter τ, modifying the relationship between competition and cooperation between agents, avoiding the emergence of monopoly effects or serious damage to individual benefits, and avoiding the behavior of agents colluding with each other to report low prices when cooperating; The specific consideration is: the larger the value of τ is, the smaller the change in the bid and model accuracy of agent i is for τ, that is, The changes tend to be stable. In order to get higher rewards, the intelligent experience tends to increase the overall reward Develop a cooperative relationship, no longer blindly increase the bid but allow other agents to buy the model and get rewards; if τ is small, a slight increase in the agent's bid and accuracy will bring about a big change, tending to increase the bidding price to obtain more reward value, which will lead to competition between agents and further reduce social welfare. In order to distribute more of the reduced total reward, the price can only continue to rise to be more competitive.

Images