A hierarchical federated multi-task learning method based on many-to-many contract design

By designing many-to-many contracts and using the MAPPO algorithm, the problem of computing resource allocation in multi-task publisher scenarios is solved, improving user participation and the adaptability of incentive mechanisms, and achieving reasonable allocation of computing resources and enhancing user willingness.

CN120952108BActive Publication Date: 2026-11-10GUANGDONG UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511090396.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-08-05
Publication Date
2026-11-10
Estimated Expiration
2045-08-05

AI Technical Summary

Technical Problem

In federated learning scenarios with multiple task publishers, existing technologies struggle to allocate computing resources effectively, resulting in low user willingness to participate in training. Furthermore, traditional contract design methods are only applicable to a single task publisher and cannot effectively incentivize participation from multiple task parties.

Method used

A hierarchical federated multi-task learning method based on many-to-many contract design is adopted. Through dynamic game theory and MAPPO algorithm, a contract incentive mechanism is designed to reasonably allocate users' limited computing resources and improve users' willingness to participate in training.

Benefits of technology

It achieves the adaptability and robustness of the incentive mechanism in complex, non-stationary, and incomplete information environments, improves the rational allocation of computing resources and user participation, and adapts to the incentive needs of multi-task scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120952108B_ABST
    Figure CN120952108B_ABST
Patent Text Reader

Abstract

This invention provides a hierarchical federated multi-task learning method based on many-to-many contract design. First, multiple agents generate multiple contracts, which are then processed by the environment and passed to each device set. The sets select the contracts with the highest utility using a dynamic programming algorithm and feed them back to the environment. The environment simulates and calculates the utility of each agent based on the set's selections and returns the average utility as a reward to the agents. The MAPPO algorithm iterates according to the above steps. In each iteration, the agents store experience in an experience pool until certain conditions are met. At this point, the agents retrieve experience from the experience pool to update the action network and the critic network. This invention expands the incentive mechanism from "one-to-many" to "many-to-many," more closely resembling real-world scenarios. Furthermore, it uses the MAPPO algorithm to design contract incentives in complex dynamic game scenarios. This incentive mechanism significantly improves adaptability and robustness in complex, non-stationary, and incomplete information environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of federated learning technology, and in particular to a hierarchical federated multitasking learning method based on many-to-many contract design. Background Technology

[0002] The Industrial Internet of Things (IIoT) is a product of the deep integration of next-generation information technologies (5G, big data, AI, and IoT) with industrial systems. It connects people, machines, things, and systems to build an intelligent manufacturing and service ecosystem covering the entire industrial chain. In the IIoT, servers connect a large number of mobile digital devices, production machinery, and industrial equipment. These devices continuously generate rich data and information, enabling production personnel to monitor, maintain, and analyze them. The heterogeneity and complexity of IIoT systems bring many technical challenges, such as privacy, heterogeneity, and resource efficiency.

[0003] Federated learning is considered a promising technique for realizing AI on distributed endpoints with privacy. Specifically, federated learning learns a shared model by aggregating local updates from endpoints while keeping training data on the endpoints to protect privacy. Currently, federated learning is widely used in many scenarios. However, traditional federated learning frameworks are designed to train a single, holistic global model, implicitly assuming that all clients share a common learning objective and that their data is drawn from similar underlying distributions. This assumption often fails in large-scale real-world networks due to significant statistical heterogeneity among clients. Federated multi-task learning redefines this problem by treating each client or set of clients as a unique task. Leveraging the principles of multi-task learning, federated multi-task learning learns a personalized model for each task while uncovering the intrinsic relationships between tasks, improving collective learning efficiency and model accuracy. This approach not only respects the heterogeneity of the network but also allows for knowledge transfer across tasks, enabling data-scarce tasks to benefit from the experience of data-rich tasks, thus transcending the constraints of traditional federated learning.

[0004] Meanwhile, traditional federated learning often faces millions or even hundreds of millions of devices. If all devices communicate directly with a central server, it will instantly lead to server overload and network congestion. This "flat" architecture has extremely poor scalability. Therefore, layered federated multi-task learning can greatly reduce the burden on the central server, allowing the system to scale to a very large scale. Due to the advantages of flexible deployment and high efficiency, unmanned aerial vehicles (UAVs) are often used as mobile base stations in edge networks.

[0005] Furthermore, in a federated learning architecture, user devices need to contribute local computing resources for federated learning training. This undoubtedly consumes user device computing resources and generates energy. Without an incentive mechanism to compensate user devices, it may lead to low user participation in training, insufficient participating user devices, and ultimately, failure to complete federated learning training. Therefore, it is necessary to design an effective incentive mechanism for learning tasks. It is worth noting that previous work on incentives has mostly been based on the existence of only one task issuer within the system, focusing primarily on the costs of energy consumption and training latency for participants. This is clearly inconsistent with the goals in diverse task scenarios.

[0006] Patent document CN115081676A discloses a multi-task wireless federated learning method based on contract theory. However, most federated learning contract design methods, including this prior art, can only implement contract incentives within a single task publisher's system. Unlike traditional hypothetical scenarios, in reality, multiple task parties often need to simultaneously publish learning tasks and train models through federated learning. In such cases, due to limited computing resources on the user-side participating in training, contracts can only be obtained within resource constraints. Therefore, how multiple task parties can design contracts to rationally allocate computing resources under their respective constraints becomes a pressing problem. Summary of the Invention

[0007] To overcome the technical problem of the prior art in achieving reasonable allocation of computing resources when facing federated learning with multiple task publishers, this invention provides a hierarchical federated multi-task learning method based on many-to-many contract design. It extends the traditional "one-to-many" contract design concept to "many-to-many" contract design, which is closer to the real-world scenario. It also rationally allocates users' limited computing resources through dynamic game between contract designers and receivers, thereby increasing users' willingness to participate in training.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A hierarchical federated multi-task learning method based on many-to-many contract design is disclosed. The method is applied to a pre-defined communication system, which includes multiple edge servers, multiple UAV base stations, and multiple terminal devices. The edge servers act as task publishers, publishing federated learning tasks and their corresponding designed contracts. The terminal devices are divided into several device sets with the goal of minimizing data distribution differences. Each device set corresponds to a UAV base station, and each terminal device in a device set communicates with the edge server through its corresponding UAV base station. The method is used to perform federated multi-task learning, and each training round includes the following steps: S1: Each set of devices obtains the global model and contract corresponding to different federated learning tasks from multiple edge servers, and selects several contracts and federated learning tasks to be executed with the goal of maximizing its own utility. S2: Use the local data of each terminal device to perform local training on the global model of the federated learning task to be executed. The local training uses the first loss function to update the parameters and obtain a local model. S3: Each terminal device uploads its own local model to the corresponding UAV base station. The UAV base station aggregates multiple local models within the device set. The model aggregation uses a second loss function to update the parameters and obtains an aggregated local model. S4: Each UAV base station sends its generated aggregated local model to its respective edge server. The edge server performs secondary aggregation on each aggregated local model. The secondary aggregation uses a third loss function to update the parameters, resulting in an updated global model.

[0009] Preferably, the preset communication system includes: An edge server One drone base station and A set of devices, denoted as... , and ; Each drone base station acts as a mobile edge node, hovering at an altitude of H, and each drone base station manages a set of devices; for the k-th set of devices... Each manages a group of terminal devices, denoted as N is the number of terminal devices within a single device set; for the nth terminal device All of them have local data set, and the local data is represented as , This indicates the amount of data in the local data. This represents the first local data sample; For the m-th edge server Record the contract he designed as ,in, The number of training rounds required for the k-th device set To provide incentives for the k-th set of devices; Each edge server is designed separately. Several different contracts, and broadcast them separately to After assemblies of devices are established, each device set selects the multiple contracts that maximize its benefit based on its total energy consumption and the maximum computing resources it can provide, and then completes the corresponding federated learning tasks according to the selected contracts; among them, using Represents the set of the kth device Total energy consumption, using Represents the set of the kth device The maximum computing resources that can be provided.

[0010] Preferably, the Specifically:

[0011]

[0012] in, This represents the required accuracy of the local model after each training iteration. and These represent the computing power consumption and computing time required by each terminal device within the k-th device set; and These represent the transmission energy consumption and transmission time required to transmit parameters for the k-th device set, respectively. and These represent the transmission energy consumption and transmission time required for the k-th UAV base station to transmit parameters, respectively. Let be the energy consumption of the k-th drone base station hovering per unit time; The energy consumption required to perform model aggregation for the kth drone base station; If the kth device set If the contract designed for the m-th edge server is selected, then the available utility is... Represented as:

[0013] in, The total energy consumption of the kth drone base station; The total utility that the m-th edge server can obtain Represented as:

[0014] in, Represents the set of the kth device The choice and decision, Represents a set of devices Edge server selected The contract designed for it, and vice versa. ; This represents the set of the k-th devices inferred by the edge server through experience. The probability of the type; For equipment collection The contribution to the edge server after obtaining the contract and completing training is represented as:

[0015] in, These are weight parameters; , representing the sensitivity of the k-th edge server to the accuracy of the aggregated local model; Represents a set of devices go through The global accuracy achieved after multiple iterations.

[0016] Preferably, in step S2, the first loss function is expressed as:

[0017] in, The value of the first loss function; Parameters of the local model It is a local model in a single data sample The loss value on.

[0018] Preferably, in step S3, the second loss function is expressed as:

[0019] in, This is the value of the second loss function. To aggregate the parameters of the local model, The total local data volume of all terminal devices within a single device set satisfies: .

[0020] Preferably, in step S4, the third loss function is expressed as:

[0021] in, This is the value of the third loss function; These are the parameters of the updated global model; The aggregate weights for each set satisfy: , The relevance weight for the k-th task is... , Let represent the similarity between the aggregated local models of the k-th device set and the j-th device set, satisfying: .

[0022] Preferably, a first objective function is constructed with the goal of maximizing its own utility, and the first objective function is expressed as follows:

[0023] st

[0024]

[0025]

[0026] Among them, constraints Represents a set of devices The maximum number of contracts that can be selected is no more than One, constraint Represents a set of devices The energy consumption required to choose multiple contracts cannot exceed the limitations of one's own computing resources, thus constraining... Represents a set of devices The choice decision can only be 0 or 1; The first objective function is solved using a dynamic programming algorithm, and several contracts and federated learning tasks to be executed are selected.

[0027] Preferably, in the edge server, the contract corresponding to the federated learning task is designed according to the following steps: To maximize the average utility of multiple task publishers, a second objective function is constructed, which is:

[0028] st

[0029]

[0030]

[0031]

[0032] in, Represents a set of devices The optimal choice made after receiving the contract from the m-th edge server. and These represent the minimum and maximum number of training rounds, respectively. The second objective function is solved by optimizing the solution using the MAPPO algorithm, and the corresponding contract is designed based on the optimization results.

[0033] Preferably, the MAPPO algorithm includes multiple action networks and a critic network; the steps of optimizing the solution of the second objective function using the MAPPO algorithm include: Each edge server is treated as an agent, generating and outputting an action policy π through an action network; in time slot t, each action network will record the system state. As input, action As output; for the agent For any given system state Define the strategy π as a mapping π: The commentator network is used to evaluate the global state value, given a global state. As input, its output is The critic network is used to compute the advantage function, which guides the agent's action generation in a better direction. It is defined as a mapping: ; For intelligent agents Its state space and action space are defined as follows:

[0034]

[0035] In the state space , indicating all The IR constraint state of a set of devices, if the agent is a set of devices. The designed contract satisfies the constraints at time t. ,but ,otherwise ; express Given a set of devices and their IC constraint states, if the agent can satisfy the constraints at time t. ,but ,otherwise ; Represents a set of devices Select edge server The intention of the contract, if the equipment set Edge server selected The contract, ,otherwise Action space middle, express Each device set has a preset global training round. It is for the collection of devices The size of the assigned task; Represented as A preset excitation vector for each device set. It is a payment to the set of devices Incentives; Each agent's action network receives system state. and generate action. At the same time, obtain the next state. and current rewards Then, the experience of each agent... The samples are stored in a preset experience pool. In each training round, samples are selected from the experience pool to update the parameters of multiple action networks and a commentator network until the average utility of multiple task publishers is maximized. The reward Calculate using the following formula: .

[0036] Preferably, in each training epoch, the parameters of multiple action networks and one critic network are updated according to the following steps: Configure and optimize the loss function of the following critic network to update the parameters of the critic network:

[0037] in, , It is the advantage function, which represents the advantage of the actual decision relative to other feasible decisions, specifically: , It is a discount factor. To represent the single-step time error, satisfying: , A scalar representing the overall real-time reward of the system; This represents the network parameters of the critics' network; This indicates that at time step The expected outcome based on a batch of collected data; Indicates the mean square error; This represents the value estimate of the state at time step t by the commentator network; Configure and optimize the loss function of the following action network to update the parameters of the action network:

[0038] in, Represents intelligent agents The ratio of the probabilities of the new and old strategies satisfies ; This is the clipping function; Represents intelligent agents The loss function of the action network.

[0039] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention provides a hierarchical federated multi-task learning method based on many-to-many contract design, applied to a pre-defined communication system. The communication system includes multiple edge servers, multiple drone base stations, and multiple terminal devices. The edge servers act as task publishers, publishing federated learning tasks and their corresponding designed contracts. The terminal devices are divided into several device sets with the goal of minimizing data distribution differences. Each device set corresponds to a drone base station, and each terminal device within a set communicates with the edge server through its corresponding drone base station. This method is used to execute federated multi-task learning. In each training round, each device set first obtains the global model and contract corresponding to different federated learning tasks from the multiple edge servers, aiming to maximize its own utility. First, several contracts and federated learning tasks to be executed are selected. Second, the global model of the federated learning tasks to be executed is locally trained using local data from each terminal device. The local training uses a first loss function to update the parameters, resulting in a local model. Then, each terminal device uploads its own local model to the corresponding UAV base station. The UAV base station aggregates multiple local models from the device set. The model aggregation uses a second loss function to update the parameters, resulting in an aggregated local model. Finally, each UAV base station sends its generated aggregated local model to its respective edge server. The edge server performs a second aggregation on each aggregated local model. The second aggregation uses a third loss function to update the parameters, resulting in an updated global model.

[0040] This invention extends the market incentive paradigm from "one-to-many" to "many-to-many." Within this framework, not only do edge device sets compete for rewards, but task publishers also fiercely compete for high-quality, limited edge computing resources, more closely resembling real-world scenarios. Furthermore, this invention uses the MAPPO algorithm to design contract incentives in complex dynamic game scenarios, transforming the contract generation process from a static optimization problem into a dynamic, adaptive learning process. This enables the incentive mechanism to endogenously produce equilibrium strategies without assuming perfect market information, significantly improving the mechanism's adaptability and robustness in complex, non-stationary, and incomplete information environments. Attached Figure Description

[0041] Figure 1 A flowchart of a hierarchical federated multi-task learning method based on many-to-many contract design in Embodiment 1; Figure 2 This is a flowchart of a federated multi-task learning method based on a hierarchical federated multi-task learning method with many-to-many contract design in Embodiment 2. Figure 3This is a flowchart of a hierarchical federated multi-task learning method based on many-to-many contract design in Example 2. Figure 4 This is a flowchart of a multi-agent proximal policy optimization method based on a hierarchical federated multi-task learning method with many-to-many contract design in Example 2. Detailed Implementation

[0042] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application. To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions; It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0043] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] Example 1 like Figure 1 As shown, this embodiment provides a hierarchical federated multi-task learning method based on many-to-many contract design. The method is applied to a preset communication system, which includes multiple edge servers, multiple UAV base stations, and multiple terminal devices. The edge servers act as task publishers, used to publish federated learning tasks and their corresponding designed contracts. The terminal devices are divided into several device sets with the goal of minimizing data distribution differences. Each device set corresponds to a UAV base station, and each terminal device in the device set communicates with the edge server through its corresponding UAV base station. The method is used to perform federated multi-task learning, and each training round includes the following steps: S1: Each set of devices obtains the global model and contract corresponding to different federated learning tasks from multiple edge servers, and selects several contracts and federated learning tasks to be executed with the goal of maximizing its own utility. S2: Use the local data of each terminal device to perform local training on the global model of the federated learning task to be executed. The local training uses the first loss function to update the parameters and obtain a local model. S3: Each terminal device uploads its own local model to the corresponding UAV base station. The UAV base station aggregates multiple local models within the device set. The model aggregation uses a second loss function to update the parameters and obtains an aggregated local model. S4: Each UAV base station sends its generated aggregated local model to its respective edge server. The edge server performs secondary aggregation on each aggregated local model. The secondary aggregation uses a third loss function to update the parameters, resulting in an updated global model.

[0045] In the specific implementation process, the first step is to construct a multi-agent network environment and initialize the system environment information, including the number of agents and the utility function of each agent. Next, a neural network is constructed, including multiple action networks and a critic network, with initialized parameters such as weights, biases, learning rate, and the number of layers. Then, each agent inputs its observed state into its action network model for training, obtaining the system actions—that is, the design of K contracts for each agent. Afterward, each agent inputs its actions (K contracts) into the environment to simulate set selection, assuming each set makes the most rational choice. The optimal choice of the set is then evaluated using a dynamic programming algorithm, and the potential utility for each agent is calculated based on the optimal choice. The average utility is used as the reward value for the actions of multiple agents. Simultaneously, the next state is obtained from the environment, and the neural network parameters are trained based on the reward value and state information. Finally, the neural network is trained using the rewards and states obtained in the above steps until the reward value stabilizes. Based on each agent's task and under the constraint of limited computing resources for the user set, the optimal contract design is obtained, achieving an optimal allocation scheme of computing resources.

[0046] Example 2 This embodiment provides a hierarchical federated multi-task learning method based on many-to-many contract design. The method is applied to a pre-defined communication system, which includes multiple edge servers, multiple UAV base stations, and multiple terminal devices. The edge servers act as task publishers, used to publish federated learning tasks and their corresponding designed contracts. The terminal devices are divided into several device sets with the goal of minimizing data distribution differences. Each device set corresponds to a UAV base station, and each terminal device in a device set communicates with the edge server through its corresponding UAV base station. The method is used to perform federated multi-task learning, and each training round includes the following steps: S1: Each set of devices obtains the global model and contract corresponding to different federated learning tasks from multiple edge servers, and selects several contracts and federated learning tasks to be executed with the goal of maximizing its own utility. S2: Use the local data of each terminal device to perform local training on the global model of the federated learning task to be executed. The local training uses the first loss function to update the parameters and obtain a local model. S3: Each terminal device uploads its own local model to the corresponding UAV base station. The UAV base station aggregates multiple local models within the device set. The model aggregation uses a second loss function to update the parameters and obtains an aggregated local model. S4: Each UAV base station sends its generated aggregated local model to its respective edge server. The edge server performs secondary aggregation on each aggregated local model. The secondary aggregation uses a third loss function to update the parameters, resulting in an updated global model.

[0047] The preset communication system includes: An edge server One drone base station and A set of devices, denoted as... , and ; Each drone base station acts as a mobile edge node, hovering at an altitude of H, and each drone base station manages a set of devices; for the k-th set of devices... Each manages a group of terminal devices, denoted as N is the number of terminal devices within a single device set; for the nth terminal device All of them have local data set, and the local data is represented as , This indicates the amount of data in the local data. This represents the first local data sample; For the m-th edge server Record the contract he designed as ,in, The number of training rounds required for the k-th device set To provide incentives for the k-th set of devices; Each edge server is designed separately. Several different contracts, and broadcast them separately to After assemblies of devices are established, each device set selects the multiple contracts that maximize its benefit based on its total energy consumption and the maximum computing resources it can provide, and then completes the corresponding federated learning tasks according to the selected contracts; among them, using Represents the set of the kth device Total energy consumption, using Represents the set of the kth device The maximum computing resources that can be provided.

[0048] The Specifically:

[0049]

[0050] in, This represents the required accuracy of the local model after each training iteration. and These represent the computing power consumption and computing time required by each terminal device within the k-th device set; and These represent the transmission energy consumption and transmission time required to transmit parameters for the k-th device set, respectively. and These represent the transmission energy consumption and transmission time required for the k-th UAV base station to transmit parameters, respectively. Let be the energy consumption of the k-th drone base station hovering per unit time; The energy consumption required to perform model aggregation for the kth drone base station; If the kth device set If the contract designed for the m-th edge server is selected, then the available utility is... Represented as:

[0051] in, The total energy consumption of the kth drone base station; The total utility that the m-th edge server can obtain Represented as:

[0052] in, Represents the set of the kth device The choice and decision, Represents a set of devices Edge server selected The contract designed for it, and vice versa. ; This represents the set of the k-th devices inferred by the edge server through experience. The probability of the type; For equipment collection The contribution to the edge server after obtaining the contract and completing training is represented as:

[0053] in, These are weight parameters; , representing the sensitivity of the k-th edge server to the accuracy of the aggregated local model; Represents a set of devices go through The global accuracy achieved after multiple iterations.

[0054] In step S2, the first loss function is expressed as:

[0055] in, The value of the first loss function; Parameters of the local model It is a local model in a single data sample The loss value on.

[0056] In step S3, the second loss function is expressed as:

[0057] in, This is the value of the second loss function. To aggregate the parameters of the local model, The total local data volume of all terminal devices within a single device set satisfies: .

[0058] In step S4, the third loss function is expressed as:

[0059] in, This is the value of the third loss function; These are the parameters of the updated global model; The aggregate weights for each set satisfy: , The relevance weight for the k-th task is... , Let represent the similarity between the aggregated local models of the k-th device set and the j-th device set, satisfying: .

[0060] The first objective function is constructed with the goal of maximizing its own utility. The first objective function is expressed as follows:

[0061] st

[0062]

[0063]

[0064] Among them, constraints Represents a set of devices The maximum number of contracts that can be selected is no more than One, constraint Represents a set of devices The energy consumption required to choose multiple contracts cannot exceed the limitations of one's own computing resources, thus constraining... Represents a set of devices The choice decision can only be 0 or 1; The first objective function is solved using a dynamic programming algorithm, and several contracts and federated learning tasks to be executed are selected.

[0065] In the edge server, the contract corresponding to the federated learning task is designed according to the following steps: To maximize the average utility of multiple task publishers, a second objective function is constructed, which is:

[0066] st

[0067]

[0068]

[0069]

[0070] in, Represents a set of devices The optimal choice made after receiving the contract from the m-th edge server. and These represent the minimum and maximum number of training rounds, respectively. The second objective function is solved by optimizing the solution using the MAPPO algorithm, and the corresponding contract is designed based on the optimization results.

[0071] The MAPPO algorithm includes multiple action networks and a critic network; the steps for optimizing the second objective function using the MAPPO algorithm include: Each edge server is treated as an agent, generating and outputting an action policy π through an action network; in time slot t, each action network will record the system state. As input, action As output; for the agent For any given system state Define the strategy π as a mapping π: The commentator network is used to evaluate the global state value, given a global state. As input, its output is The critic network is used to compute the advantage function, which guides the agent's action generation in a better direction. It is defined as a mapping: ; For intelligent agents Its state space and action space are defined as follows:

[0072]

[0073] In the state space , indicating all The IR constraint state of a set of devices, if the agent is a set of devices. The designed contract satisfies the constraints at time t. ,but ,otherwise ; express Given a set of devices and their IC constraint states, if the agent can satisfy the constraints at time t. ,but ,otherwise ; Represents a set of devices Select edge server The intention of the contract, if the equipment set Edge server selected The contract, ,otherwise Action space middle, express Each device set has a preset global training round. It is for the collection of devices The size of the assigned task; Represented as A preset excitation vector for each device set. It is a payment to the set of devices Incentives; Each agent's action network receives system state. and generate action. At the same time, obtain the next state. and current rewards Then, the experience of each agent... The samples are stored in a preset experience pool. In each training round, samples are selected from the experience pool to update the parameters of multiple action networks and a commentator network until the average utility of multiple task publishers is maximized. The reward Calculate using the following formula: .

[0074] In each training epoch, the parameters of multiple action networks and one critic network are updated according to the following steps: Configure and optimize the loss function of the following critic network to update the parameters of the critic network:

[0075] in, , It is the advantage function, which represents the advantage of the actual decision relative to other feasible decisions, specifically: , It is a discount factor. To represent the single-step time error, satisfying: , A scalar representing the overall real-time reward of the system; This represents the network parameters of the critics' network; This indicates that at time step The expected outcome based on a batch of collected data; Indicates the mean square error; This represents the value estimate of the state at time step t by the commentator network; Configure and optimize the loss function of the following action network to update the parameters of the action network:

[0076] in, Represents intelligent agents The ratio of the probabilities of the new and old strategies satisfies ; This is the clipping function; Represents intelligent agents The loss function of the action network.

[0077] In the specific implementation process, the federal multi-task learning flowchart is as follows: Figure 2 As shown, the communication system for the federated multiple task learning method designed in this invention includes a three-layer architecture: task publisher (edge ​​server), mobile base station (drone), and terminal device, wherein, An edge server One drone base station and A set of devices, denoted as... , and Each drone base station acts as a mobile edge node, hovering at an altitude of H, and each drone base station manages a set of devices; for the k-th set of devices... Each manages a group of terminal devices, denoted as N is the number of terminal devices within a single device set; for the nth terminal device All of them have local data set, and the local data is represented as , This indicates the amount of data in the local data. This represents the first local data sample.

[0078] Then, based on data similarity, the terminal devices are divided into multiple sets, each set being treated as a separate federated learning task. The terminal devices first undergo multiple local training iterations, with each smart device belonging to only one set. Represents each smart device The local model parameters are determined by the selected client downloading the latest global model from the server at the start of each global iteration and then updating the model using multi-step gradient descent on the local data. During the training rounds... In this model, each smart device uses its local data to update a local model, and its loss can be represented by the first loss function:

[0079] in The model is in a single data sample The losses on This refers to the amount of data possessed by the smart devices. The local model update process is completed independently on each smart device, providing iteratively updated model parameters for subsequent model aggregation within the set, until the local model reaches a certain level of accuracy. Then, the model parameters are passed to the drone.

[0080] Then the drone aggregates models based on the local model parameters of each smart device, using... To represent the parameters of the aggregation model, the set The loss can be expressed by the second loss function as:

[0081] in, It represents the total amount of data from all intelligent terminals within the set. The drone obtains the aggregated local model based on the steps described above.

[0082] After the drones complete model aggregation within the aforementioned set, an aggregated local model is obtained. Each drone transmits the parameters of this aggregated local model to the edge server (i.e., the task publisher). The server receives parameters from multiple sets and performs secondary model aggregation. Considering that each set performs a different federated learning task, the server needs to consider the correlation between tasks to adjust the global parameters when aggregating different model parameters. We use... To represent the model parameters after server aggregation, the global loss formula for federated multi-task learning can be expressed by a third loss function as follows:

[0083] in, This represents the aggregate weight for each set, which is related to the task's relevance. and data volume related. The relevance weight of the task. , Represents a set With sets We use cosine similarity to express the model similarity: The updated global model can be obtained through the above steps.

[0084] Many-to-many contract design flowchart as follows Figure 3 As shown, to incentivize different types of sets to participate in federated learning tasks, each task publisher designs differentiated contracts for different sets. We take the task publisher as an example. For example, the contract it designs is ,in The number of global iterations required for the set. To provide incentives for the ensemble, the contract design process must satisfy individual rationality (IR) and incentive compatibility (IC) constraints. The IR constraint states that each ensemble, after selecting a contract designed for it, must obtain a net utility greater than or equal to zero. Otherwise, the ensemble will have no incentive to participate in training. The IC constraint states that when an ensemble selects a contract of its own type, the utility it obtains must be greater than or equal to the utility it would obtain by "disguising" itself as other energy types and selecting other contracts. This constraint ensures that each ensemble honestly selects a contract tailored to its own needs, preventing "adverse selection." Assume each publisher designs... A contract, and broadcast it to After a set, the set is determined according to its own energy consumption type. and computing resource limitations This involves selecting multiple contracts that maximize one's profits and then fulfilling the task requirements based on those selected contracts. Represents a set The type of energy consumption it belongs to. , Represents a set The maximum computing resources that can be provided.

[0085] Each ensemble, after selecting a contract, needs to consume its own computing resources to participate in task training. The energy consumption of an ensemble is divided into local computing energy consumption, ensemble communication energy consumption, and drone energy consumption, etc. For example, each set has Each end user The local dataset size is and use This represents the average CPU cycle frequency of users within the set, using... This represents the number of CPU cycles consumed by a user device to train one unit of data. This reflects the CPU's energy efficiency characteristics.

[0086] Represents a set The calculation formula for the computing power required by user equipment in the process is as follows:

[0087] Represents a set The calculation formula for the computing time required by the user equipment in the process is as follows:

[0088] Each smart device needs to consume a certain amount of time and energy to transmit the trained model. The time required for a smart device to transmit model parameters is represented by the following formula:

[0089] in, Represents a set Average transmission power, For channel gain, For noise power density, The average bandwidth allocated to users, This refers to the total number of model parameters that the smart device needs to transmit.

[0090] Based on the transmission time of the model parameters, the total transmission energy required for the set can be calculated as follows: :

[0091] Furthermore, the drone acts as a mobile base station in each global iteration, primarily responsible for transmitting and aggregating model parameters, a process that also consumes energy. Therefore, we use Come These represent the transmission energy consumption and transmission latency of the drone in each global iteration, respectively. The calculation formula is:

[0092] in, Indicates the transmission power of the drone. For channel gain, For noise power density, For the bandwidth allocated to the drone, This refers to the total number of model parameters transmitted from the drone to the user.

[0093] Based on the above The required energy consumption can be calculated as follows:

[0094] In each global iteration, smart devices When training locally, the model is required to achieve an accuracy of [a certain percentage]. Only then can it be uploaded to the drone for aggregation. To ensure the accuracy of the local model meets expectations, the expected number of local iterations per training session is [number missing]. Meanwhile, we define the energy consumption of a drone hovering per unit time as... drones The energy consumption required to aggregate local model parameters from all smart devices in the set in each iteration is: Then perform a global iteration and set The total energy required is:

[0095] To attract different types of assemblies to participate in federated multi-task learning, we set The energy consumption required to perform one global iteration on a set is used to represent the set's willingness to participate in the training task. We categorize the set's energy consumption into... The classes are sorted in ascending order, that is:

[0096] Although the task issuer cannot directly know the exact energy consumption of each set, he can infer the probability that each set belongs to a certain type through experience. Here, we use This represents the probability that each set belongs to a certain type.

[0097] Due to information asymmetry between the task publisher and the set, the intelligent agent Specific contracts are provided for different types of sets, represented as ,in The number of global iterations required for the set. This is the incentive provided to the set.

[0098] Meanwhile, since drones require a certain amount of energy for transmission and hovering when transferring models to the server, therefore, the excitation... In addition to meeting the energy requirements for iteration, compensation is also needed for this part of the energy consumption. We use This represents the energy consumption compensation required for each drone, where both of these energy consumption components are related to the transmission time. The relevant calculation formula is as follows:

[0099] in, This indicates the drone's receiving power. For channel gain, For noise power density, For the bandwidth from the drone to the server, This represents the total number of model parameters transmitted by the drone to the base station.

[0100] Based on the aforementioned transmission time, The calculation formula is as follows:

[0101] Based on the contract design described above, if the set Select Task Publisher The formula for calculating the utility that can be obtained from a contract is as follows:

[0102] At the same time, we set This refers to the contribution to the task issuer after the set has obtained the contract and completed training. This represents the task publisher's sensitivity to the precision of the set. These are the corresponding weight parameters. Indicates the set has passed through The global accuracy achieved after rounds of global iteration is given by the formula. It is concluded that, among them This represents the accuracy achieved during each iteration of local training. This reflects the increasing trend in accuracy of the global iteration count.

[0103] Because there are multiple task publishers, and each set has limited computational resources, situations arise where a publisher publishes a contract, but the set cannot choose. Due to computational resource constraints, the set has the right to choose a better contract to obtain higher utility; therefore, parameters are defined. To represent a set The choice decision, if Then it represents a set Select Publisher The contract designed for it, and vice versa. ,and, .

[0104] According to the formula above, if the task publisher... Designed The utility that can be obtained from this contract is:

[0105] Based on the above theoretical derivation, the set The goal is to purposefully select multiple contracts to maximize one's utility under limited computational resources. Its optimization equation can be represented by the first objective function:

[0106] st

[0107]

[0108]

[0109] Among them, constraints Represents a set The maximum number of contracts that can be selected is no more than One. Constraints Represents a set The energy cost of choosing multiple contracts cannot exceed the limitations of one's own computing resources. Represents a set The maximum computing resources that can be provided. Represents a set The choice decision can only be 0 or 1.

[0110] We use To represent a set Received The optimal choice made after the contract is signed is determined by dynamic programming, an effective method for solving such problems. Therefore, dynamic programming can be used to find the optimal choice. Publishers cannot know in advance what choice the set will make, but they can evaluate the optimal choice based on the contract designs of other publishers. Furthermore, the utility calculation will change accordingly based on the evaluated set's choice. In this many-to-many contract design scenario, our goal is to rationally allocate the set's computational resources to maximize the utility of multiple publishers under reasonable contract design. Therefore, this invention constructs the optimization objective as maximizing the average utility of multiple task publishers, represented by a second objective function:

[0111] st

[0112]

[0113]

[0114]

[0115] Among them, constraints The Individual Rationality (IR) constraint requires that each set, after selecting a contract designed for it, must obtain a net utility greater than or equal to zero. Otherwise, the set will have no incentive to participate in training. To incentivize compatibility (IC) constraints, it requires that the utility a set obtains when choosing its own contract must be greater than or equal to the utility it would obtain if it "disguised" itself as any other type and chose any other contract. This constraint ensures that each set will honestly choose a contract tailored to its own needs, preventing "adverse selection". This means that when designing a contract, the total incentives provided by the publisher cannot exceed the maximum incentives they are capable of providing. . This indicates that the global iterations required by the contract during design must satisfy a certain range. , This represents the minimum precision or maximum loss that the set must satisfy.

[0116] The flowchart for multi-agent proximal policy optimization is as follows: Figure 4 As shown, the above problem involves two levels of optimization problems, namely, set optimization. This paper addresses the problem of how to purposefully select multiple contracts to maximize one's own utility under limited computational resources, and how to maximize the average utility of multiple task publishers, with both optimization objectives belonging to a dynamic game process. The optimal contract design is the Nash equilibrium solution of the dynamic game between the two parties, which is difficult to solve using ordinary methods. Therefore, reinforcement learning combined with contract design is used to solve this type of problem. Furthermore, since multiple publishers need to design contracts simultaneously, traditional single-agent reinforcement learning is insufficient. Therefore, to solve the above many-to-many optimal resource allocation problem, a multi-agent reinforcement learning method is used to implement contract design for multiple task parties. Since the contract design process is in a continuous space, traditional Q-learning and DQN methods suffer from the curse of dimensionality and are therefore unsuitable for this contract design. The MAPPO algorithm used in this invention has achieved breakthrough results in solving many challenges in various fields with large action continuous spaces, and has great potential in solving the many-to-many contract design problem in this invention. The Multi-Agent Proximal Policy Optimization Algorithm (MAPPO) mainly consists of multiple actor networks and a centralized critic network. Each task publisher acts as an agent, generating an action policy π through the action network. In time slot t, each action network will state the system state. As input, action As output. In the form of an intelligent agent. For any given system state Define the strategy π as a mapping π: For a centralized commentator network, the evaluation is based on the global state value, given a global state. As input, its output is The purpose of the critic network is to provide an accurate and stable global baseline for calculating the advantage function, thereby guiding the agent's action generation in a better direction. This can be defined as a mapping: .

[0117] In this system, each server, acting as a task publisher, triggers an algorithm based on its own task requirements and information from other agents. At time t, the action network of each agent receives the system state. and generate action. At the same time, obtain the next state. and rewards Then, we will analyze the experience of each agent. The samples are stored in the experience pool. During each training phase, we select samples from the experience pool to update the parameters of multiple action networks and a centralized critic network until the long-term reward is maximized.

[0118] In our scenario, the immediate reward function of the Markov process The average utility obtained after designing contracts for all agents, i.e. Therefore, the goal of this algorithm is to train a group of agents that can cooperate efficiently to maximize the long-term shared rewards obtained by the entire team. Here we take agents as an example. For example, the state space and action space of an intelligent agent can be defined as follows:

[0119]

[0120] In the state space , indicating all The IR constraint states of a set, if the agent is a set If the designed contract satisfies the IR constraint at time t, then... ,otherwise . Let represent the IC constraint state of the set. If the agent can satisfy the IC constraint at time t, then ,otherwise .also, Represents a set Select Publisher The intention of the contract, if the collection Select Publisher The contract, ,otherwise Regarding the action space In other words, Represented as The number of global iterations for a set design. It is for a set The size of the assigned task; Represented as The incentive-reward vector of a set design It is payment to the collection remuneration.

[0121] For each intelligent agent, it first generates based on environmental information. A contract is generated and input into the environment. The environment then determines whether the designed contract satisfies the IR (Individual Rationality) constraint and IC (Incentive Compatibility) constraint. If not, a negative incentive is returned; if satisfied, the contract is passed on to the environment. There are several sets. Each set receives multiple contracts from agents, and each agent, using a dynamic programming algorithm, selects the contracts that maximize its utility within its resource constraints, feeding back its selection intentions to the environment. The environment then simulates and calculates the utility of each agent based on the set's selections, and feeds back the average utility as an environmental reward to the agents.

[0122] Following the above iterative steps, in each iteration the agent will accumulate experience. The experience is stored in an experience pool. When certain conditions are met during iteration, the agent will draw experience from the pool to update multiple Actor networks and a centralized Critic network. The update of the Critic network can be represented as... ,in, This represents the loss function of the Critic network. These are the network parameters for Critic. Indicates at time step The expectation on a batch of collected data, and For the target value formula, This refers to the value estimation of the state at time step t using the Critic network. This represents the mean square error. , This represents the old state value estimate. It is an estimate of the advantage function, expressed as the advantage of a decision relative to other feasible decisions. Its calculation formula can be expressed as: It is a discount factor. It is a unique trade-off parameter in the dominance function. The single-step time error can be expressed as: , A scalar representing the overall immediate reward of the system, that is, the sum of the rewards of multiple agents at each time step. Take action Then, the average value of the "instant reward" obtained from the environment.

[0123] For updates to Actors, we use intelligent agents. For example, it can be represented as: ,in Represents intelligent agents The loss function of the Actor network, Its network parameters. Indicates at time step Expectations based on a batch of collected data, This indicates taking the minimum of the two values. Represents intelligent agents The corresponding importance sampling ratio is calculated as follows: , in For intelligent agents In state Select action The probability of. This represents the probability that the old Actor network used to collect data will choose the same action under the same conditions. It is the dominant function. For the clipping function, Limit within confidence interval This is to ensure the adaptability and stability of the action network.

[0124] By optimizing these two loss functions to update the parameters of the two networks, the optimal policy is learned, thus completing the training process. The computational resources of the set are then allocated reasonably based on the contract generated by each agent.

[0125] The same or similar labels correspond to the same or similar parts; The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this application. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A hierarchical federated multi-task learning method based on many-to-many contract design, characterized in that, The method is applied to a pre-defined communication system, which includes multiple edge servers, multiple drone base stations, and multiple terminal devices. The edge servers act as task publishers, used to publish federated learning tasks and their corresponding contract designs. The terminal devices are divided into multiple device sets with the goal of minimizing data distribution differences. Each device set corresponds to a drone base station, and each terminal device in the device set communicates with the edge server through its corresponding drone base station. The method is used to perform federated multi-task learning, and each training round includes the following steps: S1: Each set of devices obtains the global model and contract corresponding to different federated learning tasks from multiple edge servers, and selects one or more contracts and federated learning tasks to be executed with the goal of maximizing its own utility. S2: Use the local data of each terminal device to perform local training on the global model of the federated learning task to be executed. The local training uses the first loss function to update the parameters and obtain a local model. S3: Each terminal device uploads its own local model to the corresponding UAV base station. The UAV base station aggregates multiple local models within the device set. The model aggregation uses a second loss function to update the parameters and obtains an aggregated local model. S4: Each UAV base station sends its generated aggregated local model to each edge server. The edge server performs secondary aggregation on each aggregated local model. The secondary aggregation uses a third loss function to update the parameters and obtain the updated global model. The kth device set Total energy consumption Specifically: in, This represents the required accuracy of the local model after each training iteration. and These represent the computing power consumption and computing time required by each terminal device within the k-th device set; and These are the transmission energy consumption and transmission time required to transmit parameters for the k-th device set, respectively. and These represent the transmission energy consumption and transmission time required for the k-th UAV base station to transmit parameters, respectively. Let be the energy consumption of the k-th drone base station hovering per unit time; The energy consumption required to perform model aggregation for the kth drone base station; If the kth device set If the contract designed for the m-th edge server is selected, then the available utility is... Represented as: in, The total energy consumption of the kth drone base station; The total utility that the m-th edge server can obtain Represented as: in, Represents the set of the kth device The choice and decision, Represents a set of devices Edge server selected The contract designed for it, and vice versa. ; This represents the set of the k-th devices inferred by the edge server through experience. The probability of the type; For equipment collection The contribution to the edge server after obtaining the contract and completing training is represented as: in, These are weight parameters; , representing the sensitivity of the k-th edge server to the accuracy of the aggregated local model; Represents a set of devices go through The global accuracy achieved after multiple iterations.

2. The hierarchical federated multi-task learning method based on many-to-many contract design according to claim 1, characterized in that, The pre-defined communication system includes: An edge server One drone base station and A set of devices, denoted as... , and ; Each drone base station acts as a mobile edge node, hovering at an altitude of H, and each drone base station manages a set of devices; for the k-th set of devices... Each manages a group of terminal devices, denoted as N is the number of terminal devices within a single device set; for the nth terminal device All of them have local data set, and the local data is represented as , This indicates the amount of data in the local data. This represents the first local data sample; For the m-th edge server Record the contract he designed as ,in, The number of training rounds required for the k-th device set is: To provide incentives for the k-th set of devices; Each edge server is designed separately. Two different contracts, and broadcast them separately to After assemblies of devices are established, each device set selects the multiple contracts that maximize its benefit based on its total energy consumption and the maximum computing resources it can provide, and then completes the corresponding federated learning tasks according to the selected contracts; among them, using Represents the set of the kth device Total energy consumption, using Represents the set of the kth device The maximum computing resources that can be provided.

3. The hierarchical federated multi-task learning method based on many-to-many contract design according to claim 1, characterized in that, In step S2, the first loss function is expressed as: in, The value of the first loss function; Parameters of the local model It is a local model in a single data sample The loss value on.

4. The hierarchical federated multi-task learning method based on many-to-many contract design according to claim 1, characterized in that, In step S3, the second loss function is expressed as: in, This is the value of the second loss function. To aggregate the parameters of the local model, The total local data volume of all terminal devices within a single device set satisfies: .

5. The hierarchical federated multi-task learning method based on many-to-many contract design according to claim 1, characterized in that, In step S4, the third loss function is expressed as: in, This is the value of the third loss function; These are the parameters of the updated global model; The aggregate weights for each set satisfy: , The relevance weight of the k-th task. , Let represent the similarity between the aggregated local models of the k-th device set and the j-th device set, satisfying: .

6. The hierarchical federated multi-task learning method based on many-to-many contract design according to claim 2, characterized in that, The first objective function is constructed with the goal of maximizing its own utility. The first objective function is expressed as follows: s. t. Among them, constraints Represents a set of devices The maximum number of contracts that can be selected is no more than One, constraint Represents a set of devices The energy consumption required to choose multiple contracts cannot exceed the limitations of one's own computing resources, thus constraining... Represents a set of devices The choice decision can only be 0 or 1; The first objective function is solved using a dynamic programming algorithm, and one or more contracts and federated learning tasks to be executed are selected.

7. A hierarchical federated multi-task learning method based on many-to-many contract design according to claim 2, characterized in that, In the edge server, the contract corresponding to the federated learning task is designed according to the following steps: To maximize the average utility of multiple task publishers, a second objective function is constructed, which is: s. t. in, Represents a set of devices The optimal choice made after receiving the contract from the m-th edge server. and These represent the minimum and maximum number of training rounds, respectively. The second objective function is solved by optimizing the solution using the MAPPO algorithm, and the corresponding contract is designed based on the optimization results.

8. A hierarchical federated multi-task learning method based on many-to-many contract design according to claim 7, characterized in that, The MAPPO algorithm includes multiple action networks and a critic network; the steps for optimizing the second objective function using the MAPPO algorithm include: Each edge server is treated as an agent, generating and outputting an action policy π through an action network; in time slot t, each action network will record the system state. As input, action As output; for the agent For any given system state Define the strategy π as a mapping π: The commentator network is used to evaluate the global state value, given a global state. As input, its output is The critic network is used to compute the advantage function, thereby guiding the agent's action generation in a better direction, and is defined as a mapping: ; For intelligent agents Its state space and action space are defined as follows: In the state space , indicating all The IR constraint state of a set of devices, if the agent is a set of devices. The designed contract satisfies the constraints at time t. ,but ,otherwise ; express Given a set of devices and their IC constraint states, if the agent can satisfy the constraints at time t. ,but ,otherwise ; Represents a set of devices Select edge server The intention of the contract, if the equipment set Edge server selected The contract, ,otherwise Action space middle, express Each device set has a preset global training round. It is for the collection of devices The size of the assigned task; Represented as A preset excitation vector for each device set. It is a payment to the set of devices Incentives; Each agent's action network receives system state. and generate action. At the same time, obtain the next state. and current rewards Then, the experience of each agent... The samples are stored in a preset experience pool. In each training round, samples are selected from the experience pool to update the parameters of multiple action networks and a commentator network until the average utility of multiple task publishers is maximized. The reward Calculate using the following formula: .

9. A hierarchical federated multi-task learning method based on many-to-many contract design according to claim 8, characterized in that, In each training epoch, the parameters of multiple action networks and one critic network are updated according to the following steps: Configure and optimize the loss function of the following critic network to update the parameters of the critic network: in, , It is the advantage function, which represents the advantage of the actual decision relative to other feasible decisions, specifically: , It is a discount factor. To represent the single-step time error, satisfying: , A scalar representing the overall real-time reward of the system; This represents the network parameters of the critics' network; This indicates that at time step The expected outcome based on a batch of collected data; Indicates the mean square error; This represents the value estimate of the state at time step t by the commentator network; Configure and optimize the loss function of the following action network to update the parameters of the action network: in, Represents intelligent agents The ratio of the probabilities of the new and old strategies satisfies ; This is the clipping function; Represents intelligent agents The loss function of the action network.

Citation Information

Patent Citations

  • Multitask wireless federated learning method based on contract theory

    CN115081676A

  • Federal edge learning scheduling method and system based on game and multi-dimensional contract

    CN114677030A