A federated learning method and system based on reinforcement learning subnetwork allocation and residual accumulation

Abstract

This invention relates to a federated learning method and system based on reinforcement learning for subnetwork allocation and residual accumulation, belonging to the field of federated learning technology. The method includes: predefining a resilient architecture supernetwork and generating a subnetwork configuration set; acquiring the resource status and data characteristics of each client; optimizing using a near-end policy to output the optimal subnetwork architecture; extracting corresponding weights from the supernetwork according to the selected architecture and sending them to the clients; training the received subnetworks using a local dataset and calculating gradients; adding the residuals from the previous round to the current gradient to achieve knowledge inheritance; dynamically selecting important gradients based on the current bandwidth and updating the residuals; calculating rewards and uploading sparse gradients and state transition experience to the server; updating the supernetwork weighted by data volume, and simultaneously updating the reinforcement learning policy network parameters based on experience. This invention achieves resource-adaptive dynamic subnetwork allocation and cross-architecture knowledge continuity, improving training efficiency and personalized model accuracy.

Description

Technical Field

[0001] This invention relates to a federated learning method and system based on reinforcement learning for subnetwork allocation and residual accumulation, belonging to the field of federated learning technology. Background Technology

[0002] Federated learning, as a distributed machine learning paradigm, allows multiple clients to collaboratively train models without sharing raw data, offering significant advantages in privacy protection. However, in real-world deployment environments, client devices vary widely, including high-performance servers, ordinary PCs, smartphones, tablets, and resource-constrained IoT sensors. These devices naturally differ in computing power (CPU / GPU), memory capacity, network bandwidth, and battery life. Furthermore, the local data possessed by different clients also varies: user behavior, data acquisition environment, device sensor accuracy, and data labeling quality all contribute to the fact that the data distribution across clients is often non-identically independent (Non-IID). These real-world factors present two fundamental challenges to the practical application of federated learning: system heterogeneity and data heterogeneity.

[0003] System heterogeneity manifests in the fact that resource-rich clients can quickly complete local training, while resource-constrained clients are prone to becoming "lagging behind," slowing down the overall training process and even exiting due to timeouts. Traditional federated learning algorithms force all clients to use a uniform global model. This "one-size-fits-all" approach leads to resource-rich devices being severely idle, while resource-constrained devices are unable to complete training due to the excessive size of the model, thus impairing training efficiency and fairness.

[0004] Data heterogeneity manifests as significant differences in the distribution of data categories across different clients; for example, some clients primarily use "cat" images, while others primarily use "dog" images. This distributional difference means that a single global model cannot adapt to all clients simultaneously, thus requiring the training of a personalized model for each client. However, the sub-network allocation mechanism introduced to achieve personalization brings new problems: when a client is assigned a sub-network with a different structure in each round due to resource changes, the knowledge learned in the old sub-network cannot be transferred to the new sub-network, resulting in knowledge forgetting. Existing methods lack a cross-sub-network knowledge inheritance mechanism, making it difficult to maintain knowledge continuity under dynamic architectures.

[0005] To address these issues, existing research attempts to solve them from different perspectives. One approach adapts to system heterogeneity by assigning subnetworks of different sizes to different clients; however, these assignment strategies are typically static or based on simple heuristics, failing to respond to real-time resource fluctuations and training dynamics. Another approach uses reinforcement learning to learn subnetwork assignment strategies, but often employs a one-time architecture allocation, determining the subnetworks in the early stages of training and then leaving them unchanged, lacking consideration for maintaining continuity of knowledge when switching between architectures.

[0006] Furthermore, most methods addressing data heterogeneity assume a fixed model architecture and fail to consider the knowledge forgetting problem caused by changes in subnetworks. Existing methods lack an integrated solution that simultaneously addresses system heterogeneity, data heterogeneity, and ensures knowledge continuity. Therefore, how to dynamically allocate suitable subnetwork structures to heterogeneous clients, effectively inherit and retain historical knowledge across different subnetworks, avoid catastrophic forgetting in Non-IID data, and reduce communication overhead are pressing technical challenges in the field of personalized federated learning. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation. The technical problem this invention aims to solve is as follows: In the actual deployment of personalized federated learning, there are significant differences in the computing power, memory, and network bandwidth of different client devices, and the category distribution of local data also varies among clients (for example, some clients mainly use "cat" images, while others mainly use "dog" images). This system heterogeneity and data heterogeneity lead to two key issues: First, existing subnetwork allocation strategies are usually static or based on simple rules, which cannot be dynamically adjusted according to the real-time resource status of the client. This results in resource-sufficient devices being idle and resource-constrained devices falling behind due to excessively large models, leading to low training efficiency. Second, when a client is assigned a subnetwork with a different structure due to resource changes, the knowledge learned in the old subnetwork cannot be transferred to the new subnetwork, resulting in knowledge forgetting and a decline in the performance of the personalized model. Existing methods lack a cross-subnetwork knowledge inheritance mechanism and also lack means to retain complete knowledge while reducing communication overhead.

[0008] To overcome the above challenges, this invention proposes a federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation, which realizes resource-adaptive dynamic allocation of subnetworks and cross-architecture knowledge continuity, thereby improving training efficiency and personalized model accuracy.

[0009] This invention provides a federated learning system based on reinforcement learning for subnetwork allocation and residual accumulation.

[0010] The technical solution of this invention is as follows: A federated learning method based on reinforcement learning for subnetwork assignment and residual accumulation is implemented through several federated learning clients and a central server, including: Enables the pre-definition of resilient architecture supernetworks and the generation of subnetwork configuration sets; Obtain the resource status and data characteristics of each client; Optimize using a near-end strategy and output the optimal sub-network architecture based on the client state; Extract the corresponding weights from the supernetwork based on the selected architecture and send them to the client; The received subnetwork is trained using the local dataset, and gradients are calculated. The residual from the previous round is added to the current gradient to achieve knowledge inheritance; Dynamically select important gradients based on the current bandwidth and update the residuals; Calculate the reward and upload the sparse gradient and state transition experience to the server; The server receives sparse gradients and empirical data uploaded by the client, updates the super network by weighting the data volume, and updates the network parameters of the reinforcement learning strategy based on experience.

[0011] Further preferred, assuming a total of Each federated learning client and a central server record the clients. The local data distribution is Client Maintain a from Private local datasets extracted independently and identically distributed The data volume is denoted as , Client Different resource configurations ,in Indicates computing power. Indicates memory capacity. Indicates network bandwidth; The server maintains a super network weight with a resilient architecture. and predefine subnetwork configuration sets. Train a personalized sub-network model for each client, adapted to its local data distribution and resource constraints, while ensuring continuous knowledge inheritance when switching between different sub-networks; formally, this is an optimization problem that minimizes the weighted expected loss of all clients. ; In the formula Indicates the client The amount of data, Indicates the client The amount of data, Represents the local loss function. , Indicates based on sub-network architecture From the super network weight Extract the corresponding weights .

[0012] According to a preferred embodiment of the present invention, the pre-definition of a resilient architecture supernetwork and the generation of subnetwork configuration sets are implemented, including: Construct a set of sub-network configurations; the server predefines a resilient supernetwork with ResNet as its backbone. and set the width scaling factor set. Depth vector set Convolution kernel size vector set By iterating through the possible combinations of values ​​in the three dimensions, all possible sub-network architectures are generated, forming a sub-network configuration set. : in, The function is an extraction function, and the input is an elastic hypernetwork. and architecture tuple The output is the corresponding subnetwork structure; This is the channel scaling factor; It is a depth vector; The kernel size vector is used for extraction; the extraction function is based on the architecture tuple. From Elastic Hypernetwork The corresponding parameter subset is retrieved by index and assembled into the tuple of this architecture. Subnetworks below.

[0013] According to a preferred embodiment of the present invention, obtaining the resource status and data characteristics of each client includes: In each communication round The server randomly samples a subset of clients. For each selected client The server obtains the client's information. Current state Current state Including resource status and local data characteristics , recorded as Among them, resource status , , , These represent computing power, memory capacity, and network bandwidth, respectively; local data characteristics. This is a statistical description vector for the client's local dataset. For classification tasks, it is a histogram of the number of samples in each category, i.e., the proportion of samples in each category.

[0014] According to a preferred embodiment of the present invention, a near-end strategy optimization is adopted to output the optimal sub-network architecture based on the client state; including: The server utilizes a reinforcement learning policy network based on proximal policy optimization. The client's state Mapped to a sub-network architecture ,Right now Specifically, this includes: reinforcement learning strategy networks. Based on the client's state As input, the input vector is mapped through a multi-layer fully connected neural network, and the output is a... A logits vector of dimension, where The total number of centralized architectures configured for the sub-network; this logits vector is normalized to a probability distribution using the Softmax function. ,in, Indicates the selection of the first The probability of each sub-network architecture satisfies The server follows this probability distribution Random sampling is performed to obtain the selected sub-network architecture. ; The optimization objective of near-end policy optimization is to maximize the alternative objective function after pruning. This alternative objective function avoids destructive updates by limiting the probability ratio of the new and old policies, and combines advantage function estimation to guide the policy to iterate towards a better direction. Before outputting actions, a hard resource constraint is enforced: if the memory required by the recommended subnetwork exceeds the current available memory on the client. Then it will be automatically replaced with the subnet configuration set. The subnet that best matches the client's resources and does not exceed its capacity.

[0015] According to a preferred embodiment of the present invention, extracting corresponding weights from the supernetwork based on the selected architecture and sending them to the client includes: The server depends on the selected subnetwork architecture. From the super network Extract the corresponding subnetwork weights and will Send to the client Includes: the architecture of each sub-network Corresponding to a fixed subset of indices in the global parameter space of the super network, the server directly from Extract the parameters from these index positions and assemble them into sub-network weights. Parameters that are not retrieved are retained in the super network and do not participate in the local training of the client.

[0016] According to a preferred embodiment of the present invention, training the received sub-network using a local dataset and calculating the gradient includes: Client Receive subnetwork weights Then, in the local dataset Perform several rounds of local iterative training to calculate the original gradient. ( , );in, For loss function, This indicates that the gradient of the loss function is calculated. This is the original gradient vector obtained.

[0017] According to a preferred embodiment of the present invention, the residual from the previous round is added to the current gradient to achieve knowledge inheritance; including: The client stores the residual vector from the previous round. Add the current gradient to obtain the compensated gradient. : .

[0018] According to a preferred embodiment of the present invention, the important gradient is dynamically selected based on the current bandwidth, and the residual is updated; including: The client depends on the current network bandwidth. Dynamically determine the number of gradients to retain : ; in For sub-networks The total number of parameters, The preset maximum bandwidth reference value; from the compensated gradient Select the one with the largest absolute value 1 element, obtain the sparse gradient and its index: ; in This refers to sparse gradients. It means Index; Unselected gradients are retained as residuals, and the local residual vector is updated. : ; At this point, the client has completed local training and gradient compression.

[0019] According to a preferred embodiment of the present invention, calculating the reward and uploading the sparse gradient and state transition experience to the server includes: Calculate instant rewards Defined as: ; in, This represents the improvement in the client's accuracy on the local validation set in this round compared to the previous round; The total amount of data to upload sparse gradients and their corresponding indices is calculated as follows: ,in The number of bytes to store a single gradient value. The number of bytes stored for a single index; Local training time; and To balance the hyperparameters.

[0020] Further preferred, , .

[0021] According to a preferred embodiment of the present invention, the server receives sparse gradients and empirical data uploaded by the client, updates the supernetwork by weighting the data volume, and simultaneously updates the network parameters of the reinforcement learning strategy based on experience; including: The client uploads the following data to the server, including: Compressed gradient and its index Used to update the supernet weights; State transition experience Used to train the policy network; This represents the client's state at the start of the next round. After receiving the sparse gradients from all clients, the server aggregates them weighted by data volume and updates the supernetwork weights. : ; in The learning rate; At the same time, the server utilizes the collected experience data Update the parameters of the reinforcement learning policy network The update employs a near-end strategy optimization method, with the optimization objective being the pruned alternative objective function. : ; in , representing the probability ratio, measures the current reinforcement learning policy network. Compared with the old reinforcement learning policy network before the update In the same state Select the same action below The probability ratio; For the first Rotate the client's status, For the policy network in the 1st Round-robin selection of sub-network architecture, For current reinforcement learning policy networks, The old reinforcement learning strategy network before the update; This represents the average of all empirical samples within a batch; The pruning function restricts the probability ratio to an interval. Inside; For clipping hyperparameters; The advantage function is determined by the reward. Discount Factor and value function Generalized advantage estimation (GAE) is used to measure the dominance of a state. Select action The degree of advantage compared to the average level; Finally, the server outputs the trained super network weights. and policy networks For any client Client final state That is, the first The status reported by the client during round-robin communication includes resource status. and data characteristics The server will display this final state. Input reinforcement learning policy network Output a A probability distribution of dimension, where Configure the total number of centralized architectures for each sub-network, with each dimension corresponding to a probability value for the selected sub-network architecture; the server selects the sub-network architecture with the highest probability value as the client. The final sub-network architecture Then, according to the extraction function... From the super network Extract the corresponding sub-network weights from As a client The final personalized model is then distributed and deployed.

[0022] Further preferred, , .

[0023] A federated learning system based on reinforcement learning for subnetwork allocation and residual accumulation includes server-side functional modules and client-side functional modules; The server-side functional modules include a subnetwork configuration set construction module, a status acquisition module, a policy network module, a subnetwork extraction and distribution module, and a gradient aggregation and policy update module. The subnetwork configuration set building module is configured to: implement the predefinition of the elastic architecture supernetwork and the generation of the subnetwork configuration set; The status acquisition module is configured to acquire the resource status and data characteristics of each client. The policy network module is configured to: use near-end policy optimization and output the optimal sub-network architecture based on the client state; The subnetwork extraction and distribution module is configured to extract the corresponding weights from the supernetwork according to the selected architecture and send them to the client. The gradient aggregation and policy update module is configured to: receive sparse gradients and empirical data uploaded by the client, update the super network in a weighted manner according to the amount of data, and update the reinforcement learning policy network parameters based on experience. The client-side functional modules include a local training module, a residual compensation module, an adaptive sparsity module, and a gradient and experience collection and uploading module; The local training module is configured to train the received sub-network using a local dataset and compute gradients. The residual compensation module is configured to add the residual from the previous round to the current gradient to achieve knowledge inheritance. The adaptive sparsification module is configured to dynamically select important gradients based on the current bandwidth and update the residuals. The gradient and experience collection and uploading module is configured to: calculate rewards and upload sparse gradients and state transition experiences to the server.

[0024] The beneficial effects of this invention are as follows: 1. This invention breaks through the limitations of traditional static or heuristic subnetwork allocation strategies by employing a subnetwork allocation mechanism. Existing methods often allocate a fixed subnetwork to the client all at once during the initial training phase, failing to respond to real-time fluctuations in client resources and dynamic changes in training. This invention utilizes a reinforcement learning policy network to dynamically output the optimal subnetwork architecture based on the client's current computing power, memory, and bandwidth status, and enforces hard resource constraints to ensure allocation feasibility. This adaptive allocation mechanism avoids idleness on resource-rich devices and overload on resource-constrained devices, maximizing the utilization of heterogeneous client computing resources and significantly improving the training efficiency of federated learning.

[0025] 2. This invention addresses the knowledge forgetting problem during sub-network switching through a residual accumulation mechanism. When a client is assigned to a sub-network with a different structure due to resource changes, existing methods lack cross-architecture knowledge transfer mechanisms, leading to the complete loss of feature knowledge learned by the old sub-network. This invention accumulates the gradients not uploaded by the client in the previous round as residuals and compensates for them with the gradients in the current round. Simultaneously, the residuals are stored in a buffer aligned with the global parameter coordinates of the supernetwork. Even if the sub-network architecture changes, parameters with the same index maintain semantic consistency, ensuring seamless knowledge inheritance across different architectures and avoiding catastrophic forgetting.

[0026] 3. This invention achieves a balance between communication efficiency and knowledge integrity through a bandwidth-adaptive gradient sparsity strategy. Existing gradient sparsity methods typically employ a fixed compression ratio, which cannot adapt to real-time changes in client bandwidth, and the potential knowledge contained in discarded gradients is permanently lost. This invention dynamically determines the number of gradients to be uploaded based on the client's current network bandwidth. When bandwidth is ample, more gradients are retained to accelerate convergence; when bandwidth is limited, more aggressive compression is performed to save communication costs. Simultaneously, unselected gradients are retained through residuals and compensated in subsequent rounds, ensuring that potential knowledge in insignificant gradients is not permanently discarded, thus guaranteeing knowledge integrity while reducing communication overhead. Attached Figure Description

[0027] Figure 1 This is a structural diagram of a federated learning system based on reinforcement learning for subnetwork allocation and residual accumulation, according to the present invention.

[0028] Figure 2 This is a flowchart illustrating a federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation, as proposed in this invention.

[0029] Figure 3 This is a schematic diagram of the sub-network allocation mechanism of the present invention.

[0030] Figure 4 This is a schematic diagram of the residual accumulation mechanism of the present invention. Detailed Implementation

[0031] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0032] Example 1 A federated learning method based on reinforcement learning for subnetwork assignment and residual accumulation is implemented through several federated learning clients and a central server, such as... Figure 2 As shown, it includes: Enables the pre-definition of resilient architecture supernetworks and the generation of subnetwork configuration sets; Obtain the resource status and data characteristics of each client; Optimize using a near-end strategy and output the optimal sub-network architecture based on the client state; Extract the corresponding weights from the supernetwork based on the selected architecture and send them to the client; The received subnetwork is trained using the local dataset, and gradients are calculated. The residual from the previous round is added to the current gradient to achieve knowledge inheritance; Dynamically select important gradients based on the current bandwidth and update the residuals; Calculate the reward and upload the sparse gradient and state transition experience to the server; The server receives sparse gradients and empirical data uploaded by the client, updates the super network by weighting the data volume, and updates the network parameters of the reinforcement learning strategy based on experience.

[0033] Example 2 The difference between the federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation described in Example 1 and the following is: Assuming there is a total Each client has one federated learning client and one central server. Local data is determined by factors such as user behavior habits and device usage scenarios; therefore, the data from each client follows a different probability distribution. Formally, we record the client... The local data distribution is Client Maintain a from Private local datasets extracted independently and identically distributed The data volume is denoted as , Client Different resource configurations ,in Indicates computing power. Indicates memory capacity. Indicates network bandwidth;

[0034] The server maintains a super network weight with a resilient architecture. and predefine subnetwork configuration sets. The purpose of this invention is to train a personalized sub-network model for each client, adapted to its local data distribution and resource constraints, while ensuring continuous knowledge inheritance when switching between different sub-networks. This objective is formalized as an optimization problem that minimizes the weighted expected loss of all clients. ; In the formula Indicates the client The amount of data, Indicates the client The amount of data, Represents the local loss function. , Indicates based on sub-network architecture From the super network weight Extract the corresponding weights .

[0035] Sub-network allocation mechanism such as Figure 3 As shown, it includes the following: Implement the predefined and sub-network configuration sets of the resilient architecture supernetwork; including: Before federated learning training begins, the method of this invention constructs a set of sub-network configurations, and the server predefines a resilient super-network with ResNet as the backbone network. and set the width scaling factor set. Depth vector set Convolution kernel size vector set By iterating through the possible combinations of values ​​in the three dimensions, all possible sub-network architectures are generated, forming a sub-network configuration set. : in, The function is an extraction function, and the input is an elastic hypernetwork. and architecture tuple The output is the corresponding subnetwork structure; This is the channel scaling factor, which controls the scaling ratio of the number of channels in each layer; This is a depth vector, the length of which is equal to the number of stages in the network, and each element specifies the number of residual blocks in the corresponding stage; This is a convolutional kernel size vector, the length of which is the total number of layers in the network with adjustable convolutional kernel sizes. Each element specifies the convolutional kernel size of the corresponding layer. These three elements together determine a sub-network architecture. The extraction function is based on the architecture tuple. From Elastic Hypernetwork The corresponding parameter subset is retrieved by index and assembled into the tuple of this architecture. The sub-networks are constructed once at the start of training and selected only from them in subsequent communication rounds; no new architectures are dynamically generated.

[0036] Obtain the resource status and data characteristics of each client; including: Once the architecture is built, it enters the iterative training phase. In each communication round... The server randomly samples a subset of clients. (size is) For each selected client The server obtains the client's information. Current state Current state Including resource status and local data characteristics , recorded as Among them, resource status , , , These represent computing power, memory capacity, and network bandwidth, respectively; local data characteristics. This is a statistical descriptive vector for the client's local dataset. For classification tasks, it's a histogram of the number of samples in each class, representing the percentage of samples in each class. It's used to characterize the data distribution characteristics of the client.

[0037] Optimization employs a near-end strategy, outputting the optimal sub-network architecture based on client state; including: The server utilizes a reinforcement learning policy network based on proximal policy optimization (PPO). The client's state Mapped to a sub-network architecture ,Right now Specifically, this includes: reinforcement learning strategy networks. Based on the client's state As input, the input vector is mapped through a multi-layer fully connected neural network, and the output is a... A logits vector of dimension, where The total number of centralized architectures configured for the sub-network; this logits vector is normalized to a probability distribution using the Softmax function. ,in, Indicates the selection of the first The probability of each sub-network architecture satisfies The server follows this probability distribution Random sampling is performed to obtain the selected sub-network architecture. ; The PPO algorithm limits the magnitude of policy changes in each update through a pruning mechanism, thereby improving sample efficiency while ensuring training stability. Specifically, the optimization objective of proximal policy optimization is to maximize the alternative objective function after pruning. This alternative objective function avoids destructive updates by limiting the probability ratio of the new and old policies, and combines advantage function estimation to guide the policy to iterate towards a better direction. Before outputting actions, a hard resource constraint is enforced: if the memory required by the recommended subnetwork exceeds the currently available memory on the client side... Then it will be automatically replaced with the subnet configuration set. The subnet that best matches the client's resources and does not exceed its capacity is selected. This mechanism ensures that the allocation in each round is always feasible on the actual hardware and provides meaningful reward signals to the policy network.

[0038] Extracting the corresponding weights from the supernetwork based on the selected architecture and sending them to the client; including: The server depends on the selected subnetwork architecture. From the super network Extract the corresponding subnetwork weights and will Send to the client The extraction process is essentially about retrieving values ​​by index, including: each sub-network architecture The corresponding fixed subset of indices in the global parameter space of the supernetwork (determined by width, depth, and kernel size) is directly obtained from the server. Extract the parameters from these index positions and assemble them into sub-network weights. Parameters that are not retrieved are retained in the super network and do not participate in the local training of the client.

[0039] Residual accumulation mechanism such as Figure 4 As shown, it specifically includes: The received sub-network is trained using a local dataset, and gradients are calculated; this includes: Client Receive subnetwork weights Then, in the local dataset Perform several rounds of local iterative training to calculate the original gradient. ( , );in, For loss functions (such as cross-entropy loss in classification tasks). This indicates that the gradient of the loss function is calculated. This is the original gradient vector obtained.

[0040] Add the residual from the previous round to the current gradient to achieve knowledge inheritance; including: To preserve historical knowledge and achieve cross-architecture knowledge inheritance, the client will store the residual vector from the previous round. Add the current gradient to obtain the compensated gradient. : .

[0041] The residual vector is stored in a buffer aligned with the coordinates of the global parameters of the supernetwork. Even if the architecture allocated in this round is different from that in the previous round, the parameters with the same index still correspond semantically, ensuring the effectiveness of residual compensation.

[0042] Dynamically select important gradients based on the current bandwidth and update the residuals; including: To reduce communication overhead, the client adjusts the settings based on the current network bandwidth. Dynamically determine the number of gradients to retain : ; in For sub-networks The total number of parameters, The preset maximum bandwidth reference value; from the compensated gradient Select the one with the largest absolute value 1 element, obtain the sparse gradient and its index: ; in This refers to sparse gradients. It means Index; Unselected gradients are retained as residuals, and the local residual vector is updated. : ; At this point, the client has completed local training and gradient compression.

[0043] Calculate the reward, upload the sparse gradient and state transition experience to the server; including: To evaluate whether the sub-network architecture allocated in this round is suitable for the client (i.e., to measure the merits of the action), and to use this as a signal to optimize the server's policy network, the client needs to calculate an immediate reward. The reward takes into account both the improvement in local accuracy and resource overhead, and is defined as follows: ; in, This represents the improvement in the client's accuracy on the local validation set in this round compared to the previous round (i.e., the accuracy in this round minus the accuracy in the previous round). The total amount of data to upload sparse gradients and their corresponding indices is calculated as follows: ,in The number of bytes to store a single gradient value. This represents the number of bytes stored for a single index; this value reflects the actual communication load of the current round. Local training time; and To balance hyperparameters. Used to adjust accuracy gain and resource overhead to the same order of magnitude. The larger this reward value, the more suitable the sub-network allocated in this round is for the client's current state (sufficient resources, low data heterogeneity).

[0044] , The specific values ​​can be determined through optimization using a small number of validation rounds.

[0045] The server receives sparse gradients and empirical data uploaded by the client, updates the supernetwork weighted by data volume, and simultaneously updates the network parameters of the reinforcement learning policy based on experience; including: The client uploads the following data to the server, including: Compressed gradient and its index Used to update the supernet weights; State transition experience Used to train the policy network; This represents the client's state at the start of the next round (resource states may have changed). After receiving the sparse gradients from all clients, the server aggregates them weighted by data volume and updates the supernetwork weights. : ; in The learning rate is used to control the step size for updating model parameters in each round.

[0046] At the same time, the server utilizes the collected experience data Update the parameters of the reinforcement learning policy network The update employs the Proximal Policy Optimization (PPO) method, with the optimization objective being the pruned alternative objective function. : ; in , representing the probability ratio, measures the current reinforcement learning policy network. Compared with the old reinforcement learning policy network before the update In the same state Select the same action below The probability ratio; For the first Rotate the client's status, For the policy network in the 1st Round-robin selection of sub-network architecture, For current reinforcement learning policy networks, The old reinforcement learning strategy network before the update; This represents the average of all empirical samples within a batch; The pruning function restricts the probability ratio to an interval. Inside; For pruning hyperparameters; used to control the allowable deviation between the old and new strategies; The advantage function is determined by the reward. Discount Factor and value function Generalized advantage estimation (GAE) is used to measure the dominance of a state. Select action The degree of advantage compared to the average level; the specific calculation process includes: first, calculating the time difference (TD) error. This value reflects the difference between the actual reward plus the value estimate of the next state and the value estimate of the current state; then, the TD errors of multiple time steps are weighted and summed to obtain the advantage function. ,in The GAE parameter (preferably 0.95) controls the tradeoff between bias and variance. This objective function limits the single-step update magnitude of the policy through a pruning mechanism, thus ensuring training stability. The reward signal is transmitted through the advantage function. Indirectly participate in optimization, guiding the policy network to update towards actions that generate higher rewards.

[0047] The above process (from client state acquisition, subnetwork allocation, local training and residual compensation, adaptive sparsity reduction, reward calculation and experience uploading, to server aggregation and policy update) is executed iteratively until the preset maximum number of communication rounds is reached. Alternatively, the loss function of the super network may converge. In each round, the policy network continuously optimizes the allocation strategy using the experience collected in the previous round, making the matching between the sub-network and the client's resource state increasingly accurate; at the same time, the client's residual vector continuously accumulates gradient information that has not been uploaded, ensuring seamless inheritance of knowledge between different architectures.

[0048] Finally, the server outputs the trained super network weights. and policy networks For any client Client final state That is, the first The status reported by the client during round-robin communication includes resource status. and data characteristics This state records the client's resource configuration at the end of training. The server will then display this final state. Input reinforcement learning policy network Output a A probability distribution of dimension, where Configure the total number of centralized architectures for each sub-network, with each dimension corresponding to a probability value for the selected sub-network architecture; the server selects the sub-network architecture with the highest probability value as the client. The final sub-network architecture Then, according to the extraction function... From the super network Extract the corresponding sub-network weights from As a client The final personalized model was delivered and deployed. This model not only adapts to the client's local data distribution and meets its resource constraints, but also avoids catastrophic forgetting through residual accumulation during training.

[0049] , Those skilled in the art can determine the optimal value based on the specific task and model size through conventional experiments or learning rate decay strategies.

[0050] In this embodiment, the system and data configuration uses three publicly available image classification datasets—CIFAR-10, CIFAR-100, and FMNIST—as example sources of local data for the clients. Total number of clients. The number of clients randomly participating in training each round Maximum number of communication rounds (CIFAR-10, FMNIST) or (CIFAR-100). Each participating client performs 5 rounds of local iterative training, with a batch size of 64.

[0051] Model architecture configuration: The resilient supernetwork uses ResNet-34 as its backbone and is built using the OFA (Once-for-All) framework. Width scaling factor set. Depth vector set ; Set of convolution kernel size vectors The three elements are traversed and combined to form a subnetwork configuration set. There are a total of 27 different architectures. This configuration set is built once at the start of training, and subsequent selections are made from it only; no new architectures are dynamically added.

[0052] Method execution flow: Iteratively execute according to the above flow: The server collects the status (resource status and data characteristics) of participating clients, and transmits the data through the policy network. The sub-network architecture is allocated and weights are distributed; after the client completes local training, residual compensation, and adaptive sparsification, it uploads the sparse gradients and state transition experience to the server; the server aggregates gradients to update the super-network weights, and simultaneously collects experience to update the policy network parameters. In specific implementation, the learning rate... The optimizer uses an SGD optimizer with a momentum of 0.9. In the reward function... Bandwidth reference value Mbit / s. PPO clipping parameters. Discount factor GAE parameters The policy network uses a three-layer fully connected network with hidden layer dimensions of 128, 64, and 32, and the activation function is ReLU.

[0053] According to the above configuration, the present invention was implemented and verified on three sets of datasets, and the experimental results are shown in Table 1.

[0054] Table 1. Experimental results for the three datasets; Example 3 A federated learning system based on reinforcement learning for subnetwork allocation and residual accumulation, such as... Figure 1As shown, it includes server-side functional modules and client-side functional modules; The server-side functional modules include a subnetwork configuration set construction module, a status acquisition module, a policy network module, a subnetwork extraction and distribution module, and a gradient aggregation and policy update module. The subnetwork configuration set building module is configured to: implement the predefinition of the elastic architecture supernetwork and the generation of the subnetwork configuration set; The status acquisition module is configured to acquire the resource status and data characteristics of each client. The policy network module is configured to: use near-end policy optimization and output the optimal sub-network architecture based on the client state; The subnetwork extraction and distribution module is configured to extract the corresponding weights from the supernetwork according to the selected architecture and send them to the client. The gradient aggregation and policy update module is configured to: receive sparse gradients and empirical data uploaded by the client, update the super network in a weighted manner according to the amount of data, and update the reinforcement learning policy network parameters based on experience. The client-side functional modules include a local training module, a residual compensation module, an adaptive sparsity module, and a gradient and experience collection and uploading module; The local training module is configured to train the received sub-network using a local dataset and compute gradients. The residual compensation module is configured to add the residual from the previous round to the current gradient to achieve knowledge inheritance. The adaptive sparsification module is configured to dynamically select important gradients based on the current bandwidth and update the residuals. The gradient and experience collection and uploading module is configured to: calculate rewards and upload sparse gradients and state transition experiences to the server.

[0055] This invention solves the problem of knowledge forgetting caused by changes in sub-network architecture by having these modules work together, and realizes resource-adaptive sub-network allocation and cross-architecture knowledge continuity, while reducing communication overhead and ensuring training accuracy.

Claims

1. A federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation, characterized in that, This is implemented through several federated learning clients and a central server, including: Enables the pre-definition of resilient architecture supernetworks and the generation of subnetwork configuration sets; Obtain the resource status and data characteristics of each client; Optimize using a near-end strategy and output the optimal sub-network architecture based on the client state; Extract the corresponding weights from the supernetwork based on the selected architecture and send them to the client; The received subnetwork is trained using the local dataset, and gradients are calculated. The residual from the previous round is added to the current gradient to achieve knowledge inheritance; Dynamically select important gradients based on the current bandwidth and update the residuals; Calculate the reward and upload the sparse gradient and state transition experience to the server; The server receives sparse gradients and empirical data uploaded by the client, updates the super network by weighting the data volume, and updates the network parameters of the reinforcement learning strategy based on experience.

2. The federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation as described in claim 1, characterized in that, Assuming there is a total Each federated learning client and a central server record the clients. The local data distribution is Client Maintain a from Private local datasets extracted independently and identically distributed The data volume is denoted as , ; Client Different resource configurations ,in Indicates computing power. Indicates memory capacity. Indicates network bandwidth; The server maintains a super network weight with a resilient architecture. and predefine subnetwork configuration sets. ; For each client, a personalized sub-network model is trained to adapt to its local data distribution and resource constraints, while ensuring continuous knowledge inheritance when switching between different sub-networks; formally, this is an optimization problem that minimizes the weighted expected loss of all clients. ； In the formula Indicates the client The amount of data, Indicates the client The amount of data, Represents the local loss function. , Indicates based on sub-network architecture From the super network weight Extract the corresponding weights .

3. The federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation as described in claim 1, characterized in that, Implement the predefined and sub-network configuration sets of the resilient architecture supernetwork; including: Construct a set of sub-network configurations; the server predefines a resilient supernetwork with ResNet as its backbone. and set the width scaling factor set. Depth vector set Convolution kernel size vector set By iterating through the possible combinations of values ​​in the three dimensions, all possible sub-network architectures are generated, forming a sub-network configuration set. : in, The function is an extraction function, and the input is an elastic hypernetwork. and architecture tuple The output is the corresponding subnetwork structure; This is the channel scaling factor; It is a depth vector; The kernel size vector is used for extraction; the extraction function is based on the architecture tuple. From Elastic Hypernetwork The corresponding parameter subset is retrieved by index and assembled into the tuple of this architecture. Subnetworks below.

4. The federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation as described in claim 1, characterized in that, Obtain the resource status and data characteristics of each client; including: In each communication round The server randomly samples a subset of clients. For each selected client The server obtains the client's information. Current state Current state Including resource status and local data characteristics , recorded as Among them, resource status , , , These represent computing power, memory capacity, and network bandwidth, respectively; local data characteristics. This is a statistical description vector for the client's local dataset. For classification tasks, it is a histogram of the number of samples in each category, i.e., the proportion of samples in each category. Optimization employs a near-end strategy, outputting the optimal sub-network architecture based on client state; including: The server utilizes a reinforcement learning policy network based on proximal policy optimization. The client's state Mapped to a sub-network architecture ,Right now Specifically, this includes: reinforcement learning strategy networks. Based on the client's state As input, the input vector is mapped through a multi-layer fully connected neural network, and the output is a... A logits vector of dimension, where The total number of centralized architectures configured for the sub-network; this logits vector is normalized to a probability distribution using the Softmax function. ,in, Indicates the selection of the first The probability of each sub-network architecture satisfies The server follows this probability distribution Random sampling is performed to obtain the selected sub-network architecture. ; The optimization objective of near-end policy optimization is to maximize the alternative objective function after pruning. This alternative objective function avoids destructive updates by limiting the probability ratio of the new and old policies, and combines advantage function estimation to guide the policy to iterate towards a better direction. Before outputting actions, a hard resource constraint is enforced: if the memory required by the recommended subnetwork exceeds the current available memory on the client. Then it will be automatically replaced with the subnet configuration set. The subnet that best matches the client's resources and does not exceed its capacity.

5. The federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation according to claim 4, characterized in that, Extracting the corresponding weights from the supernetwork based on the selected architecture and sending them to the client; including: The server depends on the selected subnetwork architecture. From the super network Extract the corresponding subnetwork weights and will Send to the client Includes: the architecture of each sub-network Corresponding to a fixed subset of indices in the global parameter space of the super network, the server directly from Extract the parameters from these index positions and assemble them into sub-network weights. Parameters that are not retrieved are retained in the super network and do not participate in the local training of the client.

6. The federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation according to claim 5, characterized in that, The received sub-network is trained using a local dataset, and gradients are calculated; this includes: Client Receive subnetwork weights Then, in the local dataset Perform several rounds of local iterative training to calculate the original gradient. ( , );in, For loss function, This indicates that the gradient of the loss function is calculated. This is the resulting original gradient vector; Add the residual from the previous round to the current gradient to achieve knowledge inheritance; including: The client stores the residual vector from the previous round. Add the current gradient to obtain the compensated gradient. : 。 7. The federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation according to claim 1, characterized in that, Dynamically select important gradients based on the current bandwidth and update the residuals; including: The client depends on the current network bandwidth. Dynamically determine the number of gradients to retain : ； in For sub-networks The total number of parameters, The preset maximum bandwidth reference value; from the compensated gradient Select the one with the largest absolute value 1 element, obtain the sparse gradient and its index: ； in This refers to sparse gradients. It means Index; Unselected gradients are retained as residuals, and the local residual vector is updated. : ； At this point, the client has completed local training and gradient compression.

8. The federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation according to claim 7, characterized in that, Calculate the reward, upload the sparse gradient and state transition experience to the server; including: Calculate instant rewards Defined as: ； in, This represents the improvement in the client's accuracy on the local validation set in this round compared to the previous round; The total amount of data to upload sparse gradients and their corresponding indices is calculated as follows: ,in The number of bytes to store a single gradient value. The number of bytes stored for a single index; Local training time; and To balance the hyperparameters; ， 。 9. A federated learning method based on reinforcement learning for subnetwork allocation and residual accumulation as described in claim 8, characterized in that, The server receives sparse gradients and empirical data uploaded by the client, updates the supernetwork weighted by data volume, and simultaneously updates the network parameters of the reinforcement learning policy based on experience; including: The client uploads the following data to the server, including: Compressed gradient and its index Used to update the supernet weights; State transition experience Used to train the policy network; This represents the client's state at the start of the next round. After receiving the sparse gradients from all clients, the server aggregates them weighted by data volume and updates the supernetwork weights. : ； in The learning rate; At the same time, the server utilizes the collected experience data Update the parameters of the reinforcement learning policy network The update employs a near-end strategy optimization method, with the optimization objective being the pruned alternative objective function. : ； in , representing the probability ratio, measures the current reinforcement learning policy network. Compared with the old reinforcement learning policy network before the update In the same state Select the same action below The probability ratio; For the first Rotate the client's status, For the policy network in the 1st Round-robin selection of sub-network architecture, For current reinforcement learning policy networks, The old reinforcement learning strategy network before the update; This represents the average of all empirical samples within a batch; The pruning function restricts the probability ratio to an interval. Inside; For clipping hyperparameters; The advantage function is determined by the reward. Discount Factor and value function Generalized dominance estimation is used to measure the state. Select action The degree of advantage compared to the average level; Finally, the server outputs the trained super network weights. and policy networks For any client Client final state That is, the first The status reported by the client during round-robin communication includes resource status. and data characteristics The server will display this final state. Input reinforcement learning policy network Output a A probability distribution of dimension, where Configure the total number of centralized architectures for each sub-network, with each dimension corresponding to a probability value for the selected sub-network architecture; the server selects the sub-network architecture with the highest probability value as the client. The final sub-network architecture Then, according to the extraction function... From the super network Extract the corresponding sub-network weights from As a client The final personalized model is then distributed and deployed; ， 。 10. A federated learning system based on reinforcement learning for subnetwork allocation and residual accumulation, characterized in that, Includes server-side functional modules and client-side functional modules; The server-side functional modules include a subnetwork configuration set construction module, a status acquisition module, a policy network module, a subnetwork extraction and distribution module, and a gradient aggregation and policy update module. The subnetwork configuration set building module is configured to: implement the predefinition of the elastic architecture supernetwork and the generation of the subnetwork configuration set; The status acquisition module is configured to acquire the resource status and data characteristics of each client. The policy network module is configured to: use near-end policy optimization and output the optimal sub-network architecture based on the client state; The subnetwork extraction and distribution module is configured to extract the corresponding weights from the supernetwork according to the selected architecture and send them to the client. The gradient aggregation and policy update module is configured to: receive sparse gradients and empirical data uploaded by the client, update the super network in a weighted manner according to the amount of data, and update the reinforcement learning policy network parameters based on experience. The client-side functional modules include a local training module, a residual compensation module, an adaptive sparsity module, and a gradient and experience collection and uploading module; The local training module is configured to train the received sub-network using a local dataset and compute gradients. The residual compensation module is configured to add the residual from the previous round to the current gradient to achieve knowledge inheritance. The adaptive sparsification module is configured to dynamically select important gradients based on the current bandwidth and update the residuals. The gradient and experience collection and uploading module is configured to: calculate rewards and upload sparse gradients and state transition experiences to the server.

Images