Reinforcement learning-based federated learning regular term weight dynamic adjustment method and system

Abstract

This invention discloses a method and system for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning, belonging to the field of federated learning technology. The method includes: initializing training parameters on a central server; uploading model parameters and loss improvement rates after local training on each client; extracting global and local state features and substituting them into a multi-objective dynamic activation function to calculate dynamic reward values; outputting updated regularization term weights by the reinforcement learning agent and performing differentiated adaptive updates on multiple core coefficients of the activation function; dynamically weighting and aggregating the data using an attention mechanism to generate a global model; and iterating until convergence. This invention improves the convergence efficiency, stability, and generalization performance of federated learning in scenarios with heterogeneous data by dynamically adjusting the weights of regularization terms through reinforcement learning, combined with multi-objective activation and adaptive coefficient updates.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a method and system for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning. Background Technology

[0002] In the current context of deepening digital transformation and increasingly prominent data value, Federated Learning (FL), relying on its core characteristics of "local data retention and collaborative model training," effectively alleviates the privacy, security, and compliance challenges in cross-institutional and cross-terminal data sharing, and has been widely applied in fields such as financial risk control, medical diagnosis, and smart terminal collaboration. Existing federated learning algorithms typically use fixed hyperparameter settings or simple linear adjustment strategies to guide global model aggregation and local model training. However, in actual deployment, federated learning faces multiple challenges, including heterogeneous client data distribution (non-independent and identically distributed), heterogeneous training resources, and inconsistencies between global objectives and local needs. The fixed hyperparameter training mode struggles to balance model performance, convergence stability, and scenario adaptability.

[0003] To enhance the adaptability of federated learning, existing methods have attempted to incorporate Reinforcement Learning (RL) into the federated learning framework, namely the RL+FL fusion architecture. This leverages the reinforcement learning agent's ability to perceive the environment to dynamically adjust key hyperparameters in federated learning. Among these, the regularization weight μ is a crucial parameter for balancing global model consistency with local model personalization; its value directly affects the model's convergence efficiency, generalization ability, and training stability. Excessively large μ can cause the model to overly favor global parameters, ignoring local data characteristics; excessively small μ may lead to client-side training divergence or overfitting. Existing methods lack dynamism and scenario adaptability in adjusting μ, often employing fixed values ​​or simple linear changes, making it difficult to adapt to changes in training progress, client heterogeneity, and global convergence state.

[0004] Furthermore, in the RL+FL framework, the activation function is a key link in guiding the agent's policy learning. Existing activation functions generally have the following shortcomings: (1) The objective dimension is singular, usually only focusing on global accuracy or local loss, failing to comprehensively consider multiple objectives such as global performance, local training effectiveness, model consistency, convergence speed, data heterogeneity adaptation and training stability, resulting in objective imbalance during training; (2) The core coefficients in the activation function (such as accuracy reward coefficient, local loss improvement coefficient, consistency coefficient, etc.) and key parameters such as reinforcement learning network update interval rely on manual tuning, which is time-consuming, laborious and difficult to adapt to dynamically changing training environments; (3) In non-independent identically distributed (Non-IID) scenarios, training stability is poor, and the client is prone to parameter divergence and loss fluctuations. Existing solutions lack a special adaptation mechanism for the degree of data heterogeneity, fail to effectively link client heterogeneity with incentive penalties, and fail to effectively suppress training fluctuations. Summary of the Invention

[0005] To address the above issues, this invention provides a method and system for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning. The aim is to achieve intelligent dynamic adjustment of the weights of regularization terms through a multi-objective dynamic activation function and an adaptive update mechanism for core coefficients, thereby improving the training performance and stability of federated learning in scenarios with heterogeneous data.

[0006] In a first aspect, the present invention provides a method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning, comprising: S1, the central server initializes the federated learning training parameters, including the client aggregate weights, the range of regularization term weights, and multiple core coefficients of the multi-objective dynamic activation function; S2, each client performs local model training based on local private data and uploads the training results to the central server. The training results include local model parameters and local loss improvement rate. S3, the central server extracts global state features and local state features of each client based on the training results uploaded by the client, splices them together to form joint state features of the reinforcement learning agent, and substitutes them into the multi-objective dynamic activation function to calculate the dynamic reward value of each client. S4, the central server outputs the updated regularization weights by the reinforcement learning agent based on the joint state features and the dynamic reward value; S5, the central server performs differentiated adaptive updates on multiple core coefficients of the multi-objective dynamic activation function based on the joint state characteristics and the updated regularization term weights; S6, the central server uses an attention mechanism to dynamically weight and aggregate local model parameters based on the updated regularization weights and client aggregation weights to generate a global model; S7, the central server sends the global model back to each client and determines whether the convergence condition has been met. If it has not converged, it iterates through steps S2 to S6; otherwise, it outputs the final global model.

[0007] Furthermore, S2 includes: Each client generates synthetic training data based on local private data, and uses the synthetic training data as samples, combined with the current regularization weights, to train the local model parameters for this round, based on the global model received from the central server in the previous round. Calculate the local loss improvement rate: ; in, Let be the loss improvement rate of client i in round t; This represents the baseline loss value of client i at the start of training; This represents the loss value of client i after the t-th round of training; Indicates an indicator function; `max` indicates that only positive improvements are retained. This represents the minimum finite threshold at which the loss decreases.

[0008] By generating synthetic training data of uniform quantity and consistent with the distribution of real data across all clients, the additional computational overhead caused by differences in client data volume is eliminated, while ensuring that the synthetic data fully captures the core features of the real data. Combined with the calculation of local loss improvement rate, incentives are only given when the local loss effectively decreases, avoiding training stagnation caused by excessively large regularization term weights or overfitting caused by excessively small weights, thus significantly improving the stability and effectiveness of local training.

[0009] Furthermore, the multi-objective dynamic activation function is specifically as follows: ; in, Let be the dynamic reward value for client i in round t; These are the core coefficients of the excitation function; This represents the change in accuracy on the global validation set. Improve the rate of local loss on the client side; This serves as a consistency indicator for client parameters. Weights for regularization terms; This is a penalty term for non-independent and identically distributed components; To train the fluctuation penalty term; This is an indicator of the convergence speed of global parameters.

[0010] The multi-objective dynamic activation function comprehensively considers indicators such as global accuracy changes, local loss improvement, parameter consistency, regularization penalty, non-independent and identically distributed penalty, training fluctuation penalty, and global convergence speed. It achieves multi-dimensional and accurate feedback to the reinforcement learning agent, effectively solves the training objective imbalance problem caused by traditional single-objective activation, and improves the global performance and convergence efficiency of federated learning in data heterogeneous scenarios.

[0011] Furthermore, the differential adaptive update of multiple core coefficients of the multi-objective dynamic activation function includes: Accuracy Bonus Gradient ascent update is performed using an exponential function combined with historical cumulative feedback, where the historical cumulative feedback is calculated from the global validation set accuracy. Local loss improvement coefficient A threshold-triggered update is adopted. When the local loss improvement exceeds a preset threshold, it is accelerated by the square root function; otherwise, it decays slowly. The local loss improvement is calculated from the local loss improvement rate. Consistency-Convergence Coefficient A dual-feedback balanced update is adopted, which directly integrates consistency feedback and convergence speed feedback. The consistency feedback and convergence speed feedback are calculated from the divergence of the client parameters and the change amplitude of the global parameters, respectively. Excessive penalty coefficient The update is based on a squared nonlinear penalty derived from the deviation between the updated regularization term weights and the preset ideal value. Non-independent and identically distributed fitness penalty coefficient An exponential function based on heterogeneity grading coupled with volatility suppression is used for updating. The exponential function includes a coupled feedback term, which is calculated from the change in the global average loss and the degree of non-independent and identically distributed data. The updated coefficients are all restricted to their corresponding value boundaries by the clipping function.

[0012] By implementing a differentiated adaptive update strategy for the five core coefficients, each coefficient can automatically adapt and adjust its magnitude based on real-time training feedback, eliminating the reliance on manual parameter tuning and significantly improving the system's adaptability and optimization efficiency to diverse data and dynamic environments.

[0013] Furthermore, the method also includes: Calculate the weighted average regularization weight, weighted average heterogeneity feedback, and weighted average training fluctuation feedback based on the updated regularization weight, non-independent and identically distributed penalty, and training fluctuation penalty for each client. Generate regularization term correlation coefficients based on the weighted average regularization term weights: ; in, This is the adjustment coefficient for the weight of the regularization term; These are the lower and upper bounds of the correlation coefficient for the regularization term weights, respectively; The correlation coefficient is the weight of the regularization term; The weights are the weights of the weighted average regularization term; The updated regular expression weights; clip This is the clipping function; This represents the ideal value for the regularization term weight; These are the lower and upper bounds of the regularization term weights, respectively; The dynamic correction factor is calculated based on the weighted average heterogeneous feedback and the weighted average training fluctuation feedback: ; in, , These are the heterogeneity feedback adjustment coefficient and the training fluctuation feedback adjustment coefficient, respectively. For weighted average heterogeneous feedback; The training fluctuation feedback is weighted average. These are the lower and upper limits of the dynamic correction factor, respectively. It is a dynamic correction factor; The actual update interval is calculated based on the preset base update interval T0, the regularization term weight correlation coefficient, and the dynamic correction factor: ; in, These are the lower and upper limits of the update interval, respectively; T t This refers to the actual update interval; round is a function that evaluates a function.

[0014] An update interval adjustment mechanism combining a basic update interval, a regularization term weight correlation coefficient, and a dynamic correction factor is introduced. This enables the update frequency of the reinforcement learning network to adapt in real time to changes in the training state and the regularization term weights. This avoids the inefficiency of fixed intervals in multiple scenarios and ensures training stability through a boundary protection mechanism, achieving a dynamic balance between training efficiency and policy learning quality.

[0015] Furthermore, S4 specifically includes: Reinforcement learning agents use joint state features As input, through the policy network Output intermediate action values And generate candidate updated regularization term weights based on the range of regularization term weight values. , where sigmoid is the activation function; Boundary protection is performed on the candidate updated regularization weights to obtain the final updated regularization weights. ,in These are the lower and upper bounds of the regularization term weights, respectively.

[0016] By combining the policy network of the reinforcement learning agent with the sigmoid activation function to generate candidate regularization weights, the weight output is smooth and always within a reasonable value range. With the boundary protection mechanism, the candidate weights are verified and corrected a second time, which not only ensures the rationality and stability of the regularization weights, but also avoids the risk of training failure caused by weights going out of bounds.

[0017] Furthermore, if the candidate updated regularization term weights Then additional penalties will be applied. The value is added to the dynamic reward value, where k is the boundary violation penalty coefficient, and k > 0.

[0018] When the weight of a candidate regularization term is lower than the preset lower limit, an additional penalty term is introduced into the dynamic reward value to immediately punish the agent's behavior of outputting extremely small values. This guides the reinforcement learning strategy to avoid actions that may lead to local training divergence or overfitting. It strengthens the constraint on the boundary of the regularization term weight value from the source of the reward signal, effectively prevents training oscillations, and ensures the convergence stability of federated learning in heterogeneous scenarios.

[0019] Furthermore, S6 specifically includes: Based on the updated regularization weights and client aggregation weights, the dynamic aggregation weights for each client are calculated using an attention mechanism. Based on dynamic aggregation weights, the local model parameters uploaded by all clients are weighted and summed to generate intermediate parameters for the global model. The intermediate parameters are fine-tuned and optimized to obtain the final global model.

[0020] By utilizing the updated regularization weights and client aggregation weights, the aggregation weights of each client are dynamically calculated through an attention mechanism. More attention is allocated to clients with high data heterogeneity and good training performance, while less attention is allocated to clients with divergent parameters and large training fluctuations. This effectively strengthens high-contribution clients and reasonably suppresses abnormal clients during the global model aggregation process, significantly improving the generalization ability and aggregation efficiency of the global model.

[0021] Secondly, this invention provides a federated learning regularization term weight dynamic adjustment system based on reinforcement learning, comprising: Multiple clients, each client performs local model training based on local private data, and uploads the training results to the central server. The training results include local model parameters and local loss improvement rate. A central server, which communicates with each client, includes: The state feature extraction module is used to extract global state features and local state features of each client based on the training results uploaded by the client, and splice them together to form the joint state features of the reinforcement learning agent. The dynamic incentive calculation module is used to substitute the joint state features into the multi-objective dynamic incentive function to calculate the dynamic reward value for each client; The reinforcement learning decision module is used to output updated regularization weights by the reinforcement learning agent based on the joint state features and the dynamic reward value. The coefficient update module is used to perform differentiated adaptive updates on multiple core coefficients of the multi-objective dynamic activation function based on the joint state characteristics and the updated regularization term weights. The dynamic aggregation module is used to dynamically weight and aggregate local model parameters based on the updated regularization weights and client-side aggregation weights, using an attention mechanism to generate a global model. The convergence judgment module is used to determine whether the convergence condition is met and to control the training iteration.

[0022] Compared with existing technologies, the beneficial effects of this invention are as follows: By enabling the reinforcement learning agent to perceive the global and local states in real time and combining a multi-objective dynamic activation function, intelligent dynamic adjustment of the regularization term weights is achieved, overcoming the shortcomings of fixed or linear adjustment that cannot adapt to data heterogeneity and dynamic changes in training; the activation function comprehensively considers multiple objectives such as global accuracy, local loss improvement, parameter consistency, non-independent and identically distributed penalty, and convergence speed, avoiding imbalance in training objectives; simultaneously, differentiated adaptive updates are implemented on multiple core coefficients of the activation function, eliminating reliance on manual parameter tuning and dynamically adjusting the update interval of the reinforcement learning network, thereby improving training efficiency; an attention mechanism is used to dynamically aggregate client models, highlighting high-contribution clients and improving the robustness of the global model. This invention significantly improves the convergence efficiency, stability, and generalization performance of federated learning in heterogeneous scenarios. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the overall architecture of the present invention; Figure 3 This is a schematic diagram of the reinforcement learning federated optimization system of the present invention; Figure 4 This is a comparison chart of model accuracy when the dataset used in this embodiment of the invention is Cifar10 and the client selection ratio is 0.1. Figure 5 This is a comparison chart of model accuracy when the dataset used in this embodiment of the invention is Cifar10 and the client selection ratio is 0.5; Figure 6 This is a comparison chart of model accuracy when the dataset used in this embodiment of the invention is Cifar100 and the client selection ratio is 0.1. Figure 7 This is a comparison chart of model accuracy when the dataset used in this embodiment of the invention is Cifar100 and the client selection ratio is 0.5. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments provided by this invention without inventive effort are within the scope of protection of this invention.

[0026] This invention provides a method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning, such as... Figure 1 , 2 As shown, the specific steps include: S1, the central server initializes the federated learning training parameters, including the privacy budget, client aggregation weights, the range of regularization weights, and multiple core coefficients of the reinforcement learning activation function.

[0027] Specifically, the central server first determines the batch size of the dataset for each training round, and receives the maximum number of training rounds reported by each client based on its own total data volume. The central server selects the minimum number of rounds reported by all clients as the basic upper limit of the number of rounds for the entire training process.

[0028] Simultaneously, the central server completes the initialization configuration of the following key parameters: privacy budget Pr∈(0,1), client aggregation weight, and the boundary values ​​of the regularization term weight μ [μ]. min ,μ max [and the initial values ​​of several core coefficients of the reinforcement learning incentive function, including the accuracy reward coefficient] Local loss improvement coefficient Consistency-convergence coefficient Excessive penalty coefficient Non-independent and identically distributed fitness penalty coefficient All of the above coefficients are positive numbers.

[0029] S2, each client performs local model training based on local private data, obtains local model parameters and local loss improvement rate, and uploads the training results to the central server.

[0030] Specifically, each client first generates synthetic training data based on its local private data, which is consistent with the distribution of its real data. During the generation process, the amount of synthetic data generated by all clients is strictly guaranteed to be uniform, so as to eliminate the computational bias caused by the difference in the amount of original data of the clients, and at the same time ensure that the synthetic data can fully capture the core features and distribution information of the real data.

[0031] The client uses the generated synthetic training data as training samples, based on the global model received from the central server in the previous round (the initial model is used in the first round of training), and combines it with the current regularization term weights. Perform local model training to obtain the local model parameters for this round. and the corresponding training effect score .

[0032] Calculate the improvement rate of the local loss for client i in this round: ; in, Let i be the loss improvement rate of client i in the t-th round of training; This represents the baseline loss value of client i at the start of training; This represents the loss value of client i after the t-th round of training; Indicates an indicator function; `max` indicates that only positive improvements are retained. This represents the minimum finite threshold at which the loss decreases.

[0033] After training is complete, each client will encrypt and upload the training results to the central server. The uploaded content specifically includes: local model parameters. Training effect score Local loss improvement rate Degree of non-independent and identically distributed data and the current regularization term weight .

[0034] S3, the central server extracts global state features and local state features of each client, splices them together to form joint state features of the reinforcement learning agent, and substitutes them into the multi-objective dynamic activation function to calculate the dynamic reward value of each client.

[0035] Specifically, the central server integrates all client-uploaded data and extracts global state features of the agent that reflect the overall training state. Including global validation set accuracy Client parameter divergence Global parameter variation range Global average loss change wait.

[0036] For each client i, extract its agent local state features. Including local loss improvement rate Degree of non-independent and identically distributed data Local loss changes Current regularization term weight .

[0037] global state features Local state characteristics of a single client The features are then combined to form the joint state features of the reinforcement learning agent. .

[0038] Substituting the joint state characteristics into the multi-objective dynamic incentive function, the dynamic reward value for each client is calculated. This function is specifically as follows: ; in, Let be the dynamic reward value for client i in round t; These are the core coefficients of the excitation function; This represents the change in accuracy on the global validation set. , Let be the change in global validation set accuracy after the t-th round of local training and global aggregation. Positive incentives are given to encourage regularization terms to improve overall performance. The penalty is based on negative incentives, with the weights of the regularization terms applied accordingly. The local loss improvement rate is only incentivized when the local loss effectively decreases, thus avoiding excessively large regularization weights that cause local training to stagnate or excessively small weights that cause overfitting. For client parameter consistency indicators, , exp It is a natural exponential function. This is the consistency decay coefficient, used to control the intensity of the impact of parameter divergence on the client parameter consistency index. ; To reinforce the learning agent's updated regularization weights based on the current joint state features output in round t; This is a non-independent and identically distributed penalty term. ; To train the fluctuation penalty term, ; This is an indicator of the convergence speed of global parameters. ,∥ ∥2 is the Euclidean norm. This is the convergence speed sensitivity coefficient, used to control the intensity of the influence of global parameter changes on the global parameter convergence speed index. .

[0039] Furthermore, client parameter divergence Calculated using the following formula: ; ; in, S t Let be the set of clients participating in the training round t; W t These are global model parameters; To prevent extremely small constants with a denominator of 0; This represents the difference between the local model and the global model for client i. Let i be the aggregate weight of client i in round t. , For the amount of data of client i, This represents the total amount of data from all clients participating in the training.

[0040] S4, the central server outputs the updated regularization weights by the reinforcement learning agent based on the joint state features and the dynamic reward value.

[0041] Specifically, reinforcement learning agents use joint state features As input, through the policy network Output intermediate action values And generate candidate updated regularization term weights based on the range of regularization term weight values. Where sigmoid is the activation function, and then boundary protection is performed on the candidate updated regularization weights to obtain the final updated regularization weights. ,Will Limited to Among them These represent the upper and lower boundaries of the regularization term weights, respectively.

[0042] Furthermore, if the candidate updated regularization term weights Then additional penalties will be applied. Add dynamic reward value In the formula, k is the boundary violation penalty coefficient, which is used to control the penalty intensity on the dynamic reward value when the weight of the candidate regularization term is lower than the lower limit, and k > 0.

[0043] S5, the central server performs differentiated adaptive updates on multiple core coefficients of the multi-objective dynamic activation function based on the joint state characteristics and the updated regularization term weights.

[0044] Specifically, the central server adjusts the five core coefficients of the activation function based on the dynamic reward value and historical training feedback. ) Implement differentiated update strategies and restrict the updated coefficients to their respective preset boundaries through pruning functions to avoid extreme values.

[0045] The central server generates the target priority vector. (The sum of all components is 1), and the parameter influence factor. This provides a weighting basis for adjusting core parameters.

[0046] Specifically, the central server first calculates the feedback intensity values ​​corresponding to the five training objectives based on the feedback from the t-th round of training. ,in These correspond to five types of feedback: accuracy improvement, local loss improvement, parameter consistency and convergence speed improvement, regularization term bias, and non-independent and identically distributed (i.i.d.) and training fluctuations. The central server then... The feedback intensity values ​​are normalized to obtain the parameter influence factor. This parameter's influence factor serves as an influence factor in the subsequent reverse constraint process.

[0047] Accuracy Bonus Gradient ascent update is performed using an exponential function combined with historical cumulative feedback. ; in, The cumulative historical feedback for client i up to round t is used to characterize the change in global validation set accuracy across historical rounds. The ongoing impact of the current accuracy reward coefficient update can be obtained by weighting the historical cumulative feedback from the previous round of historical cumulative feedback with the change in accuracy on the current global validation set, i.e. , This is the historical feedback attenuation coefficient; These represent the upper and lower boundaries of the accuracy bonus coefficient; clip This is the clipping function; , These are the accuracy bonus coefficients for the current round and the next round, respectively. The accuracy reward coefficient is updated with adjustment parameters to control the change in global validation set accuracy and the impact of historical cumulative feedback on the accuracy reward coefficient. The adjustment range, when When the value is large, the accuracy bonus coefficient More sensitive to changes in accuracy The updates are even more drastic, when When the value is small, the accuracy bonus coefficient The updates are now more gradual.

[0048] The above update process uses an exponential function to amplify the adjustment power of effective feedback. Furthermore, from the perspective of parameter influence factors... Extracting the local loss improvement coefficient The inverse constraint is applied to the influencing factors, and its mechanism is as follows: based on and The real-time value is used to construct implicit reverse constraint logic—when When it is too big, The contribution ratio will increase relatively, thereby inhibiting Further increase the size to avoid excessive suppression of local training improvements by global accuracy optimization.

[0049] Local loss improvement coefficient Threshold-triggered updates are used. When the local loss improvement exceeds a preset threshold, the improvement is accelerated using a square root function; otherwise, it decays slowly. ; in, The loss improvement threshold is used to distinguish between effective and ineffective improvements; Improvements for local losses; , These are the local loss improvement coefficients for the current round and the next round, respectively. These represent the upper and lower boundaries of the local loss improvement coefficient; Update the adjustment parameters for the local loss improvement coefficient, which are used to control the effect of local loss improvement feedback on the local loss improvement coefficient. The adjustment range, when local losses effectively decrease, control The magnitude of the enhancement, when the local loss improvement is ineffective. control The rate of decay.

[0050] when When this occurs, it indicates a significant improvement in local loss. In this case, the square root function is used to strengthen the adjustment and accelerate the increase in the coefficient. When the value is low, it indicates that the improvement is weak or ineffective. A slow decay strategy is adopted to avoid training instability caused by excessive decrease in coefficients.

[0051] In addition, from the parameter influence factor Extract the accuracy bonus coefficient The influencing factors are subject to reverse constraints when When it is too big, The contribution percentage will increase accordingly, thereby avoiding... Over-dominance of local training can prevent the model from overfitting to local data and losing global consistency.

[0052] Consistency-Convergence Coefficient Employing a dual-feedback balanced update method that directly integrates consistency feedback and convergence rate feedback: ; in, , These are the consistency-convergence coefficients for the current round and the next round, respectively. This is a globally consistent feedback value. ; This is the global convergence speed feedback value. ; The consistency-convergence synergy coefficient is updated with adjustment parameters to control the global consistency feedback value. and global convergence speed feedback value Consistency-convergence coherence coefficient The overall adjustment range or normalized scale, when When the value is large, and right The regulatory effect is weakened, making The updates are more gradual, when When the value is small, and right The regulatory effect increases; These are the upper and lower boundaries of the consistency-convergence synergy coefficients, respectively.

[0053] Excessive penalty coefficient The update method employs a non-linear penalty based on the squared deviation between the updated regularization term weights and the preset ideal value. ; in, , These are the excessive penalty coefficients for the current round and the next round, respectively. These represent the upper and lower boundaries of the excessive penalty coefficient; The adjustment parameters are updated to compensate for excessively large penalty coefficients, and are used to control the weights of the updated regularization term. Relative to the preset ideal value of the regularization term weight The degree of deviation affects the penalty coefficient. The adjustment range, when the updated regularization term weight Deviation from ideal value When it is large, control The magnitude of the increase is to enhance the penalty for excessive weighting of regularization terms.

[0054] Non-independent and identically distributed fitness penalty coefficient The update is performed using an exponential function coupled with heterogeneity grading and volatility suppression. ; in, , These are the non-independent identically distributed adaptation penalty coefficients for the current round and the next round, respectively; The adjustment parameter is used to update the non-independent and identically distributed (IID) fitting penalty coefficient, which controls the impact of client data heterogeneity feedback and training fluctuation feedback on the non-independent and identically distributed (IID) fitting penalty coefficient. The adjustment range should be adjusted when the client data is highly heterogeneous or the training fluctuations are large. control The magnitude of the enhancement is increased to improve the penalty intensity for non-independent identically distributed scenarios and training unstable states; These are the upper and lower boundaries of the non-independent and identically distributed fitness penalty coefficient, respectively. This is a coupled feedback term.

[0055] Specifically, the coupling feedback term Obtain it through the following steps: The central server applies non-independent, identically distributed penalty items uploaded by each client. and training fluctuation penalty item Combined with client-side aggregated weight Calculate the weighted average heterogeneous feedback in round t. and weighted average training fluctuation feedback : ; ; The central server is based on weighted average heterogeneous feedback. and weighted average training fluctuation feedback Generate coupled feedback terms: ; in, , These are the heterogeneity feedback adjustment coefficient and the training fluctuation feedback adjustment coefficient, respectively.

[0056] This invention also dynamically adjusts the update interval of the reinforcement learning network, calculating the actual update interval based on the basic update interval, the correlation coefficient of the regularization term weights, and the dynamic correction factor, so that the network update frequency is adapted to the training state in real time.

[0057] Specifically, the central server first calculates the weighted average regular expression weight for round t based on the updated regular expression weights of each client. And based on the weighted average regularization term weight, a regularization term weight correlation coefficient is generated. : ; in, This is the adjustment coefficient for the weight of the regularization term; These are the lower and upper limits of the correlation coefficient for the regularization term weights, respectively.

[0058] The central server further provides feedback based on weighted average heterogeneity. and weighted average training fluctuation feedback Calculate the dynamic correction factor : ; in, These are the lower and upper limits of the dynamic correction factor, respectively.

[0059] Based on the basic update interval T0, the regularization term weight correlation coefficient and dynamic correction factor Calculate the actual update interval T t : ; in, These are the lower and upper limits of the update interval, respectively; round is a function that evaluates to a value.

[0060] When training fluctuates significantly or data is highly heterogeneous, the actual update interval is shortened to allow the reinforcement learning network to update more frequently; when the training state is relatively stable, the actual update interval is appropriately extended to reduce unnecessary update overhead.

[0061] In S6, the central server uses an attention mechanism to dynamically weight and aggregate the local model parameters based on the updated regularization weights and client aggregation weights to generate a global model.

[0062] Specifically, the central server uses the updated regular expression weights. Aggregate weights with clients The dynamic aggregation weight for each client is calculated using an attention mechanism: ; in, Attention score; Subsequently Normalization is performed to obtain dynamic aggregate weights, which allows clients with more dramatic changes in regularization weights to receive more attention.

[0063] Based on dynamic aggregation weights, the local model parameters uploaded by all clients are weighted and summed to generate intermediate parameters for the global model. The intermediate parameters are fine-tuned to eliminate potential biases in the aggregation process, resulting in the formal global model.

[0064] S7, the central server sends the global model back to each client and determines whether the convergence condition has been met. If it has not converged, it iterates through steps S2 to S6; otherwise, it outputs the final global model.

[0065] Specifically, the central server will send the official global model back to all clients participating in this round of training as the initial model for the next round of local training. Simultaneously, the central server will share the performance of this round of global model... If the current model performs better than the historical best model, then the current model is saved as the global best model.

[0066] The central server determines whether the preset optimal convergence condition has been met. The optimal convergence condition is set by combining multiple dimensions, including global validation set accuracy, training loss, parameter stability, and convergence speed.

[0067] If the optimal convergence condition is not met, the updated core coefficients and regularization weights from step S4 are used as the initial parameters for the next round of training, and the process returns to step S2 to start the next iteration. If the optimal convergence condition has been met, the training process is terminated, and the currently saved global optimal model is output as the final model.

[0068] To verify the effectiveness of the method of this invention, a comparative experiment was designed in this embodiment to compare the performance of the method of this invention with that of existing federated learning algorithms FedAvg (Federated Averaging Algorithm) and FedProx (Federated Proximal Optimization Algorithm) in federated learning tasks. Reinforcement learning is updated every 5 rounds, and a total of 40 clients were set up in the experiment. F represents the proportion of clients selected in each round of training. For example, when F=0.1, 10% of the 40 clients are randomly selected for training in each round; when F=0.5, 50% of the 40 clients are randomly selected for training in each round. Cifar10 and Cifar100 datasets were used respectively, and the experimental results are shown in Table 1 and [Table data would be inserted here]. Figure 4-7 As shown.

[0069] Table 1 Comparative Experiment Results

[0070] This invention provides a dynamic weight adjustment system for regularization terms in federated learning based on reinforcement learning, such as... Figure 3 As shown, it includes: Multiple clients, each client performs local model training based on local private data, and uploads the training results to the central server. The training results include local model parameters and local loss improvement rate. A central server, which communicates with each client, includes: The state feature extraction module is used to extract global state features and local state features of each client based on the training results uploaded by the client, and splice them together to form the joint state features of the reinforcement learning agent. The dynamic incentive calculation module is used to substitute the joint state features into the multi-objective dynamic incentive function to calculate the dynamic reward value for each client; The reinforcement learning decision module is used to output updated regularization weights by the reinforcement learning agent based on the joint state features and the dynamic reward value. The coefficient update module is used to perform differentiated adaptive updates on multiple core coefficients of the multi-objective dynamic activation function based on the joint state characteristics and the updated regularization term weights. The dynamic aggregation module is used to dynamically weight and aggregate local model parameters based on the updated regularization weights and client-side aggregation weights, using an attention mechanism to generate a global model. The convergence judgment module is used to determine whether the convergence condition is met and to control the training iteration.

[0071] It should be noted that the present invention is not limited to the above-described embodiments. The above embodiments are merely examples, and any embodiments that have the same structure and perform the same effects as the technical concept within the scope of the present invention are included within the scope of the present invention. Furthermore, various modifications that can be conceived by those skilled in the art to the embodiments, and other ways of constructing by combining some of the constituent elements of the embodiments, without departing from the spirit of the present invention, are also included within the scope of the present invention.

Claims

1. A method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning, characterized in that, include: S1, the central server initializes the federated learning training parameters, including the client aggregation weight, the range of regularization term weight values, and multiple core coefficients of the multi-objective dynamic activation function. The multiple core coefficients include the accuracy reward coefficient, the local loss improvement coefficient, the consistency-convergence synergy coefficient, the excessive penalty coefficient, and the non-independent and identically distributed adaptation penalty coefficient. S2, each client performs local model training based on local private data and uploads the training results to the central server. The training results include local model parameters and local loss improvement rate. S3, the central server extracts global state features and local state features of each client based on the training results uploaded by the client, splices them together to form joint state features of the reinforcement learning agent, and substitutes them into the multi-objective dynamic activation function to calculate the dynamic reward value of each client. The multi-objective dynamic activation function is calculated based on the change in global validation set accuracy, local loss improvement rate, client parameter consistency index, regularization term weight, non-independent and identically distributed penalty term, training fluctuation penalty term, and global parameter convergence speed index, combined with the core coefficients of the multi-objective dynamic activation function consisting of accuracy reward coefficient, local loss improvement coefficient, consistency-convergence synergy coefficient, excessive penalty coefficient, and non-independent and identically distributed adaptation penalty coefficient. S4, the central server outputs the updated regularization weights by the reinforcement learning agent based on the joint state features and the dynamic reward value; S5, performing differentiated adaptive updates on the multiple core coefficients of the multi-objective dynamic activation function, specifically including: The accuracy reward coefficient is updated using an exponential function combined with historical cumulative feedback for gradient ascent. The local loss improvement coefficient is updated using a threshold-triggered update. When the local loss improvement exceeds a preset threshold, it is accelerated by the square root function; otherwise, it decays slowly. The consistency-convergence synergy coefficient is updated using a dual-feedback balance that directly integrates consistency feedback and convergence speed feedback; The excessive penalty coefficient is updated using a squared nonlinear penalty based on the deviation between the updated regularization term weight and the preset ideal value. The non-independent and identically distributed fitting penalty coefficient is updated using an exponential function based on the coupling of heterogeneity grading and volatility suppression; The updated coefficients are all restricted to their corresponding value boundaries by the clipping function; S6, the central server uses an attention mechanism to dynamically weight and aggregate the local model parameters based on the updated regularization weights and the client aggregation weights to generate a global model. S7, the central server sends the global model back to each client and determines whether the convergence condition has been met. If it has not converged, it iterates through steps S2 to S6; otherwise, it outputs the final global model.

2. The method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning according to claim 1, characterized in that, S2 includes: Each client generates synthetic training data based on local private data, and uses the synthetic training data as samples, combined with the current regularization weights, to train the local model parameters for this round, based on the global model received from the central server in the previous round. Calculate the local loss improvement rate: ； in, Let be the loss improvement rate of client i in round t; This represents the baseline loss value of client i at the start of training; This represents the loss value of client i after the t-th round of training; Indicates an indicator function; `max` indicates that only positive improvements are retained. This represents the minimum finite threshold at which the loss decreases.

3. The method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning according to claim 1, characterized in that, The multi-objective dynamic activation function is specifically as follows: ； in, Let be the dynamic reward value for client i in round t; This is the accuracy bonus coefficient; This is the local loss improvement factor; The consistency-convergence coefficient; The penalty coefficient is too high; The fitting penalty coefficient is for non-independent and identically distributed fits. Let be the change in accuracy of the global validation set in round t. , This represents the change in global validation set accuracy after the t-th round of local training and dynamic weighted aggregation. The improvement rate of the client's local loss in round t; This is the consistency index of client parameters in round t; The updated regularization weights; This is the penalty term for non-independent and identically distributed events in round t; The training fluctuation penalty term for the t-th round; Let be the convergence speed index of the global parameters in round t.

4. The method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning according to claim 1, characterized in that, The differential adaptive update of multiple core coefficients of the multi-objective dynamic activation function includes: Accuracy Bonus The update formula is: ； in, This refers to the historical cumulative feedback for client i up to round t. These represent the upper and lower boundaries of the accuracy bonus coefficient; This represents the change in accuracy on the global validation set. clip This is the clipping function; , These are the accuracy bonus coefficients for the current round and the next round, respectively. Update the adjustment parameters for the accuracy bonus coefficient; This is the priority vector for the accuracy reward coefficient; Local loss improvement coefficient The update formula is: ； in, The loss improvement threshold is used to distinguish between effective and ineffective improvements; Improvements for local losses; , These are the local loss improvement coefficients for the current round and the next round, respectively. These represent the upper and lower boundaries of the local loss improvement coefficient; Update the adjustment parameters to improve the local loss coefficient; This is the priority vector for the local loss improvement coefficients; Consistency-Convergence Coefficient The update formula is: ； in, , These are the consistency-convergence coefficients for the current round and the next round, respectively. This is a globally consistent feedback value; This is the global convergence speed feedback value; Update the adjustment parameters to achieve consistency-convergence synergy. This is the priority vector of consistency-convergence synergy coefficients; These are the upper and lower boundaries of the consistency-convergence synergy coefficients, respectively. Excessive penalty coefficient The update formula is: ； in, , These are the excessive penalty coefficients for the current round and the next round, respectively. These represent the upper and lower boundaries of the excessive penalty coefficient; Update the adjustment parameters to compensate for excessively high penalty coefficients; This represents the ideal value for the regularization term weight; The updated regular expression weights; The priority vector for excessively large penalty coefficients; Non-independent and identically distributed fitness penalty coefficient The update formula is: ； in, , These are the non-independent identically distributed adaptation penalty coefficients for the current round and the next round, respectively. The adjustment parameters are updated by fitting the penalty coefficient to non-independent and identically distributed systems. These are the upper and lower boundaries of the non-independent and identically distributed fitness penalty coefficient, respectively. For coupling feedback terms; This is the priority vector for non-independent and identically distributed adaptation penalty coefficients.

5. The method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning according to claim 1, characterized in that, The method further includes: Calculate the weighted average regularization weight, weighted average heterogeneity feedback, and weighted average training fluctuation feedback based on the updated regularization weight, non-independent and identically distributed penalty, and training fluctuation penalty for each client. Generate regularization term correlation coefficients based on the weighted average regularization term weights: ； in, This is the adjustment coefficient for the weight of the regularization term; These are the lower and upper bounds of the correlation coefficient for the regularization term weights, respectively; The correlation coefficient is the weight of the regularization term; The weights are the weights of the weighted average regularization term; The updated regularization weights; clip This is the clipping function; This represents the ideal value for the regularization term weight; These are the lower and upper bounds of the regularization term weights, respectively; The dynamic correction factor is calculated based on the weighted average heterogeneous feedback and the weighted average training fluctuation feedback: ； in, , These are the heterogeneity feedback adjustment coefficient and the training fluctuation feedback adjustment coefficient, respectively. For weighted average heterogeneous feedback; For weighted average training fluctuation feedback; These are the lower and upper limits of the dynamic correction factor, respectively; It is a dynamic correction factor; The actual update interval is calculated based on the preset base update interval T0, the regularization term weight correlation coefficient, and the dynamic correction factor: ； in, These are the lower and upper limits of the update interval, respectively; T t This refers to the actual update interval; round is a function that evaluates a function.

6. The method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning according to claim 1, characterized in that, S4 specifically includes: Reinforcement learning agents use joint state features As input, through the policy network Output intermediate action values And generate candidate updated regularization term weights based on the range of regularization term weight values. , where sigmoid is the activation function; Boundary protection is performed on the candidate updated regularization weights to obtain the final updated regularization weights. ,in These are the lower and upper bounds of the regularization term weights, respectively.

7. The method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning according to claim 6, characterized in that, If the candidate updated regularization term weight Then additional penalties will be applied. The value is added to the dynamic reward value, where k is the boundary violation penalty coefficient, and k >

0.

8. The method for dynamically adjusting the weights of regularization terms in federated learning based on reinforcement learning according to claim 1, characterized in that, S6 specifically includes: Based on the updated regularization weights and the client aggregation weights, the dynamic aggregation weights for each client are calculated using an attention mechanism. Based on dynamic aggregation weights, the local model parameters uploaded by all clients are weighted and summed to generate intermediate parameters for the global model. The intermediate parameters are fine-tuned and optimized to obtain the final global model.

9. A federated learning regularization term weight dynamic adjustment system based on reinforcement learning, characterized in that, include: Multiple clients, each client performs local model training based on local private data, and uploads the training results to the central server. The training results include local model parameters and local loss improvement rate. A central server, which communicates with each client, includes: The state feature extraction module is used to extract global state features and local state features of each client based on the training results uploaded by the client, and splice them together to form the joint state features of the reinforcement learning agent. The dynamic incentive calculation module is used to substitute the joint state features into the multi-objective dynamic incentive function to calculate the dynamic reward value of each client. The multi-objective dynamic incentive function is calculated based on the global validation set accuracy change, local loss improvement rate, client parameter consistency index, regularization term weight, non-independent and identically distributed penalty term, training fluctuation penalty term, and global parameter convergence speed index, combined with the core coefficients of the multi-objective dynamic incentive function, which consist of accuracy reward coefficient, local loss improvement coefficient, consistency-convergence synergy coefficient, excessive penalty coefficient, and non-independent and identically distributed adaptation penalty coefficient. The reinforcement learning decision module is used to output updated regularization weights by the reinforcement learning agent based on the joint state features and the dynamic reward value. The coefficient update module is used to perform differentiated adaptive updates on multiple core coefficients of the multi-objective dynamic activation function based on the joint state features and the updated regularization term weights, specifically including: The accuracy reward coefficient is updated using an exponential function combined with historical cumulative feedback for gradient ascent. The local loss improvement coefficient is updated using a threshold-triggered update. When the local loss improvement exceeds a preset threshold, it is accelerated by the square root function; otherwise, it decays slowly. The consistency-convergence synergy coefficient is updated using a dual-feedback balance that directly integrates consistency feedback and convergence speed feedback; The excessive penalty coefficient is updated using a squared nonlinear penalty based on the deviation between the updated regularization term weight and the preset ideal value. The non-independent and identically distributed fitting penalty coefficient is updated using an exponential function based on the coupling of heterogeneity grading and volatility suppression; The updated coefficients are all restricted to their corresponding value boundaries by the clipping function; The dynamic aggregation module is used to dynamically weight and aggregate local model parameters based on the updated regularization weights and client-side aggregation weights, using an attention mechanism to generate a global model. The convergence judgment module is used to determine whether the convergence condition is met and to control the training iteration.

Images