A parameter quantization method for federated learning models based on deep reinforcement learning

Abstract

A method for parameter quantization of federated learning models based on deep reinforcement learning includes the following steps: S1, obtaining the current global model parameters; S2, statistically analyzing the M percentiles of the current global model parameters as the environmental state observed by the deep reinforcement learning agent; S3, constructing L quantization order points according to a given rule based on the agent's output actions, serving as a mapping set for quantization operations; S4, quantization transmission: performing X rounds of model quantization and transmission, with each round using the quantization mapping set from the previous step, and statistically analyzing the quantization error and training error over these X rounds, calculating their mean, and obtaining the reward value according to the reward function, which is then input into the agent as feedback; S5, the agent continuously records the state, actions, and reward status for each round, updating the agent's network model when the number of records reaches a given threshold. This method exhibits small quantization error and high test accuracy.

Description

Technical Field

[0001] This invention relates to the field of distributed artificial intelligence, and in particular to a method for quantizing parameters of a federated learning model based on deep reinforcement learning. Background Technology

[0002] Machine learning (ML) is one of the most representative artificial intelligence technologies. It can learn optimization strategies from massive amounts of data and has shown great potential in many applications, especially in computer vision and natural language processing. However, with increasing emphasis on privacy by individuals, institutions, and even nations, collecting large amounts of data and placing it on a central server for training has become increasingly impractical. Federated learning (FL), as a distributed ML method, has emerged to address this issue. It completes training through model exchange between users and servers, without requiring the upload of users' original data, thus mitigating privacy concerns to some extent. However, machine learning network models are large in scale and require hundreds or thousands of iterations to converge. This results in the training process of federated learning consuming significant communication resources, a problem particularly prominent in wireless communication systems.

[0003] To improve the communication efficiency of wireless federated learning (FL) and reduce the communication overhead required for distributed model training, compression of the federated learning model is essential. Current main model compression methods include low-rank approximation, sparsity reduction, and quantization. Quantization refers to converting high-precision neural network model parameters into low-precision approximations that can be represented by a few bits (i.e., quantization bit width) for transmission; this method has been proven to significantly reduce communication overhead without excessively affecting the performance of the neural network model. For example, some researchers have proposed randomly rounding the model parameter vector to a finite set of discrete values ​​and utilizing the unequal probabilities of these discrete values ​​to perform efficient lossless encoding of the model, which can effectively improve the communication efficiency of FL.

[0004] Existing technical solutions primarily focus on model parameter quantization during uplink communication (user transmitting the model to the server), while paying less attention to model parameter quantization during downlink communication (server broadcasting the model to the user). Some researchers have considered the quantization compression problem in downlink communication, proposing algorithms that only transmit the difference between the broadcast global model and the previous model. Leveraging the smaller dynamic range of the difference compared to the model itself, this approach can achieve lower errors. However, regardless of whether the user participates in training, the latest global model must be downloaded every round, which adds additional communication overhead from the user's perspective. Furthermore, current technical solutions only consider uniform quantization schemes because while non-uniform quantization can further reduce quantization errors, setting and optimizing the quantization order is quite difficult. Existing wireless federated learning model parameter quantization methods suffer significant performance degradation at low quantization bit widths.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to propose a parameter quantization method for federated learning models based on deep reinforcement learning, so as to reduce the performance loss when the model is transmitted using a low quantization bit width.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In the first aspect, a method for quantizing parameters of a federated learning model based on deep reinforcement learning includes the following steps:

[0009] S1. Get the current global model parameters;

[0010] S2. State Processing: The M percentiles of the current global model parameters are counted as the environmental state observed by the deep reinforcement learning agent; wherein, the deep reinforcement learning agent includes an action network and an evaluation network;

[0011] S3. Action processing: Based on the action output by the deep reinforcement learning agent, construct L quantization order points according to given rules, as a mapping set for quantization operations;

[0012] S4. Quantization and Transmission: Perform model quantization and transmission for X rounds. Each round of quantization uses the quantization mapping set from the previous step. Calculate the quantization error and training error for these X rounds, calculate their mean, and obtain the reward value according to the reward function. Input the reward value into the reinforcement learning agent as feedback.

[0013] S5. Model Update: The reinforcement learning agent continuously records the state, action, and reward each time. When the number of records reaches a given threshold, the network model of the deep reinforcement learning agent is updated.

[0014] In a second aspect, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the aforementioned method for quantizing parameters of a federated learning model based on deep reinforcement learning.

[0015] The present invention has the following beneficial effects:

[0016] This invention proposes a parameter quantization method for federated learning models based on deep reinforcement learning. Existing wireless federated learning model parameter quantization methods suffer severe performance degradation at low bit widths. The method of this invention can effectively reduce the performance loss caused by low quantization bit widths during model transmission by interacting with the environment to formulate a reasonable quantization strategy. This invention reduces the performance loss of low quantization bit widths; therefore, to achieve equivalent performance, a lower quantization bit width can be used than in general methods, indirectly reducing communication overhead. The method of this invention employs non-uniform quantization and autonomously optimizes the selection of quantization order points through a reinforcement learning agent. It is applicable to both uplink and downlink communication links and achieves smaller quantization errors and higher test accuracy compared to traditional uniform quantization methods. Comparative testing shows that, under the same training epochs, the model using the method of this invention achieves higher accuracy on the test set. Attached Figure Description

[0017] Figure 1 This is a flowchart of an embodiment of the present invention.

[0018] Figure 2 This is the parameter quantization process for a federated learning model based on deep reinforcement learning, as described in an embodiment of the present invention.

[0019] Figure 3 This is a comparison chart of training and quantization errors between embodiments of the present invention and existing methods.

[0020] Figure 4 This is a comparison chart of the test accuracy of the embodiments of the present invention and existing methods. Detailed Implementation

[0021] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0022] See Figure 1 This invention provides a method for parameter quantization of a federated learning model based on deep reinforcement learning, comprising the following steps:

[0023] S1. Get the current global model parameters;

[0024] S2. State Processing: The M percentiles of the current global model parameters are counted as the environmental state observed by the deep reinforcement learning agent; wherein, the deep reinforcement learning agent includes two deep neural networks: an action network and an evaluation network.

[0025] S3. Action processing: Based on the action output by the deep reinforcement learning agent, construct L quantization order points according to given rules, as a mapping set for quantization operations;

[0026] S4. Quantization and Transmission: Perform model quantization and transmission for X rounds. Each round of quantization uses the quantization mapping set from the previous step. Calculate the quantization error and training error for these X rounds, calculate their mean, and obtain the reward value according to the reward function. Input the reward value into the reinforcement learning agent as feedback.

[0027] S5. Model Update: The reinforcement learning agent continuously records the state, action, and reward each time. When the number of records reaches a given threshold, the network model of the deep reinforcement learning agent is updated.

[0028] This invention proposes a parameter quantization method for federated learning models based on deep reinforcement learning. By interacting with the environment to formulate a reasonable quantization strategy, it effectively reduces the performance loss caused by the model using low quantization bit widths for transmission. This method employs non-uniform quantization and autonomously optimizes the selection of quantization order points through a reinforcement learning agent. It is applicable to both uplink and downlink communication links and achieves smaller quantization errors and higher test accuracy compared to traditional uniform quantization methods.

[0029] The following describes specific embodiments of the present invention.

[0030] The parameter quantization method for federated learning models based on deep reinforcement learning, as described in this invention, mainly includes: constructing a deep reinforcement learning agent, which comprises two deep neural networks: an action network and an evaluation network; state processing, in which the system statistically analyzes the M percentiles of the current global model parameters as the environmental state observed by the reinforcement learning agent; subsequent action processing, in which the system constructs L quantization order points according to a given rule based on the actions output by the reinforcement learning agent, serving as a mapping set for quantization operations; next, quantization transmission, in which the system performs X rounds of model quantization and transmission, with each round using the quantization mapping set from the previous step, and statistically analyzes the quantization error and training error over these X rounds, calculates their mean, and obtains the reward value according to the reward function, which is then input into the reinforcement learning agent as feedback; finally, the reinforcement learning agent continuously records the state, action, and reward status each time during this process, and updates the agent's network model when the number of these records reaches a given threshold. The specific process is as follows: Figure 2 As shown.

[0031] 1. Constructing reinforcement learning intelligent agents

[0032] The Deep Reinforcement Learning (DRL) agent is the core unit responsible for decision-making in the method of this invention. It consists of two neural networks, including: an action network π(a,s; θ) a ), where θ α Let π(a, s; θ) represent the parameters of the neural network, and s represent the state vector observed by the agent at the moment of observation. a The network V(s; θ) is evaluated based on the probability that the agent will decide on action 'a' in this state. v ), where θ v This represents the parameters of the neural network, and the function gives the value of the current state s as perceived by the agent.

[0033] 2. Status handling

[0034] Assume the neural network model to be transmitted is Where d is the dimension of the network model, that is, the number of parameters it contains. Assume that w is obtained by sorting the parameters in w in ascending order of their absolute values. s Define the xth percentile as

[0035]

[0036] in, It is w s The i-th element, This indicates rounding up to the nearest integer.

[0037] Assuming the dimension of the input state of the reinforcement learning agent is M≥1, construct the following vector.

[0038] s = [p1, p2, ..., p M ] T ,

[0039] As the observed state at the current moment, the m-th element of this vector is

[0040]

[0041] Here, by adjusting the value of M, a balance can be achieved between representational precision and state space dimension.

[0042] 3. Action processing

[0043] Assume the action dimension of the DRL agent output is L / 2. After obtaining the state vector s in step 4.2, it is input into the action network π(a|s; θ) of the DRL agent. a This allows us to obtain the network's output vector.

[0044] μ = [μ1, μ2, ..., μ L / 2 ] T .

[0045] Given the sampling variance of all actions as σ, then from the normal distribution N(μ) i Random sampling is performed in σ to obtain action a. i Let i = 1, 2, ..., L / 2, and form an action vector.

[0046] a = [a1, a2, ..., a L / 2 ] T ,

[0047] Based on this vector, the quantization order points and the complete mapping set can be further calculated. Assuming that the quantization order points are searched and optimized only within the given range [-B, B], B > 0, the interval [-B, 0] is divided into L / 2 sub-intervals, and the i-th interval is denoted as I. i =[l i ,u i ], where the left endpoint l i , right endpoint u i Calculate according to the following formulas:

[0048]

[0049] And define a uniformly distributed reference vector c, whose i-th element is the center point of the i-th sub-interval, i.e.

[0050]

[0051] Next, the quantization order point vector for the negative part is calculated.

[0052]

[0053] Sort(x) represents sorting the elements of vector x in ascending order. Based on this, the quantization order point vector for the non-negative part can be calculated.

[0054]

[0055] It can be derived from vector q n Reverse the order and then take the opposite number to get q. n ,q p Concatenating two vectors together yields a complete quantized point vector of dimension L.

[0056]

[0057] 4. Quantization transmission

[0058] Based on the quantization order point vector q obtained in step 4.3, a mapping set Q = {q1, q2, ..., q} can be constructed. L}, where q l It is the l-th element of vector q. A round consists of the server broadcasting the global model, the user device receiving and training the model, the user device uploading the model, and the server aggregating the uploaded models. Assuming the current global model is w, the quantization transmission will proceed according to the following steps over the next X rounds:

[0059] The first step, assuming the current round is t, where t = 1, 2, ..., X, is to randomly quantize each parameter w in the global model for the current round according to the following rules:

[0060]

[0061] Wherein ξ(w,q) i ,q j ) is a random variable, defined as

[0062]

[0063] Assume the quantized model parameters are denoted as Q(w) = [Q(w1), Q(w2), ..., Q(w... d )] T Then the quantization error e for that round can be calculated. t =||Q(w)-w|| 2 .

[0064] The second step involves broadcasting the quantized global model Q(w) to the users, who then train their own models on their local datasets based on this model. Assuming the user set is S, and the training error of the k-th user is... After each user completes training, the updated model w will be sent to them. k The training error and the data are uploaded to the server, and the server determines the data based on {w}. k} k∈S The aggregation yields a new global model w, and the average training error is calculated using the following formula:

[0065]

[0066] After X rounds of computation, a total of X quantization error and average training error values ​​can be collected. Their mean is calculated using the following formula:

[0067]

[0068] Further calculate the reward for this training process based on them.

[0069]

[0070] in, Let represent the average training error obtained from the previous quantization transfer. If this is the first quantization transfer, then let . This condition is true; if steps S1 to S5 are considered an iterative process, then this variable represents the result obtained in this step of the previous iterative process. α>1 is a constant factor used to amplify the impact of quantization error; β1,β2>0 are also constant weight factors used to adjust the impact of quantization error and training error on the reward value; I(·) is an indicator function, which takes the value of 1 when the condition in parentheses is true, and takes the value of 0 otherwise. Δ>0 is the penalty given when gradient explosion occurs during the training process, i.e., when there is a value in w that exceeds INF, where INF represents a very large value (usually 1.796E308).

[0071] 5. Model Update

[0072] In step 4.2, the agent acquires state s; in step 4.3, the agent outputs action a; in step 4.4, the agent receives reward r and can calculate the next state s' based on the latest w using the method in step 4.2. The quadruple (s, a, r, s') from this iteration is stored in the agent's record buffer B. If the number of records in the buffer is less than a given threshold P, the next record is generated according to steps 4.2-4.3; otherwise, the agent model is updated according to the following steps.

[0073] First, assume the i-th record in the cache is p.i =(s i ,a i ,r i ,s' i Let A, where i = 1, 2, ..., P. P =0, and calculate the dominance function for each state-action pair in turn according to the following formula.

[0074] A i-1 =r i-1 +γ·V(s i ';θ v )-V(s i ;θ v )+γλ·A i ,

[0075] Both γ and λ are constants used to adjust the impact of future gains on current advantages.

[0076] The second step is to group the records in the cache, with each group containing C records. Let θ' a =θ a Then, for each set of data, calculate the c-th data (assuming its position in B is i). c The corresponding action loss function

[0077]

[0078] Where c = 1, 2, ..., C, the probability ratio ρ is defined as:

[0079]

[0080] The clipping function clip(·) is defined as follows:

[0081]

[0082] Update the model using the following formula:

[0083]

[0084] Where, η a It's the learning rate. It is the gradient of the loss function.

[0085] Similarly, let θ' c =θ c Then, for each set of data, calculate the value loss function corresponding to the c-th data point.

[0086]

[0087] Then update the model according to the following formula:

[0088]

[0089] Where, η c It is the learning rate.

[0090] The third step, after completing all the above training, is to update the model, that is, let θ a =θ' a ,θ c =θ' c Then clear all records in cache B. Return to step 4.2 and continue the interactive process to obtain training data.

[0091] Performance Analysis

[0092] To verify the benefits of the method of this invention, simulation experiments were conducted based on the publicly available dataset CIFAR-10. This dataset is an object recognition image dataset, with a training set containing 50,000 samples. In this test case, these samples were randomly divided into 100 non-overlapping smaller datasets to simulate the local datasets of 100 users. During FL training, 10 users were randomly selected for local computation each round, with a local batch size of 50, an epoch of 5, an initial learning rate of 0.15, and a decay rate of 0.99 every 10 rounds.

[0093] For the DRL agent, two identical multilayer perceptron models are used as the action network and the evaluation network, respectively. Each model has 150 neurons in its intermediate layers. During the DRL training phase, each user only uses 20% of their local dataset for training to accelerate the process. The state dimension M is set to 5, and considering 3-bit quantization, the total number of quantization points is L = 8. The action dimension is 4, the action sampling variance σ = 0.1, the optimization range of quantization points R is set to 0.15, and the execution rounds for each action are X = 4. In the reward function, α = 10, β1 and β2 are set to 150000 and 0.3, respectively; the penalty factor Δ is set to 5. In the model training parameters, the cutoff rate ò = 0.2, the reward adjustment factors γ and λ are set to 0.99 and 0.95, respectively, and the learning rate η of the action network and the evaluation network is set to... a ,η c All values ​​were set to 0.0004, the cache threshold was set to P = 2048, and the group size was C = 16. The DRL training iterations were 40,000.

[0094] After training the DRL agent, it was used for 1000 rounds of FL training. All users used the complete training set during this period. Accuracy testing was then performed on a test set containing 10,000 samples to verify the model's performance; higher accuracy indicates better performance. Here, it is compared with traditional uniform quantization, where the quantization order point vector is fixed at 0.0375 × [-4, -3, -2, -1, 1, 2, 3, 4]. T .

[0095] Figure 3 The diagram shows a comparison of the quantization errors of the neural network model parameters for downlink communication processes using existing uniform quantization and the method proposed in this invention on the CIFAR-10 dataset. According to the reward function of the DRL agent, it receives a reward if its quantization order selection reduces both training and quantization errors; otherwise, it is penalized. Therefore, the final scheme achieves lower quantization and training errors compared to the general uniform quantization scheme.

[0096] Figure 4 This paper presents a comparison of the accuracy achieved on the test set using uniform quantization and the method proposed in this invention for downlink communication processes on the CIFAR-10 dataset. Because the proposed method reduces quantization and training errors, it means that the error between the broadcast model received by each user during training and the unquantized model is smaller, and it converges faster on the local dataset. This results in faster model convergence when using the proposed method, leading to higher accuracy on the test set within the same number of training rounds.

[0097] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0102] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A parameter quantization method for federated learning models based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Get the current global model parameters; S2. State Processing: Calculate the M percentiles of the current global model parameters as the environmental state observed by the deep reinforcement learning agent; wherein, the deep reinforcement learning agent includes an action network and an evaluation network; S3. Action Processing: Based on the action output by the deep reinforcement learning agent, construct L quantization points according to given rules as a mapping set for quantization operations; the quantization points are non-uniform quantization points, corresponding to low quantization bit width. S4. Quantization and Transmission: Perform X rounds of model quantization and transmission. Each round of quantization uses the quantization mapping set from the previous step, and calculates the quantization error and training error over these X rounds. The average of these errors is then calculated, and a reward value is obtained based on the reward function, which is input into the deep reinforcement learning agent as feedback. In step S4, the global model parameters are quantized according to the quantization mapping set, and multiple rounds of model quantization and transmission are performed. The model quantization and transmission includes: the server broadcasting the quantized global model to the user device; the user device receiving the model and training it on its local dataset, then uploading the updated model to the server; in each round, the model parameters are randomly quantized, and the quantization error and training error for the current round are calculated; after multiple rounds, the average of the quantization error and training error is calculated, and a reward value is generated based on the reward function and fed back to the deep reinforcement learning agent; the reward function is designed based on the average of the quantization error and training error to guide the agent in optimizing the quantization strategy. S5. Model Update: The reinforcement learning agent continuously records the state, action and reward each time. When the number of records reaches a given threshold, the network model of the deep reinforcement learning agent is updated. This reduces the performance loss when the model uses low quantization bit width for transmission, thereby reducing the communication overhead of wireless federated learning.

2. The parameter quantization method for federated learning models based on deep reinforcement learning as described in claim 1, characterized in that, The action network is represented as ,in This represents the parameters of the neural network. These represent the state vectors observed by the agent at present. Given the action that the agent decides to take in this state. The probability; the evaluation network is represented as ,in This represents the parameters of the neural network. Give the current state as perceived by the agent. The magnitude of its value.

3. The parameter quantization method for federated learning models based on deep reinforcement learning as described in claim 1 or 2, characterized in that, In step S2, let the neural network model to be transmitted be... ,in It is the dimension of the network model, for The parameters in the array are sorted by absolute value from smallest to largest. , define the first The percentile is: ； in, yes The One element, Indicates rounding up; Let the dimension of the input state of the deep reinforcement learning agent be . Construct the following vector: ， As the state observed at the current moment, the first digit of this vector is... m The elements are ； By adjusting The value of is balanced between representational precision and state space dimension.

4. The parameter quantization method for federated learning models based on deep reinforcement learning as described in claim 2, characterized in that, In step S3, the state vector is obtained in step S2. Then, it is input into the action network. Thus, the output vector of the network is obtained: ； Let be the action dimension output by the deep reinforcement learning agent, given the sampling variance of all actions as . Then from the normal distribution Random sampling is used to obtain actions. And form an action vector: ， Based on this vector, calculate the quantization order points and the complete mapping set, where the quantization order points are within a given range. Search and optimize between intervals Average score The number of sub-intervals, the first The intervals are denoted as The left endpoint Right endpoint Calculate according to the following formulas: ； Define a uniformly distributed reference vector. , its first The element is the first The center point of each sub-interval, i.e.: ； Next, the quantization order point vector for the negative part is calculated: ； in Represents a vector The elements in the array are sorted in ascending order; Calculate the quantization order point vector for the non-negative part: ； Will Concatenating two vectors together yields the complete dimension. quantization order vector 。 5. The parameter quantization method for federated learning models based on deep reinforcement learning as described in claim 1 or 2, characterized in that, In step S4, based on the quantization order point vector obtained in step S3... Construct a mapping set ,in It is a vector The Each element; a round consists of the server broadcasting the global model, the user device receiving and training the model, the user device uploading the model, and the server aggregating the uploaded models. For the current global model... In the following Each round is quantized and transmitted according to the following steps: First step, let the current position be the [number]th [unit]. rounds, of which For each parameter in the global model of the current round Random quantization is performed according to the following rules: ， in, It is a random variable, defined as follows: ； The quantized model parameters are denoted as Calculate the quantization error for this round. ; The second step is to use a broadcast-quantized global model. The model is provided to users so that they can train it on their local datasets; the user set is... , No. The training error for each user is The server receives updated models from each user after training. and training error, according to Aggregation yields a new global model The average training error is calculated using the following formula: ； Finish After each round of calculation, the data is collected. The mean of the individual quantization errors and the average training error is calculated using the following formula: ； The reward for this training process is further calculated based on the mean. ； in, Let represent the average training error obtained from the previous quantization transfer. If this is the first quantization transfer, then let . This condition is true; This is a constant factor used to amplify the effect of quantization error; It is a constant weighting factor used to adjust the impact of quantization error and training error on the reward value; This is an indicator function; it takes the value 1 when the condition within the parentheses is true, and 0 otherwise. It is a gradient explosion that occurs during the training process, i.e. There are large values ​​in it that exceed the set value. The punishment given at that time.

6. The parameter quantization method for federated learning models based on deep reinforcement learning as described in claim 1 or 2, characterized in that, In step S5, based on the state obtained by the agent in step S2... The action output by the agent in step S3 The reward received by the agent in step S4 and based on the latest The next state is calculated based on the method in step S2. The quadruplets in such an iteration process The number of records stored in the agent's record buffer is less than a given threshold. If the record is correct, continue with steps S2-S4 to generate the next record; otherwise, update the agent model according to the following steps: First step, assuming the cache contains the first... The record is ,in ;make And calculate the dominance function for each state-action pair in turn according to the following formula: ； in and All are constants, used to adjust for the impact of future gains on current advantages; The second step is to group the records in the cache, with each group including... Record; Order Then, for each set of data, calculate the first... The action loss function corresponding to each data point: ； in , No. Data in The position in the middle is probability ratio for: ， Clipping function for ； Update the model using the following formula: ， in, It's the learning rate. It is the gradient of the loss function; make Then, for each set of data, calculate the first... Value loss function corresponding to each data point ； Then update the model using the following formula: ， in, It is the learning rate; The third step, after completing all the above training, is to update the model, that is, to let Then clear all records in the cache; return to step S2 and continue to interact to obtain training data.

7. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when run by a processor, implements the parameter quantization method for federated learning models based on deep reinforcement learning as described in any one of claims 1 to 6.

Images