A model gradient updating method and device

Abstract

The present application provides a model gradient update method and device to improve the accuracy of model training. The central server repeatedly executes the gradient update process until the stop condition is met; wherein, a gradient update process includes: receiving the first gradients sent by multiple nodes respectively, the first gradient is obtained by training the model to be trained in the node once or multiple times using sample data for each node; obtaining the second gradient based on the multiple first gradients and the probability of each node in this gradient update process, the probability of each node in this gradient update process is determined by the Actor-Critic network based on the probability of each node in the previous gradient update process; sending the second gradient to multiple nodes respectively, so that multiple nodes use the second gradient to update the weights of their respective models to be trained. Taking into account the probability of each node, the node participation can be optimized to make the determined model better.

Description

Technical Field

[0001] The present application relates to the field of model training technology, and in particular to a model gradient updating method and device. Background Art

[0002] Horizontal federated learning, also known as sample-based federated learning, can be applied to scenarios where the data sets of various participants in federated learning have the same characteristics but different samples.

[0003] It is usually assumed that all participants in a horizontal federated learning system are honest, and the object to be guarded against is an honest but curious central server. That is, it is usually assumed that only the central server can threaten the privacy and security of data participants. In a horizontal federated learning system, multiple participants with the same data characteristics collaboratively train a model with the help of a central server. The main steps include: Each participant calculates the model gradient locally and sends the gradient (the gradient needs to be encrypted) to the central server. The central server aggregates multiple gradients. The central server sends the aggregated gradient (the gradient also needs to be encrypted) to each participant. Each participant uses the received gradient to update its own model parameters.

[0004] The above steps are iterated continuously until the loss function converges or reaches the upper limit of the allowed number of iterations or the allowed training time. This architecture is independent of specific machine learning algorithms (such as logistic regression and deep neural networks), and all participants will share the final model parameters.

[0005] Currently, in horizontal federated learning scenarios, the central server averages and aggregates the gradients. Considering the different performances of different participants, the model trained using gradient averaging produces poor results. Summary of the invention

[0006] The present application provides a method and device for updating a model gradient to improve the accuracy of model training.

[0007] To achieve the above object, the embodiment of the present application discloses a model gradient updating method, which is applied to a central server and includes:

[0008] The central server repeatedly executes the gradient update process until the stop condition is met; wherein the gradient update process once includes:

[0009] Receive first gradients sent by multiple nodes respectively, where the first gradients are obtained by training the model to be trained in the node once or multiple times using sample data for each node; obtain a second gradient based on the multiple first gradients and the probability of each node in this gradient update process, where the probability of each node in this gradient update process is determined by the Actor-Critic network based on the probability of each node in the previous gradient update process; send the second gradients to the multiple nodes respectively, so that the multiple nodes use the second gradient to update the weights of their respective models to be trained.

[0010] In an optional example, the Actor-Critic network includes an Actor network, at least one Critic network, and a reward function;

[0011] The reward function is used to determine a reward value based on the probabilities of the multiple nodes determined in the last gradient update process, and transmit the reward value to the at least one Critic network;

[0012] The at least one Critic network is used to determine a target Q value and transmit the target Q value to the Actor network;

[0013] The Actor network is used to determine the probability of each node in this gradient update process based on the target Q value.

[0014] In an optional example, the target Q value is the minimum Q value among the Q values ​​determined by multiple Critic networks.

[0015] In an optional example, the reward function satisfies:

[0016]

[0017] Among them, A is the first accuracy rate, B is the second accuracy rate, and g is greater than or equal to 1, wherein the first accuracy rate is the accuracy rate of the trained model obtained by the central server and the multiple nodes based on the federated average learning algorithm; the second accuracy rate is the average value of the third accuracy rates sent by the multiple nodes respectively, and the third accuracy rate is obtained in the same model training when the model to be trained in the node is trained by the sample data at the node as the first gradient.

[0018] In an alternative example, when When is greater than 1, g is greater than 1; when When it is less than or equal to 1, g is 1.

[0019] In an optional example, the Actor-Critic network includes three Critic networks. For any Critic network, the Q value determined in this gradient update process is based on the Q value gradient and the Q value determined in the previous gradient update process. The Q value gradient is determined based on a first parameter, and the first parameter satisfies the following formula:

[0020]

[0021] in,

[0022] Wherein, J is the first parameter; t is the number of times the gradient is updated; k>0, l>0, k+l=1; θ 1 ,θ 2 ,θ 3 Represents three Critic networks, θ i is θ 1 ,θ 2 ,θ 3 They represent the network corresponding to the minimum value of the Q value recently determined by the three Critic networks; t is the state in the tth gradient update process; a t is the probability of the multiple nodes in the t-th gradient update process; is θ in the tth gradient update process i The corresponding Critic network is in s t ,a t The Q value determined in the case; is θ in the tth gradient update process 3 The corresponding Critic network is in s t ,a t The output Q value in the case of t ,a t ) is the gradient update process in the tth time at s t ,a t The reward value in the case; γ is greater than 0; π t (a t |s t ) is in s t Make a t The probability of; q is the exponent of entropy, ln q is entropy, α t Not 0.

[0023] In an optional example, the α used in this gradient update process is determined based on the α gradient and the α used in the previous gradient update process, and the α gradient satisfies the following formula:

[0024]

[0025] Among them, J(α) is the gradient of α, α t-1 is the α used in the last gradient update, and H is the ideal minimum expected entropy.

[0026] In an optional example, the k and l are determined based on the variance of the accuracy of the model before the gradient update is performed in the multiple nodes.

[0027] In an optional example, the probability of the node output by the Actor network during this gradient update process is determined based on the probability gradient and the probability of the node output during the previous gradient update process, and the probability gradient satisfies the following formula:

[0028]

[0029] Among them, J(π φ ) is the probability gradient, t is the number of times the gradient is updated; θ 1 ,θ 2 ,θ 3 Represents three Critic networks, θ i is θ 1 ,θ 2 ,θ 3 They represent the network corresponding to the minimum value of the Q value recently determined by the three Critic networks; t is the state in the tth gradient update process; a t is the probability of the multiple nodes in the t-th gradient update process; Q θi (s t ,a t ) is the value of θ in the tth gradient update process i The corresponding Critic network is in s t ,a t Q value determined under the circumstances; π t (a t |s t ) is in s t Make a t The probability of; q is the exponent of entropy, ln q is entropy, α t is the α used in this gradient update, α t Not 0.

[0030] In an optional example, the central server and the multiple nodes perform gradient updates based on a federated learning architecture.

[0031] The embodiment of the present application provides a model gradient updating device, comprising:

[0032] Repeat the gradient updating process until the stopping condition is met; wherein the gradient updating process once includes:

[0033] A receiving module, configured to receive first gradients respectively sent by a plurality of nodes, wherein the first gradients are obtained by training a model to be trained in each node once or multiple times using sample data;

[0034] A processing module, configured to obtain a second gradient based on the plurality of first gradients and a probability of each node in the current gradient update process, wherein the probability of each node in the current gradient update process is determined by the Actor-Critic network based on the probability of each node in the previous gradient update process;

[0035] A sending module is used to send the second gradient to the multiple nodes respectively, so that the multiple nodes use the second gradient to update the weights of their respective models to be trained.

[0036] The embodiment of the present application provides a model gradient updating device, including a processor and a memory;

[0037] The memory is used to store computer programs or instructions;

[0038] The processor is used to execute part or all of the computer programs or instructions in the memory, and when the part or all of the computer programs or instructions are executed, it is used to implement any of the above-mentioned model gradient updating methods.

[0039] An embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program includes instructions for implementing any one of the model gradient updating methods described.

[0040] This application takes into account the probability of each node and can optimize the node participation to make the determined model better. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a model gradient update process provided for this application;

[0043] Figure 2 A model gradient update system architecture diagram provided for this application;

[0044] Figure 3 A structural diagram of a model gradient updating device provided in this application;

[0045] Figure 4 A structural diagram of a model gradient updating device provided in this application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The central server repeats the gradient update process until a stop condition is met, such as when the loss function converges, when the upper limit of the allowed number of iterations is reached, or when the allowed training time is reached.

[0048] Next, Figure 1 As shown, any gradient update process is introduced:

[0049] Step 101: multiple nodes (eg, node 1, node 2, ..., node m) send first gradients to a central server respectively. Correspondingly, the central server receives the first gradients sent by the multiple nodes respectively.

[0050] The first gradient is obtained by training the model to be trained in each node once or multiple times using sample data. The gradient sent by the node to the central server is called the first gradient, and the first gradients sent by multiple nodes may be the same or different.

[0051] The node can be a car terminal, and the sample data can be the data generated during autonomous driving. Different driving data is generated during driving, and the driving data is scattered across various nodes (car terminals), with uneven data quality and uneven node performance. The model can be a model that requires autonomous driving and is related to user habits.

[0052] Step 102: The Actor-Critic network determines the probability of each node in this gradient update process based on the probability of each node in the previous gradient update process.

[0053] It can be understood that if this gradient update is the first gradient update, the probability of each node in the previous gradient update process is the probability of each node initially set.

[0054] The sum of the probabilities of multiple nodes can be 1.

[0055] The Actor-Critic network can output the probabilities of multiple nodes in each loop, or it can output the probabilities of nodes after multiple loops.

[0056] The Actor-Critic network may be on a central server or not.

[0057] Step 103: The central server obtains a second gradient based on multiple first gradients and multiple probabilities.

[0058] When the second gradient is obtained based on multiple first gradients and multiple probabilities, the multiple first gradients and multiple probabilities can be used to determine the second gradient in a weighted average manner. For example, the first gradients sent by node 1, node 2, and node 3 are p1, p2, and p3, respectively, and the probabilities of the three nodes are 0.2, 0.4, and 0.4, then the second gradient is 0.2p1+0.4p2+0.4p3. This process can also be called a deep reinforcement learning data fusion algorithm based on multiple first gradients and multiple probabilities to obtain the second gradient.

[0059] Step 104: the central server sends the second gradient to the multiple nodes (eg, node 1, node 2, ... node m) respectively, and correspondingly, the multiple nodes receive the second gradient from the central server.

[0060] The gradient sent by the central server to the node is called the second gradient, and the second gradient sent to multiple nodes is the same.

[0061] Step 105: Multiple nodes use the second gradient to update the weights of their respective models to be trained.

[0062] If the stopping condition is met, the updated model is the trained model.

[0063] If the stopping condition is not met, each node may use sample data to train the model to be trained in the node once or multiple times to obtain a new first gradient, and continue to repeat steps 101 to 105.

[0064] Taking into account the probability of each node, the node participation can be optimized to make the determined model better.

[0065] Optionally, the central server and the multiple nodes perform gradient updates based on a federated learning architecture. The sample data of each node is private and not shared with other nodes and the central server, and when the node and the central server transmit the first gradient and the second gradient, the first gradient and the second gradient are encrypted.

[0066] like Figure 2As shown in the figure, a gradient update system architecture diagram is introduced, including multiple nodes, a central server, and an Actor-Critic network. The Actor-Critic network can be located on the central server or not. As the name suggests, the Actor-Critic consists of two parts, the actor and the critic. The actor is responsible for generating actions and interacting with the environment. The critic is responsible for evaluating the performance of the actor and guiding the actor's actions in the next stage.

[0067] The Actor-Critic network includes an Actor network, at least one Critic network, and a reward function;

[0068] The reward function is used to determine a reward value based on the probabilities of the multiple nodes determined in the last gradient update process, and transmit the reward value to the at least one Critic network;

[0069] The at least one Critic network is used to determine the target Q value and transmit the target Q value to the Actor network. Each Critic network determines a Q value. If there is one Critic network, the Q value determined by the Critic network is the target Q value. If there are multiple Critic networks, one Q value can be selected from multiple Q values ​​as the target Q value. For example, the target Q value is the minimum Q value among the Q values ​​determined by multiple Critic networks. When there are multiple Critic networks, it is equivalent to setting up multiple evaluators, evaluating the performance of the Actor more accurately, making the actions taken by the Actor more accurate, and then the probabilities of multiple nodes obtained conform to the performance of multiple nodes.

[0070] The Actor network determines the probability of each node in this gradient update process based on the target Q value, and transmits the probabilities of the multiple nodes to the reward function, and repeats multiple times until it stops.

[0071] The present application can use the existing Actor-Critic network to determine the probability, and can also improve the existing Actor-Critic network, such as setting up multiple Critic networks, such as improving the algorithms involved in the Critic network, such as improving the algorithms involved in the Actor network, such as improving the reward function. It can be understood that the improvement only involves specific details, and the operating mechanism of the improved Actor-Critic network is similar to that of the existing Actor-Critic network.

[0072] Combination Figure 2 The system introduced hereby provides a detailed introduction to the gradient updating process of this application.

[0073] The parameters involved in the Actor-Critic network may be initialized first, including but not limited to: initializing the parameters involved in the Critic network, the parameters involved in the Actor network, and the parameters involved in the reward function.

[0074] The central server and multiple nodes obtain a trained model based on a federated average learning algorithm, and determine a first accuracy rate A of the trained model.

[0075] Node 1, node 2, ..., node m perform one or more trainings based on the currently saved model (which may also be referred to as the model to be trained, for example, a model that is still executing the gradient update process of step 101 to step 105, or a model that has executed the gradient update process of step 101 to step 105 one or more times) to obtain a first gradient and a third accuracy rate B'. Each node sends the first gradient and the third accuracy rate B' to the central server. The third accuracy rate and the first gradient are obtained in the same model training performed on the node.

[0076] The central server calculates an average value of multiple third accuracy rates B' to obtain a second accuracy rate B.

[0077] The reward function determines a reward value r based on a first accuracy rate A and a second accuracy rate B, wherein the first accuracy rate is the accuracy rate of the trained model obtained by the central server and the multiple nodes based on federated learning; the second accuracy rate is the average value of third accuracy rates sent by the multiple nodes respectively, and the third accuracy rate is obtained in the same model training when the model to be trained in the node is trained using sample data at the node as the first gradient.

[0078] For example, the reward function is expressed as:

[0079]

[0080] The higher the result B / result A, the higher the reward value r. It is understandable that in any gradient update process, the value of A is the same, and the value of B may be the same or different. g is greater than or equal to 1. Optionally, this application sets two reward functions, namely: When g is greater than 1, it takes a constant greater than 1, which can be strongly guided to complete gradient training faster. When less than or equal to 1, g is set to 1.

[0081] The Q value determined by the Critic network in this gradient update process (i.e., the updated Q value) is determined based on the Q value gradient and the Q value determined in the last gradient update process (i.e., the Q value before the update), for example, the updated Q value = the Q value before the update + the Q value gradient. It can be understood that if this gradient update is the first gradient update, the Q value outputted last time is the initially set Q value.

[0082] In one example, the Q value gradient is determined based on a first algorithm and a second algorithm, wherein the first algorithm has a characteristic of being biased toward a specific action during training, and the second algorithm has a characteristic of balancing the selection of various tasks during training. For example, the first algorithm may be a Deep Deterministic Policy Gradient (DDPG) algorithm, and the second algorithm may be a SAC algorithm, which may be a SAC (Soft actorcritic with automatic entropy adjustment, SAC-AEA) reinforcement learning algorithm with automatic entropy adjustment function. DDPG is subject to the update strategy of the algorithm itself, and will tend to be biased toward a specific action in the later stage of training, which is not conducive to the probability of scheduling multiple nodes, and is not conducive to the realization of overall model fusion. It will cause the final trained model to be highly correlated with the data of a certain node, and the model data of other nodes will contribute less to the model results, that is, the utilization efficiency of multi-party data is greatly reduced, and may even lead to poor model training results or overfitting problems. The SAC-AEA reinforcement learning algorithm itself can select each action in a relatively balanced manner. However, for the actual federated learning framework, the data quality, contribution to the model, local computing power (such as the computing efficiency of the local device) of each node are different (they can be described as dominant and non-dominant parties in this application). Equilibrium fusion of their data is obviously not conducive to the improvement of model training results, or underfitting occurs, and the complete data characteristics cannot be fully characterized. The Q value determined by the Critic network described in this application is updated based on the DDPG algorithm and the SAC algorithm. Fusion of the DDPG algorithm and the SAC algorithm can combine the advantages of the two algorithms so that the trained model integrates the performance of multiple nodes, and the model is better. This application can set a first weight for the DDPG algorithm and a second weight for the SAC algorithm, and determine the Q value gradient based on the DDPG algorithm and the first weight, the SAC algorithm and the second weight. The first weight and the second weight are determined based on the variance of the accuracy of the model before the gradient update in the multiple nodes.

[0083] In one example, in the Critic network, a Q value update algorithm based on composite adaptive adjustable weights is proposed; the Q value gradient is determined based on the first parameter J, for example, the Q value gradient is the product of the first parameter J and the step length, and the step length is not 0. Taking three Critic networks as an example, the first parameter J satisfies the following formula:

[0084]

[0085] or,

[0086]

[0087] in,

[0088] This example uses the example of an Actor-Critic network that can output the probabilities of multiple nodes in each cycle. t is the number of gradient updates, for example, the tth gradient update process, and t is an integer greater than or equal to 1.

[0089] k>0, l>0, k+l=1; k and l can be fixed values, can be artificially set values, or can be determined based on the variance of the accuracy of the model before the gradient update in the multiple nodes; the variance of the accuracy can represent the performance (such as computing power) difference of the nodes to a certain extent. The larger the variance, the greater the performance difference of the nodes, and vice versa, the smaller the variance, the smaller the performance difference of the multiple nodes.

[0090] θ 1 ,θ 2 ,θ 3 Represents three Critic networks, θ i is θ 1 ,θ 2 ,θ 3 They represent the network corresponding to the minimum value of the Q value determined by the three Critic networks. It can be understood that in this gradient update process, the Q value determined by the three Critic networks refers to the Q value determined in the last gradient update (if this gradient update is the tth time, the last time is the t-1th time);

[0091] s t is the state in the tth gradient update process; s t It can be the accuracy of each node or the average accuracy, the gradient of each node or the average gradient, the variance of each node, etc.

[0092] a t is the probability of the multiple nodes in the t-th gradient update process; a t It can also be called action;

[0093] is θ in the tth gradient update process i The corresponding Critic network is in s t ,a t The Q value determined in the case; is θ in the tth gradient update process 3 The corresponding Critic network is in s t ,a t The Q value of the output in this case;

[0094] r(s t ,a t ) is the gradient update process in the tth time at s t ,a t The reward value in the case;

[0095] γ is the attenuation factor, γ is greater than 0;

[0096] π t (a t |s t ) is the conditional probability, π t (a t |s t ) is in s t Make a t probability;

[0097] q is the exponent of entropy (such as Tasslis entropy), q is an integer greater than or equal to 1, which can be 1, 2 or 3, ln q is entropy, which is a curve. When q is different, ln q is a family of curves;

[0098] E is the numerical expectation. E calculates the expectation of the data in [*]. The independent variable is s t and a t , the content in [] in the above formula is s t and a t Implicit expression of

[0099] D is the memory bank (also called the experience replay pool or cache space), (s, a) ~ D refers to the s and a in the memory bank; assuming that the memory bank D can store M s and M a, and the memory bank D is a loop cover, then the Actor-Critic network can first self-loop M times to obtain M s and M a, and the Actor-Critic network will output the probability of the node in the M+1th loop. The probability that the Actor-Critic network does not output the node in the previous M loops, or the probability of the node output by the Actor-Critic network in the previous M loops can be ignored.

[0100] a~π φ Represents π φ Determined based on a, for example, multiple a's make up π φ Curve (or set).

[0101] α t is the α used in the t-th gradient update process. α can be a fixed value other than 0, or a variable (i.e., an adaptive parameter). The (updated) α used in this gradient update is determined based on the α gradient and the (pre-updated) α used in the last gradient update. For example, the (updated) α used in this gradient update = the α gradient and the (pre-updated) α used in the last gradient update. It is understandable that if this gradient update is the first gradient update, the α used last time is the initial setting α. The α gradient satisfies the following formula:

[0102]

[0103] Among them, J(α) is the gradient of α, α t-1 is the α used in the last gradient update, H is the ideal minimum expected entropy, and the entropy is, for example, Tasslis entropy.

[0104] Through the Q-value updating method of composite adaptive adjustable weights, the gradient weight parameters can be adaptively adjusted. According to the actual scenario, only the weight parameters need to be adjusted to select specific actions or balanced actions, so that the model information of each participant can be scheduled according to the specific situation.

[0105] The probability of the node output by the Actor network this time (i.e., the probability of the node after the update) is the sum of the probability of the node output last time (i.e., the probability of the node before the update) and the probability gradient.

[0106] Incorporating the Tasslis entropy concept and adaptive parameters into the Actor network, the probability gradient is determined based on the following formula:

[0107]

[0108] Among them, J(π φ ) is the probability gradient, α t is the α obtained in this (tth) gradient update process; θ i is θ 1 ,θ 2 ,θ 3 They respectively represent the network corresponding to the minimum value of the Q value most recently determined by the three Critic networks. It can be understood that in this gradient update process, the Q value most recently determined by the three Critic networks refers to the Q value determined in this gradient update process (for example, the tth gradient update).

[0109] On the server side of the federated framework, the data fusion algorithms of different nodes are adjusted, and an optimized fusion strategy combining the deep reinforcement learning data fusion model with the federated average algorithm is designed. The participation of different nodes and the degree of data utilization during training can be adjusted.

[0110] The above describes the method of the embodiment of the present application, and the following describes the device in the embodiment of the present application. The method and the device are based on the same technical concept. Since the principles of the method and the device to solve the problem are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0111] The embodiment of the present application can divide the functional modules of the device according to the above method example. For example, each function can be divided into each functional module, or two or more functions can be integrated into one module. These modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in the specific implementation.

[0112] Based on the same technical concept as the above method, see Figure 3 , provides a model gradient updating device, comprising:

[0113] A receiving module 301 is used to receive first gradients respectively sent by multiple nodes, where the first gradients are obtained by training a model to be trained in each node once or multiple times using sample data;

[0114] A processing module 302 is used to obtain a second gradient based on the multiple first gradients and the probability of each node in the current gradient update process, where the probability of each node in the current gradient update process is determined by the Actor-Critic network based on the probability of each node in the previous gradient update process;

[0115] The sending module 303 is used to send the second gradient to the multiple nodes respectively, so that the multiple nodes use the second gradient to update the weights of their respective models to be trained.

[0116] The above process is a gradient update process, and the gradient update process is repeated until the stop condition is met.

[0117] Based on the same technical concept as the above method, see Figure 4 , provides a model gradient updating device, including a processor 401 and a memory 402, and optionally, also including a transceiver 403;

[0118] The memory 402 is used to store computer programs or instructions;

[0119] The processor 401 is used to execute part or all of the computer programs or instructions in the memory, and when the part or all of the computer programs or instructions are executed, it is used to implement any of the above-mentioned model gradient updating methods. For example, the transceiver 403 performs receiving and sending actions, and the processor 401 performs other actions besides receiving and sending actions.

[0120] An embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program includes instructions for implementing any one of the model gradient updating methods described.

[0121] An embodiment of the present application also provides a computer program product, including: a computer program code, when the computer program code is executed on a computer, the computer can execute the model gradient updating method provided above.

[0122] An embodiment of the present application also provides a communication system, which includes: a node and a central server that execute the above-mentioned model gradient updating method.

[0123] In addition, the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), a baseband processor, the baseband processor and the CPU may be integrated together or separated, or may be a network processor (NP) or a combination of a CPU and a NP. The processor may further include a hardware chip or other general-purpose processor. The above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) and other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. or any combination thereof. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0124] The memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that the memory described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0125] The transceiver mentioned in the embodiments of the present application may include a separate transmitter and / or a separate receiver, or the transmitter and the receiver may be integrated into one. The transceiver may operate under the instruction of the corresponding processor. Optionally, the transmitter may correspond to a transmitter in a physical device, and the receiver may correspond to a receiver in a physical device.

[0126] Those of ordinary skill in the art will appreciate that the various method steps and units described in the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0128] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0131] The term "and / or" in this application describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. The multiple involved in this application refers to two or more. In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0132] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0133] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A model gradient updating method, characterized in that: Applied to the central server, including: The central server repeatedly executes the gradient update process until the stop condition is met; wherein the gradient update process once includes: Receive first gradients respectively sent by multiple nodes, where the first gradients are obtained by training the model to be trained in the node once or multiple times using sample data for each node; obtain a second gradient based on the multiple first gradients and the probability of each node in this gradient update process, where the probability of each node in this gradient update process is determined by the Actor-Critic network based on the probability of each node in the previous gradient update process; send the second gradients to the multiple nodes respectively, so that the multiple nodes use the second gradients to update the weights of their respective models to be trained; The Actor-Critic network includes a reward function and at least one Critic network; The reward function is used to determine a reward value and transmit the reward value to the at least one Critic network; The reward function satisfies: Among them, A is the first accuracy rate, B is the second accuracy rate, and g is greater than or equal to 1, wherein the first accuracy rate is the accuracy rate of the trained model obtained by the central server and the multiple nodes based on the federated average learning algorithm; the second accuracy rate is the average value of the third accuracy rates sent by the multiple nodes respectively, and the third accuracy rate is obtained in the same model training when the model to be trained in the node is trained by the sample data at the node using the first gradient.

2. The method according to claim 1, characterized in that The Actor-Critic network includes an Actor network; The at least one Critic network is used to determine a target Q value and transmit the target Q value to the Actor network; The Actor network is used to determine the probability of each node in this gradient update process based on the target Q value.

3. The method according to claim 2, characterized in that The target Q value is the minimum Q value among the Q values ​​determined by multiple Critic networks.

4. The method according to claim 1, characterized in that when When is greater than 1, g is greater than 1; when When it is less than or equal to 1, g is 1.

5. The method according to claim 2, characterized in that The Actor-Critic network includes three Critic networks. For any Critic network, the Q value determined in this gradient update process is based on the Q value gradient and the Q value determined in the previous gradient update process. The Q value gradient is determined based on a first parameter, and the first parameter satisfies the following formula: in, Where J is the first parameter; t is the number of times the gradient is updated; k>0, l>0, k+l=1; θ1, θ2, θ3 represent the three Critic networks respectively, θ i θ1, θ2, and θ3 represent the network corresponding to the minimum value of the Q value recently determined by the three Critic networks; s t is the state in the tth gradient update process; a t is the probability of the multiple nodes in the t-th gradient update process; is θ in the tth gradient update process i The corresponding Critic network is in s t ,a t The Q value determined in the case; is the Critic network corresponding to θ3 in the t-th gradient update process at s t ,a t The output Q value in the case of t ,a t ) is the gradient update process in the tth time at s t ,a t The reward value in the case; γ is greater than 0; π t (a t |s t ) is in s t Make a t The probability of; q is the exponent of entropy, ln q is entropy, α t is the α used in this gradient update, α t Not 0.

6. The method according to claim 5, characterized in that The α used in this gradient update process is determined based on the α gradient and the α used in the previous gradient update process, and the α gradient satisfies the following formula: Among them, J(α) is the gradient of α, α t-1 is the α used in the last gradient update, and H is the ideal minimum expected entropy.

7. The method according to claim 5, characterized in that The k and l are determined based on the variance of the accuracy of the model before the gradient update is performed in the multiple nodes.

8. The method according to claim 2, characterized in that The probability of the node output by the Actor network during this gradient update is determined based on the probability gradient and the probability of the node output during the previous gradient update. The probability gradient satisfies the following formula: Among them, J(π φ ) is the probability gradient, t is the number of times the gradient is updated; θ1, θ2, θ3 represent the three Critic networks, θ i θ1, θ2, and θ3 represent the network corresponding to the minimum value of the Q value recently determined by the three Critic networks; s t is the state in the tth gradient update process; a t is the probability of the multiple nodes in the t-th gradient update process; is θ in the tth gradient update process i The corresponding Critic network is in s t ,a t Q value determined under the circumstances; π t (a t |s t ) is in s t Make a t The probability of; q is the exponent of entropy, ln q is entropy, α t is the α used in this gradient update, α t Not 0.

9. The method according to any one of claims 1 to 8, characterized in that The central server and the multiple nodes perform gradient update based on a federated learning architecture.

10. A central server, characterized in that: include: Repeat the gradient updating process until the stopping condition is met; wherein the gradient updating process once includes: A receiving module, configured to receive first gradients respectively sent by a plurality of nodes, wherein the first gradients are obtained by training a model to be trained in each node once or multiple times using sample data; A processing module, configured to obtain a second gradient based on the plurality of first gradients and a probability of each node in the current gradient update process, wherein the probability of each node in the current gradient update process is determined by the Actor-Critic network based on the probability of each node in the previous gradient update process; A sending module, used for sending the second gradient to the multiple nodes respectively, so that the multiple nodes use the second gradient to update the weights of their respective models to be trained; The Actor-Critic network includes a reward function and at least one Critic network; The reward function is used to determine a reward value and transmit the reward value to the at least one Critic network; The reward function satisfies: Among them, A is the first accuracy rate, B is the second accuracy rate, and g is greater than or equal to 1, wherein the first accuracy rate is the accuracy rate of the trained model obtained by the central server and the multiple nodes based on the federated average learning algorithm; the second accuracy rate is the average value of the third accuracy rates sent by the multiple nodes respectively, and the third accuracy rate is obtained in the same model training when the model to be trained in the node is trained by the sample data at the node using the first gradient.

11. A model gradient updating device, characterized in that: including a processor and a memory; The memory is used to store computer programs or instructions; The processor is used to execute part or all of the computer programs or instructions in the memory, and when the part or all of the computer programs or instructions are executed, it is used to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program includes instructions for implementing the method according to any one of claims 1 to 9.

Images