A personalized federated learning acceleration method and device

Abstract

The present invention is applicable to the technical field of machine learning of artificial intelligence, and provides a personalized federated learning acceleration method and device; the method uses the information of the service node server model in federated learning to train a meta-automatic pruning network, and then sends the meta-automatic pruning network to the child node, retrains it with the data of the child node, generates personalized network weights, and iteratively optimizes the coding vector composed of the number of channels in each layer of the network model to accelerate the improvement of the accuracy of the personalized network weights; finally, the network after compression and acceleration at the child node meets the constraints in various personalized scenarios; the device using this method also has the same technical effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning in artificial intelligence, and in particular, relates to a personalized federated learning acceleration method and device. Background Art

[0002] In 2017, Google used federated learning to solve the problem of updating models locally on mobile devices. After that, federated learning was extended to fields such as medicine and finance. Since the data in these scenarios are more sensitive and private, federated learning proposes to use multiple computing nodes to update the model while ensuring security and privacy during data exchange. The specific approach is to establish a virtual initial model on a public node, and other nodes update parameters to the node without violating privacy regulations. The final result is that the virtual model aggregates all the data together to form an optimal model.

[0003] However, existing optimization methods for federated learning are limited to exploring the communication efficiency between service nodes and sub-nodes, resulting in low computational efficiency in the training and execution of federated learning models. Summary of the invention

[0004] The purpose of the present invention is to provide a personalized federated learning acceleration method and device, aiming to solve the problem of low computational efficiency in the training and execution of federated learning models by tailoring the model.

[0005] In one aspect, the present invention provides a personalized federated learning acceleration method, the method comprising the following steps:

[0006] S1. Use the service node to unify the initial models updated by each child node into an aggregate model, and train a meta-automatic pruning network consisting of two fully connected layers through the encoding vector of the aggregate model;

[0007] S2. Send the meta-automatic pruning network to each of the child nodes, input the encoding vector of the aggregation model and the personalized small sample data of the child node for matrix pruning;

[0008] S3. construct an acceleration network model using the generated weight matrix;

[0009] S4. Under hard constraints, evaluate and rank the accuracy of the acceleration network model corresponding to the randomly selected encoding vector;

[0010] S5. Select the encoding vector corresponding to the acceleration network model with an accuracy higher than the threshold value for mutation and / or cross-breeding to generate a new encoding vector;

[0011] S6. After eliminating the new encoding vectors with unqualified accuracy, determine whether the accuracy achieved by the remaining new encoding vectors meets the requirements. If not, repeat steps S4 to S6;

[0012] S7. If the new encoding vector meets the requirements, construct a new acceleration network model using the new encoding vector.

[0013] On the other hand, the present invention also provides a personalized federated learning acceleration device, the device comprising:

[0014] The meta-automatic pruning network unit is used to train a meta-automatic pruning network consisting of two fully connected layers using the encoding vector of the service node aggregation model; the meta-automatic pruning network is sent down to the child node, the encoding vector of the aggregation model and the personalized small sample data of the child node are input for matrix pruning, and the generated weight matrix is ​​used to construct an acceleration network model;

[0015] The acceleration network unit is used to evaluate and sort the accuracy of the acceleration network model corresponding to the randomly selected coding vector under hard constraints; select the coding vector corresponding to the acceleration network model with an accuracy higher than a threshold value for mutation and / or cross-breeding to generate a new coding vector; then eliminate the new coding vectors with unqualified accuracy and determine whether the accuracy achieved by the remaining new coding vectors meets the requirements. If not, repeat the above steps; if the new coding vector meets the requirements, construct a new acceleration network model with the new coding vector.

[0016] The method and device of the present invention can accelerate the construction of a high-accuracy model by automatically pruning the model and iteratively optimizing the encoding vector after the meta-automatic pruning network training is completed, thereby accelerating the neural network model, achieving the improvement of the overall efficiency of personalized federated learning acceleration and improving the execution efficiency of sub-nodes in personalized scenario task algorithm models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart for implementing the personalized federated learning acceleration method provided in the first embodiment of the present invention;

[0018] Figure 2 It is a functional structure block diagram of a personalized federated learning acceleration device provided in Embodiment 2 of the present invention;

[0019] Figure 3 It is a flowchart of a personalized federated learning acceleration method provided in Embodiment 1 of the present invention;

[0020] Figure 4 It is a schematic diagram of the overall process of the personalized federated learning acceleration method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following Figure 1-3 It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

[0023] Embodiment 1:

[0024] Figure 1 , 3 4 shows the implementation process of the personalized federated learning acceleration method provided in the first embodiment of the present invention. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0025] A personalized federated learning acceleration method, the method comprising the following steps:

[0026] S1. Use the service node to unify the initial models updated by each sub-node according to its own local database into an aggregate model; train a meta-automatic pruning network consisting of two fully connected layers through the encoding vector of the aggregate model;

[0027] S2. Send the meta-automatic pruning network to each of the child nodes, input the encoding vector of the aggregation model and the personalized small sample data of the child node (i.e., the attached Figure 4 Personalized data in the matrix) for matrix trimming;

[0028] S3. Construct an acceleration network model with the generated weight matrix (i.e., attached Figure 4 The acceleration model in );

[0029] In the forward pass, the meta-automatic pruning network uses the encoding vector of the service node aggregation model (i.e., the number of channels per layer of the network model) as input, and also adds the personalized data of the child node (attached Figure 3 Not shown, see the attached Figure 4 ), and the output is the weight matrix of the generated accelerated personalized network model (accelerated network model) of the child node. The accelerated network model structure of the child node is composed of the output channel width of each layer. The width of the output channel is equal to the element value in the encoding vector of the service node aggregation network model.

[0030] The weight matrix generated by the meta-automatic pruning network is pruned to match the number of input and output channels in the child node acceleration network model. Then, given a small batch of input images, we can use the generated weights to calculate the loss function of the accelerated personalized model.

[0031] In the reverse process, the present invention calculates the gradient and weight in the meta-automatic pruning network instead of directly updating the weight in the meta-automatic pruning network in the prior art. Because the reshaping and convolution operations between the fully connected output layer of the meta-automatic pruning network and the output of the front convolution layer of the acceleration network are differentiable, the gradient of the weight of the meta-automatic pruning network can be easily calculated using the chain rule.

[0032] Simply put, after the training of the meta-automatic pruning network is completed, the encoding vector and personalized data of the aggregated network are input into the meta-automatic pruning network to generate the weight matrix of the corresponding acceleration network, and the verification data is evaluated to obtain the pruning accuracy of each potential acceleration network.

[0033] Since the number of network coding vectors is large, we cannot use enumeration to compare and search. Therefore, we use an evolutionary algorithm, which can easily find the weights required for the final acceleration network under the soft and hard constraints of personalized scene data.

[0034] S4. Under hard constraints, evaluate and rank the accuracy of the acceleration network model corresponding to the randomly selected encoding vector;

[0035] S5. Select the encoding vector corresponding to the acceleration network model with an accuracy higher than the threshold, perform mutation and / or cross-breeding, and generate a new encoding vector;

[0036] S6. After eliminating the new coding vectors with unqualified accuracy, determine whether the accuracy of the remaining new coding vectors meets the requirements. If not, repeat steps S4 to S6;

[0037] S7. If the new encoding vector meets the requirements, a new acceleration network model is constructed using the new encoding vector.

[0038] Among them, the accuracy of the accelerated network model corresponding to the coding vector needs to be specifically set according to the actual situation. The new coding vector that meets the requirements must ensure that the accuracy achieved by the corresponding accelerated network model is higher than the qualified line (qualified value) of the accuracy, so as to achieve the improvement of the operation accuracy on the basis of accelerated network operation.

[0039] Preferably, in step S3, the number of channels of a layer is randomly selected in each iteration by random structure sampling to generate a coding vector; different acceleration networks are constructed according to different coding vectors, and the output of the meta-automatic pruning network is used to give its corresponding weight matrix.

[0040] Preferably, in step S1, the meta-automatic pruning network is defined as:

[0041]

[0042] Among the parameters of the expression, X is the network model before acceleration; c is the channel width of the network model; c1, c2, ……, cj are the encoding vectors composed of the channel widths in the neural network model; j is a letter code representing the quantity, indicating from 1 to J; L is the abbreviation of Loss function, which means the loss function and is an objective function used to train the network in the meta-auto pruning network; the smaller the loss function calculated by the trained model, the closer it is to the model required in the environment; w is the weight, that is, the weighting of each neuron function in the meta-auto pruning network. It represents all the channel widths that ensure the minimum accuracy loss after the personalized retraining of the child node; D is the weighting, and s.t. D<constraint means that the weighting D of each neuron function in the neural network satisfies a certain computational amount and delay constraint.

[0043] Preferably, in step S5, the mutation is: randomly changing the digital ratio of each vector element in the encoding vector.

[0044] Preferably, in step S5, the crossover is randomly recombining two or more encoding vectors to generate a child encoding vector.

[0045] Preferably, the accelerated network model structure of the child node is composed of the encoding vectors output by each fully connected layer in the meta-auto pruning network, and the encoding vector is equal to the element value in the encoding vector of the aggregation model of the service node.

[0046] As described above, in order to train the meta-auto pruning network, the present invention also proposes random structure sampling. In the model training stage (step S2), the number of channels in each layer is randomly selected in each iteration to generate a weight matrix. Different accelerated networks are constructed according to different weight matrices, and the corresponding weights are given by the output of the meta-auto pruning network. By randomly training different encoding vectors, the meta-auto pruning network learns to predict reasonable weight matrices for the accelerated networks in different environments.

[0047] Embodiment 2:

[0048] Figure 2 The functional schematic diagram of the personalized federated learning acceleration device provided in Embodiment 1 of the present invention is shown. For the convenience of description, only the parts related to Embodiment 1 of the present invention are shown and are described in detail as follows:

[0049] A personalized federated learning acceleration device, the device includes:

[0050] The meta-automatic pruning network unit is used to train a meta-automatic pruning network consisting of two fully connected layers using the encoding vector of the service node aggregation model; the meta-automatic pruning network is sent down to the child node, the encoding vector of the aggregation model and the personalized small sample data of the child node are input for matrix pruning, and the generated weight matrix is ​​used to construct an acceleration network model;

[0051] The acceleration network unit is used to evaluate and sort the accuracy of the acceleration network model corresponding to the randomly selected coding vector under hard constraints; select the coding vector corresponding to the acceleration network model with an accuracy higher than a threshold for mutation and / or cross-breeding to generate a new coding vector; then eliminate the new coding vectors with unqualified accuracy and determine whether the accuracy achieved by the remaining new coding vectors meets the requirements. If not, repeat the above steps; if the new coding vector meets the requirements, construct a new acceleration network model with the new coding vector.

[0052] Preferably, the meta-automatic pruning network unit comprises:

[0053] The iteration module is used to randomly select the number of channels of a layer in each iteration through random structure sampling to generate a coding vector; construct different acceleration networks according to different coding vectors, and use the output of the meta-automatic pruning network to give its corresponding weight matrix.

[0054] Preferably, the acceleration network unit includes:

[0055] The mutation module is used to randomly change the numerical ratio of each vector element in the encoding vector.

[0056] Preferably, the acceleration network unit further includes:

[0057] The crossover module is used to randomly recombine two or more encoding vectors to generate a child encoding vector.

[0058] After the meta-automatic pruning network training is completed, the acceleration network unit compresses and accelerates the search for model weights with higher accuracy, thereby accelerating the model execution while maintaining the accuracy of operation in personalized scenarios.

[0059] The method and device of the present invention can accelerate the construction of a high-accuracy model by automatically pruning the model and iteratively optimizing the encoding vector after the meta-automatic pruning network training is completed, thereby accelerating the neural network model, achieving the improvement of the overall efficiency of personalized federated learning acceleration and improving the execution efficiency of sub-nodes in personalized scenario task algorithm models.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A personalized federated learning acceleration method, characterized in that: The method comprises the following steps: S1. Use the service node to unify the initial models updated by each child node into an aggregate model, and train a meta-automatic pruning network consisting of two fully connected layers through the encoding vector of the aggregate model; S2. Send the meta-automatic pruning network to each of the child nodes, input the encoding vector of the aggregation model and the personalized small sample data of the child node for matrix pruning; S3. construct an acceleration network model using the generated weight matrix; S4. Under hard constraints, evaluate and rank the accuracy of the acceleration network model corresponding to the randomly selected encoding vector; S5. Select the encoding vector corresponding to the acceleration network model with an accuracy higher than the threshold value for mutation and / or cross-breeding to generate a new encoding vector; S6. After eliminating the new encoding vectors with unqualified accuracy, determine whether the accuracy achieved by the remaining new encoding vectors meets the requirements. If not, repeat steps S4 to S6; S7. If the new encoding vector meets the requirements, construct a new acceleration network model with the new encoding vector; In step S1, the meta-automatic pruning network is defined as: ,s.t. D < constraint; in, X is the network model before acceleration; c is the channel width of the network model; c1, c2, ..., cj are the encoding vectors composed of the channel widths in the neural network model; j is a letter code representing the number, ranging from 1 to J; L is the abbreviation of Lossfunction, which means loss function, which is an objective function used to train the network in the meta-automatic pruning network; the smaller the loss function calculated by the trained model, the closer it is to the model required in the environment; w is the weight, that is, the weighting of each neuron function in the meta-automatic pruning network; It indicates the width of all channels that ensure the minimum accuracy loss after personalized retraining of the child nodes; D is the weight, and st D < constraint means that the weight D of each neuron function in the neural network satisfies certain computational and delay constraints.

2. The method according to claim 1, characterized in that In step S3, the number of channels of a layer is randomly selected in each iteration through random structure sampling to generate a coding vector; different acceleration networks are constructed according to different coding vectors, and the output of the meta-automatic pruning network is used to give its corresponding weight matrix.

3. The method according to claim 1, characterized in that In step S5, the mutation is: randomly changing the digital ratio of each vector element in the encoding vector.

4. The method according to claim 1, characterized in that In step S5, the crossover is to randomly recombine two encoding vectors to generate a child encoding vector.

5. The method according to claim 1, characterized in that The acceleration network model structure of the subnode is composed of the encoding vector output by each fully connected layer in the meta-automatic pruning network, and the encoding vector is equal to the element value in the encoding vector of the aggregation model of the service node.

6. A personalized federated learning acceleration device, characterized in that: The device comprises: The meta-automatic pruning network unit is used to train a meta-automatic pruning network consisting of two fully connected layers using the encoding vector of the service node aggregation model; the meta-automatic pruning network is sent down to the child node, the encoding vector of the aggregation model and the personalized small sample data of the child node are input for matrix pruning, and the generated weight matrix is ​​used to construct an acceleration network model; The acceleration network unit is used to evaluate and sort the accuracy of the acceleration network model corresponding to the randomly selected coding vector under hard constraints; select the coding vector corresponding to the acceleration network model with an accuracy higher than a threshold value for mutation and / or cross-breeding to generate a new coding vector; then eliminate the new coding vectors with unqualified accuracy and determine whether the accuracy achieved by the remaining new coding vectors meets the requirements. If not, repeat the above steps; if the new coding vector meets the requirements, construct a new acceleration network model with the new coding vector; The meta-automatic pruning network is defined as: ,s.t. D < constraint; in, X is the network model before acceleration; c is the channel width of the network model; c1, c2, ..., cj are the encoding vectors composed of the channel widths in the neural network model; j is a letter code representing the number, ranging from 1 to J; L is the abbreviation of Lossfunction, which means loss function, which is an objective function used to train the network in the meta-automatic pruning network; the smaller the loss function calculated by the trained model, the closer it is to the model required in the environment; w is the weight, that is, the weighting of each neuron function in the meta-automatic pruning network; It indicates the width of all channels that ensure the minimum accuracy loss after personalized retraining of the child nodes; D is the weight, and st D < constraint means that the weight D of each neuron function in the neural network satisfies certain computational and delay constraints.

7. The device according to claim 6, characterized in that The meta-automatic pruning network unit comprises: The iteration module is used to randomly select the number of channels of a layer in each iteration through random structure sampling to generate a coding vector; construct different acceleration networks according to different coding vectors, and use the output of the meta-automatic pruning network to give its corresponding weight matrix.

8. The device according to claim 6, characterized in that The acceleration network unit comprises: The mutation module is used to randomly change the numerical ratio of each vector element in the encoding vector.

9. The device according to claim 6, characterized in that The acceleration network unit also includes: The crossover module is used to randomly recombine two encoding vectors to produce a child encoding vector.

Images