Model pruning method and system for federated learning

Abstract

The application provides a model pruning method and system for federated learning, belongs to the technical field of federated learning, calculates the importance index of convolution kernels in each convolution layer of a neural network model, prunes the convolution kernels with an importance index lower than a threshold value in terms of the number of convolution kernels, calculates the L1 norm of each vector inside the remaining convolution kernels, prunes the convolution kernels based on a preset threshold value, completes convolution kernel size pruning, and thus reduces the dimension of the convolution kernel size. The application realizes sufficient pruning of redundant parameters, greatly improves the model compression efficiency, and the proposed dynamic threshold model pruning framework dynamically adjusts the pruning range in each training round, makes the setting of the pruning rate more reasonable, and dynamically prunes the sensitive layers and non-sensitive layers of the model at different compression rates, so as to better control the pruning strength of each part of the model in the pruning process, improve the performance of the model, and be beneficial to the recovery of the accuracy of the model after pruning.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and specifically to a model pruning method and system for federated learning. Background Technology

[0002] Federated learning is an emerging distributed machine learning framework that allows training data to be distributed across mobile devices and learned from shared models by aggregating updates from locally computed datasets. Federated learning effectively addresses the issues of data silos and privacy protection, enabling joint training and the creation of shared machine learning models while ensuring that data does not leave local storage.

[0003] As can be seen from the workflow of federated learning, the central server and various clients need to constantly exchange a large number of model parameters through uplink and downlink communication links, resulting in high communication overhead. Therefore, how to reduce the communication overhead and improve communication efficiency in the federated learning process has become a top priority in the practical application of federated learning. The most common solution is model pruning, which explores the redundancy in the weights of the neural network model and prunes non-critical weights without significantly reducing model performance, thereby reducing the model size and accelerating model training. However, current mainstream compression methods, such as model pruning, fixed sparsity pruning, and some single-index channel pruning methods, generally suffer from drawbacks such as excessively long pruning time, insufficient removal of redundant parameters in the model, and difficulty in controlling the pruning intensity. Summary of the Invention

[0004] The purpose of this invention is to provide a model pruning method and system for federated learning, so as to solve at least one of the technical problems existing in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] On the one hand, this invention provides a model pruning method for federated learning, comprising:

[0007] For the neural network model to be pruned on the client side, calculate the importance index of the convolutional kernel in each convolutional layer;

[0008] Using a dynamic threshold model pruning method, the number of convolution kernels with importance indices below a threshold is pruned.

[0009] For the convolution kernels retained after kernel number pruning, calculate the L1 norm of each vector inside these convolution kernels;

[0010] Based on the calculated L1 norm, the convolution kernel is pruned according to a preset threshold to complete the kernel size pruning, thereby reducing the dimensionality of the convolution kernel.

[0011] Preferably, the convolutional kernel importance index is determined according to the following calculation steps: calculate the information entropy of each convolutional kernel in each convolutional layer; calculate the mean and standard deviation of the information entropy of the convolutional kernel to obtain the convolutional kernel importance index.

[0012] Preferably, the information entropy of the convolution kernel is calculated as follows: the information entropy in the convolution kernel is calculated according to the following formula:

[0013]

[0014] Among them, W i,j ∈R M×K×K W is the weight of the j-th convolutional kernel in the i-th convolutional layer, M is the number of input channels in this layer (i.e., the number of convolutional kernels), and K×K is the size of the convolutional kernel; i,j The element values ​​in the dataset are evenly divided into N intervals, and then the frequency p corresponding to each interval is calculated. k This represents the ratio of the number of elements in the k-th interval to the total number of elements;

[0015] Assume that the probability distribution of the convolution kernel weights approximately follows a Gaussian normal distribution; when the probability distribution of the weights is known, the information entropy of the j-th convolution kernel in the i-th convolutional layer is transformed into the difference entropy of a continuous random variable; the discrete weight information of the convolution kernel is transformed into information entropy calculated using a continuous conceptual distribution.

[0016] Preferably, the mean and standard deviation of the information entropy of the convolutional kernel are calculated to obtain the importance index of the convolutional kernel. The mean and standard deviation of the information entropy of the convolutional kernel in each convolutional layer are calculated according to the following formula:

[0017]

[0018]

[0019] Where e is the current pruning cycle, E(W) i,j ) (e) It is the information entropy of the j-th convolutional kernel in the i-th layer during the e-th round of pruning. It is the mean of the information entropy of the i-th convolutional kernel. It is the standard deviation of the information entropy of the i-th convolutional kernel.

[0020] Preferably, in the dynamic threshold model pruning method, determining the threshold includes: calculating the mean of the information entropy. and standard deviation Then, a parameter λ related to the pruning cycle is introduced. (e) :λ (e) =tanh(eE / 2);

[0021] Where tanh(·) is the hyperbolic tangent function, whose domain is the entire real number field and whose range is (-1,1); E is the number of convolution kernels and the total number of pruning rounds; the function value is negative when the independent variable is less than 0 and positive when the independent variable is greater than 0;

[0022] The pruning threshold for each round is then:

[0023] Preferably, the L1 norm of the vectors inside the convolution kernel includes: the L1 norm of a vector x inside the convolution kernel is calculated according to the following formula:

[0024] Secondly, the present invention provides a model pruning system for federated learning, comprising:

[0025] The first calculation module is used to calculate the importance index of the convolution kernel in each convolutional layer for the neural network model to be pruned on the client side.

[0026] The first pruning module is used to prune the number of convolution kernels that are below the importance index using a dynamic threshold model pruning method.

[0027] The second calculation module is used to calculate the L1 norm of each vector inside the convolution kernels that are retained after the number of convolution kernels has been pruned.

[0028] The second pruning module is used to prune the convolution kernel based on the calculated L1 norm and a preset threshold, thereby reducing the dimensionality of the convolution kernel.

[0029] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the model pruning method for federated learning as described above.

[0030] Fourthly, the present invention provides a computer program product, including a computer program that, when run on one or more processors, implements the federated learning-oriented model pruning method as described above.

[0031] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the model pruning method for federated learning as described above.

[0032] The beneficial effects of this invention are as follows: Based on the dual screening of convolutional kernel importance index and L1 norm of convolutional kernel internal vectors, and the dynamic threshold pruning method, the model can be pruned from two aspects: the number of convolutional kernels and the size of convolutional kernels. This achieves full pruning of redundant parameters and greatly improves the model compression efficiency. The proposed dynamic threshold model pruning framework dynamically adjusts the pruning range in each training round, making the pruning rate setting more reasonable. Furthermore, it dynamically prunes the sensitive and insensitive layers of the model with different compression rates, achieving better control over the pruning intensity of each part of the model during the pruning process, improving model performance, and helping to restore the accuracy of the pruned model.

[0033] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the dual screening and dynamic threshold pruning method based on the importance index of the convolution kernel and the L1 norm of the vector inside the convolution kernel, as described in an embodiment of the present invention. Detailed Implementation

[0036] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0037] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0039] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0040] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0041] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0042] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0043] Example 1

[0044] In this embodiment 1, a model pruning system for federated learning is first provided, including:

[0045] The first calculation module is used to calculate the importance index of the convolution kernel in each convolutional layer for the neural network model to be pruned on the client side.

[0046] The first pruning module is used to prune the number of convolution kernels that are below the importance index using a dynamic threshold model pruning method.

[0047] The second calculation module is used to calculate the L1 norm of each vector inside the convolution kernels that are retained after the number of convolution kernels has been pruned.

[0048] The second pruning module is used to prune the convolution kernel based on the calculated L1 norm and a preset threshold, thereby reducing the dimensionality of the convolution kernel.

[0049] In this embodiment 1, the above-described system is used to implement a model pruning method for federated learning, including:

[0050] The first calculation module is used to calculate the kernel importance index in each convolutional layer for the neural network model to be pruned on the client side.

[0051] The first pruning module uses a dynamic threshold model pruning method to prune the number of convolution kernels whose importance index is below the threshold.

[0052] The second computation module is used to calculate the L1 norm of each vector inside the convolution kernels that are retained after the number of convolution kernels has been pruned.

[0053] The second pruning module uses the calculated L1 norm and a preset threshold to prune the convolution kernel, thereby reducing the dimensionality of the convolution kernel.

[0054] The importance index of the convolutional kernel is determined according to the following calculation steps: Calculate the information entropy of each convolutional kernel in each convolutional layer; calculate the mean and standard deviation of the information entropy of the convolutional kernels to obtain the importance index of the convolutional kernels. Calculating the information entropy of the convolutional kernels includes: the information entropy in the convolutional kernel is calculated according to the following formula:

[0055]

[0056] Among them, W i,j ∈R M×K×K W is the weight of the j-th convolutional kernel in the i-th convolutional layer, M is the number of input channels in this layer (i.e., the number of convolutional kernels), and K×K is the size of the convolutional kernel; i,j The element values ​​in the dataset are evenly divided into N intervals, and then the frequency p corresponding to each interval is calculated. k This represents the ratio of the number of elements in the k-th interval to the total number of elements;

[0057] Assume that the probability distribution of the convolution kernel weights approximately follows a Gaussian normal distribution; when the probability distribution of the weights is known, the information entropy of the j-th convolution kernel in the i-th convolutional layer is transformed into the difference entropy of a continuous random variable; the discrete weight information of the convolution kernel is transformed into information entropy calculated using a continuous conceptual distribution.

[0058] The mean and standard deviation of the information entropy of the convolutional kernel are calculated to obtain the importance index of the convolutional kernel. The mean and standard deviation of the information entropy of the convolutional kernel in each convolutional layer are calculated according to the following formula:

[0059]

[0060]

[0061] Where e is the current pruning cycle, E(W) i,j ) (e) It is the information entropy of the j-th convolutional kernel in the i-th layer during the e-th round of pruning. It is the mean of the information entropy of the i-th convolutional kernel. It is the standard deviation of the information entropy of the i-th convolutional kernel.

[0062] To determine the dynamic threshold in the dynamic threshold model pruning method, the following steps are taken: After calculating the mean of the information entropy... and standard deviation Then, a parameter λ related to the pruning cycle is introduced. (e) :λ (e) =tanh(eE / 2);

[0063] Where tanh(·) is the hyperbolic tangent function, whose domain is the entire real number field and whose range is (-1,1); E is the number of convolution kernels and the total number of pruning rounds; the function value is negative when the independent variable is less than 0 and positive when the independent variable is greater than 0;

[0064] The pruning threshold for each round is then:

[0065] The L1 norm of vectors inside the convolution kernel, including the L1 norm of a vector x inside the convolution kernel, is calculated using the following formula:

[0066] Example 2

[0067] In this embodiment 2, a model pruning method for federated learning is proposed, which can improve the performance of the pruned model, greatly improve the compression efficiency, and better control the fine granularity of pruning, thereby improving the communication efficiency of federated learning.

[0068] The model pruning method for federated learning in this embodiment 2 is a dual screening and dynamic threshold pruning method based on the importance index of convolutional kernels and the L1 norm of the internal vectors of convolutional kernels. It calculates the importance index of convolutional kernels in each convolutional layer, determines the threshold based on the obtained importance index and the current pruning round, and uses the dynamic threshold model pruning method to prune all convolutional kernels with importance indices below the threshold. Pre-training and pruning iterations are performed simultaneously to complete multiple rounds of dynamic threshold pruning. Then, the L1 norm of the internal vectors of convolutional kernels is calculated, and the convolutional kernels are pruned again with a preset fixed threshold.

[0069] Specifically, the method in this embodiment 2 includes the following steps:

[0070] Step S1: Each client uses its local dataset to train the local model and prunes the model at the same time;

[0071] Step S2: For the convolutional neural network to be pruned on the client side, calculate the importance index of the convolutional kernel in each convolutional layer;

[0072] The importance index of the convolution kernel is determined according to the following calculation steps:

[0073] Step S21: Calculate the information entropy of each convolutional kernel in each convolutional layer;

[0074] Calculate the information entropy of each convolutional kernel in each convolutional layer. The information entropy of a single convolutional kernel is calculated using the following formula:

[0075]

[0076] Among them, W i,j ∈R M×K×K W is the weight of the j-th convolutional kernel in the i-th convolutional layer, M is the number of input channels in this layer (i.e., the number of convolutional kernels), and K×K is the size of the convolutional kernel; i,j The element values ​​in the dataset are evenly divided into N intervals, and then the frequency p corresponding to each interval is calculated. k This represents the ratio of the number of elements in the k-th interval to the total number of elements;

[0077] When the number of weights in the convolution kernel is large, according to the central limit theorem, we can assume that the probability distribution of the convolution kernel weights approximately follows a Gaussian normal distribution:

[0078]

[0079] in, It is the mean of the Gaussian distribution that it approximately follows. It is the standard deviation. This indicates that the weights of the j-th convolutional kernel in the i-th convolutional layer follow a mean of 1. Standard deviation is Gaussian distribution;

[0080] When the probability distribution of the weights is known, the information entropy of the j-th convolutional kernel of the i-th convolutional layer is transformed into the difference entropy of a continuous random variable:

[0081]

[0082] The discrete weight information of the convolution kernel is transformed into information entropy calculated using a continuous conceptual distribution. The information entropy is only related to the standard deviation of the convolution kernel, so it is no longer necessary to traverse the information of each element of the convolution kernel.

[0083] Step S22: Calculate the mean and standard deviation of the convolution kernel information entropy to obtain the convolution kernel importance index;

[0084] The mean and standard deviation of the information entropy of the convolutional kernel in each convolutional layer are calculated using the following formula:

[0085]

[0086]

[0087] Where e is the current pruning cycle, E(W) i,j ) (e) It is the information entropy of the j-th convolutional kernel in the i-th layer during the e-th round of pruning. It is the mean of the information entropy of the i-th convolutional kernel. It is the standard deviation of the information entropy of the i-th convolutional kernel.

[0088] Step S3: Determine the threshold based on the importance index of the convolution kernel and the current pruning round. Use the dynamic threshold model pruning method to prune convolution kernels with an importance index below the threshold, thereby completing the convolution kernel number pruning operation and filtering the convolution kernels.

[0089] To determine the threshold, the mean of the information entropy is calculated in step S2. and standard deviation Finally, a parameter λ related to the pruning cycle needs to be introduced. (e) It is calculated according to the following formula:

[0090] λ (e ) = tanh(eE / 2);

[0091] Where tanh(·) is the hyperbolic tangent function, whose domain is the entire real number field and whose range is (-1,1); E is the number of convolution kernels and the total number of pruning rounds; the function value is negative when the independent variable is less than 0 and positive when the independent variable is greater than 0;

[0092] In summary, the pruning threshold for each round is:

[0093]

[0094] Determine the threshold Then pruning can be performed, reducing the information entropy to less than 1%. The convolutional kernels are discarded, thus filtering out the important convolutional kernels in the network model.

[0095] Step S4: Repeat steps S2 to S3, that is, perform multiple rounds of convolution kernel number pruning operation until the preset pruning round E is reached;

[0096] Step S5: Calculate the L1 norm of the vectors inside the convolution kernel;

[0097] After round E of pruning based on the information entropy of the convolution kernel is completed, another round of pruning based on the L1 norm of the vectors inside the convolution kernel is performed.

[0098] A K×K convolution kernel can be viewed as K K-dimensional row vectors or K K-dimensional column vectors; for the convolution kernels retained after step S4, the L1 norm of each vector inside these convolution kernels is calculated, and the selection is performed from the perspectives of rows and columns.

[0099] The L1 norm of a vector x within the convolution kernel is calculated using the following formula:

[0100]

[0101] Step S6: Prune the convolution kernel with a preset fixed threshold to complete the pruning of the convolution kernel size, thereby reducing the dimensionality of the convolution kernel size;

[0102] First, we filter from the perspective of rows. For each row vector, we calculate its L1 norm and then sort it by size.

[0103] After selecting L row vectors with larger L1 norms, the convolution kernel size becomes L×K, which can be regarded as K L-dimensional column vectors.

[0104] Calculate the L1 norm of K column vectors. Similarly, sort them and select L column vectors with larger L1 norms. At this time, the convolution kernel size becomes L×L, and the pruned client-side local training model is obtained.

[0105] Step S7: The updated models from each client are uploaded to the central server for aggregation. The server then returns the globally updated models to each client to complete one iteration of the federated learning process.

[0106] In this embodiment 2, the importance index of the convolutional kernel and the L1 norm of the convolutional kernel's internal vectors are used as the pruning evaluation metrics for the model pruning method for federated learning. The model pruning method for federated learning includes two pruning operations: E rounds of convolutional kernel number pruning based on the convolutional kernel information entropy and 1 round of convolutional kernel size pruning based on the L1 norm of the convolutional kernel's internal vectors; wherein, the convolutional kernel size pruning based on the L1 norm of the convolutional kernel's internal vectors begins after all the convolutional kernel number pruning operations based on the convolutional kernel information entropy have been completed.

[0107] The convolutional kernel importance metrics, including the mean and standard deviation of the information entropy of each convolutional kernel layer, reflect the amount of information the network model can extract from the convolutional kernels. Throughout the pruning process, the mean and standard deviation of the convolutional kernel information entropy continuously change, λ. (e) The pruning threshold changes from -1 to 1 as the pruning process progresses. It is negative when the number of pruning iterations is less than half the total number of iterations, and positive when it is greater than half. Convolutional kernels exceeding the mean are pruned. Therefore, the pruning intensity is relatively high in the early stages of training, resulting in larger model oscillations. As the iteration process continues, the pruning threshold gradually increases, the pruning intensity gradually decreases, and model oscillations decrease until convergence, thus avoiding over-pruning and under-pruning.

[0108] Example 3

[0109] In this embodiment 3, a model pruning method for federated learning is provided. By calculating the importance index of the convolutional kernels in each convolutional layer, a threshold is determined based on the obtained importance index and the current pruning round. Using a dynamic threshold model pruning method, all convolutional kernels with importance indices below the threshold are pruned. Pre-training and pruning iterations are performed simultaneously to complete multiple rounds of dynamic threshold pruning. Then, the L1 norm of the vectors inside the convolutional kernels is calculated, and the convolutional kernels are pruned again with a preset fixed threshold.

[0110] Specifically, the following steps are included:

[0111] Step S1: Each client uses its local dataset to train the local model and prunes the model at the same time;

[0112] The training environment and federated learning framework can be set up according to the implementer's needs. For illustration, the federated learning framework selected is Flower, which prunes the AlexNet and VGG16 models. The datasets are Cifar10 and ILSVRC2012. The Cifar10 dataset contains 10 categories, with a total of 50,000 training images and 10,000 validation images. The ILSVRC2012 dataset contains 1,000 categories, with a total of 1.2 million training images, 50,000 validation images, and 150,000 test images.

[0113] Step S2: For the convolutional neural network to be pruned on the client side, calculate the importance index of the convolutional kernel in each convolutional layer;

[0114] To illustrate, regarding the structure of pruned convolutional neural network models, taking AlexNet and VGG16 as examples, AlexNet has a total of 8 layers, of which the first 5 are convolutional layers and the last 3 are fully connected layers. VGG16 has a total of 16 layers, including 13 convolutional layers and 3 fully connected layers. Compared with other models, the significant features of these two models are their complex structure and large number of parameters. AlexNet has a total of 60 million learning parameters, and VGG16 has a total of 130 million learning parameters.

[0115] The importance index of the convolution kernel is determined according to the following calculation steps:

[0116] Step S21: Calculate the information entropy of each convolutional kernel in each convolutional layer;

[0117] Calculate the information entropy of each convolutional kernel in each convolutional layer. The information entropy of a single convolutional kernel is calculated using the following formula:

[0118]

[0119] Among them, W i,j ∈R M×K×KW is the weight of the j-th convolutional kernel in the i-th convolutional layer, M is the number of input channels in this layer (i.e., the number of convolutional kernels), and K×K is the size of the convolutional kernel; i,j The element values ​​in the dataset are evenly divided into N intervals, and then the frequency p corresponding to each interval is calculated. k This represents the ratio of the number of elements in the k-th interval to the total number of elements;

[0120] When the number of weights in the convolution kernel is large, according to the central limit theorem, we can assume that the probability distribution of the convolution kernel weights approximately follows a Gaussian normal distribution:

[0121]

[0122] in, It is the mean of the Gaussian distribution that it approximately follows. It is the standard deviation. This indicates that the weights of the j-th convolutional kernel in the i-th convolutional layer follow a mean of 1. Standard deviation is Gaussian distribution;

[0123] When the probability distribution of the weights is known, the information entropy of the j-th convolutional kernel of the i-th convolutional layer is transformed into the difference entropy of a continuous random variable:

[0124]

[0125] The discrete weight information of the convolution kernel is transformed into information entropy calculated using a continuous conceptual distribution. The information entropy is only related to the standard deviation of the convolution kernel, so it is no longer necessary to traverse the information of each element of the convolution kernel.

[0126] Step S22: Calculate the mean and standard deviation of the convolution kernel information entropy to obtain the convolution kernel importance index;

[0127] The mean and standard deviation of the information entropy of the convolutional kernel in each convolutional layer are calculated using the following formula:

[0128]

[0129]

[0130] Where e is the current pruning cycle, E(W) i,j ) (e) It is the information entropy of the j-th convolutional kernel in the i-th layer during the e-th round of pruning. It is the mean of the information entropy of the i-th convolutional kernel. It is the standard deviation of the information entropy of the i-th convolutional kernel.

[0131] Step S3: Determine the threshold based on the importance index of the convolution kernel and the current pruning round. Use the dynamic threshold model pruning method to prune convolution kernels with an importance index below the threshold, thereby completing the convolution kernel number pruning operation and filtering the convolution kernels.

[0132] To determine the threshold, the mean of the information entropy is calculated in step S2. and standard deviation Finally, a parameter λ related to the pruning cycle needs to be introduced. (e) It is calculated according to the following formula:

[0133] λ (e) =tanh(eE / 2);

[0134] Where tanh(·) is the hyperbolic tangent function, whose domain is the entire real number field and whose range is (-1,1); E is the number of convolution kernels and the total number of pruning rounds; the function value is negative when the independent variable is less than 0 and positive when the independent variable is greater than 0;

[0135] In summary, the pruning threshold for each round is:

[0136]

[0137] Determine the threshold Then pruning can be performed, reducing the information entropy to less than 1%. The convolutional kernels are discarded, thus filtering out the important convolutional kernels in the network model.

[0138] Step S4: Repeat steps S2 to S3, that is, perform multiple rounds of convolution kernel number pruning operation until the preset pruning round E is reached.

[0139] Step S5: Calculate the L1 norm of the vectors inside the convolution kernel;

[0140] After round E of pruning based on the information entropy of the convolution kernel is completed, another round of pruning based on the L1 norm of the vectors inside the convolution kernel is performed.

[0141] A K×K convolution kernel can be viewed as K K-dimensional row vectors or K K-dimensional column vectors; for the convolution kernels retained after step S4, the L1 norm of each vector inside these convolution kernels is calculated, and the selection is performed from the perspectives of rows and columns.

[0142] The L1 norm of a vector x within the convolution kernel is calculated using the following formula:

[0143]

[0144] S6: Prune the convolution kernel with a preset fixed threshold to complete the pruning of the convolution kernel size, thereby reducing the dimensionality of the convolution kernel size;

[0145] First, we filter from the perspective of rows. For each row vector, we calculate its L1 norm and then sort it by size.

[0146] After selecting L row vectors with larger L1 norms, the convolution kernel size becomes L×K, which can be regarded as K L-dimensional column vectors.

[0147] Calculate the L1 norm of the K column vectors. Similarly, sort them and select the L column vectors with larger L1 norms. At this time, the convolution kernel size becomes L×L, and the pruned client-side local training model is obtained.

[0148] To illustrate, taking AlexNet as an example, AlexNet has 5 convolutional layers. The first layer has 96 11×11 convolutional kernels, which are pruned into 8 3×3 convolutional kernels. The second layer has 256 5×5 convolutional kernels, which are pruned into 18 3×3 convolutional kernels. The third to fifth layers have 384, 384 and 256 3×3 convolutional kernels respectively, which are pruned to retain 25, 23 and 19 3×3 convolutional kernels. Therefore, the model pruning method not only reduces the number of convolutional kernels, but also reduces the size of the convolutional kernels.

[0149] S7: The updated models from each client are uploaded to the central server for aggregation. The server then returns the globally updated models to each client to complete one iteration of the federated learning process.

[0150] The pruning evaluation metrics for the model pruning method in federated learning include the convolution kernel importance index and the L1 norm of the vectors inside the convolution kernel.

[0151] The model pruning method for federated learning includes two pruning operations: E rounds of pruning based on the kernel entropy to determine the number of convolutional kernels, and one round of pruning based on the L1 norm of the kernel's internal vectors to determine the kernel size. The pruning based on the L1 norm of the kernel's internal vectors begins after all the pruning operations based on the kernel entropy to determine the number of convolutional kernels have been completed.

[0152] The convolutional kernel importance metrics include the mean and standard deviation of the information entropy of each convolutional kernel, which reflect the amount of information that the network model can extract from the convolutional kernel.

[0153] The dynamic threshold model pruning method described above continuously changes the mean and standard deviation of the convolution kernel information entropy throughout the pruning process, λ. (e)The pruning threshold changes from -1 to 1 as the pruning process progresses. It is negative when the number of pruning iterations is less than half the total number of iterations, and positive when it is greater than half. Convolutional kernels exceeding the mean are pruned. Therefore, the pruning intensity is relatively high in the early stages of training, resulting in larger model oscillations. As the iteration process continues, the pruning threshold gradually increases, the pruning intensity gradually decreases, and model oscillations decrease until convergence, thus avoiding over-pruning and under-pruning.

[0154] Example 4

[0155] This embodiment 4 provides a non-transitory computer-readable storage medium for storing computer instructions. When executed by a processor, these computer instructions implement the federated learning-oriented model pruning method described above. The method includes:

[0156] For the neural network model to be pruned on the client side, calculate the importance index of the convolutional kernel in each convolutional layer;

[0157] Using a dynamic threshold model pruning method, the number of convolution kernels with importance indices below a threshold is pruned.

[0158] For the convolution kernels retained after kernel number pruning, calculate the L1 norm of each vector inside these convolution kernels;

[0159] Based on the calculated L1 norm, the convolution kernel is pruned according to a preset threshold to complete the kernel size pruning, thereby reducing the dimensionality of the convolution kernel.

[0160] Example 5

[0161] This embodiment 5 provides a computer program product, including a computer program that, when run on one or more processors, implements the federated learning-oriented model pruning method described above. The method includes:

[0162] For the neural network model to be pruned on the client side, calculate the importance index of the convolutional kernel in each convolutional layer;

[0163] Using a dynamic threshold model pruning method, the number of convolution kernels with importance indices below a threshold is pruned.

[0164] For the convolution kernels retained after kernel number pruning, calculate the L1 norm of each vector inside these convolution kernels;

[0165] Based on the calculated L1 norm, the convolution kernel is pruned according to a preset threshold to complete the kernel size pruning, thereby reducing the dimensionality of the convolution kernel.

[0166] Example 6

[0167] This embodiment 6 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions implementing the federated learning-oriented model pruning method as described above, the method including:

[0168] For the neural network model to be pruned on the client side, calculate the importance index of the convolutional kernel in each convolutional layer;

[0169] Using a dynamic threshold model pruning method, the number of convolution kernels with importance indices below a threshold is pruned.

[0170] For the convolution kernels retained after kernel number pruning, calculate the L1 norm of each vector inside these convolution kernels;

[0171] Based on the calculated L1 norm, the convolution kernel is pruned according to a preset threshold to complete the kernel size pruning, thereby reducing the dimensionality of the convolution kernel.

[0172] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A model pruning method for federated learning, characterized in that, The method comprises the following steps: For a neural network model for image processing to be pruned by a client, an importance index of a convolution kernel in each convolution layer is calculated; The importance index of the convolution kernel is determined according to the following calculation steps: calculating the information entropy of each convolution kernel in each convolution layer; The mean and standard deviation of the information entropy of the convolution kernel are calculated to obtain the importance index of the convolution kernel; The information entropy of the convolution kernel is calculated, including that the information entropy in the convolution kernel is calculated according to the following formula: ； in, It is the first The first convolutional layer The weights of each convolution kernel, It is the number of input channels of this layer, i.e., the number of convolutional kernels. It is the size of the convolution kernel; The element values ​​in the data are evenly divided into There are several intervals, and then the frequency corresponding to each interval is calculated. Indicates the first The ratio of the number of elements in each interval to the total number of elements; Assume the probability distribution of the convolution kernel weights approximately follows a Gaussian normal distribution; given the probability distribution of the weights, the... The first convolutional layer The information entropy of each convolutional kernel is transformed into the difference entropy of a continuous random variable; the discrete weight information of the convolutional kernel is transformed into information entropy calculated using a continuous probability distribution; the mean and standard deviation of the information entropy of the convolutional kernel are calculated to obtain the importance index of the convolutional kernel. The mean and standard deviation of the information entropy of the convolutional kernel in each convolutional layer are calculated according to the following formula: ； ； wherein, is the current pruning round, is the first layer convolution kernel information entropy of the first layer convolution kernel in the first round of pruning, is the mean of the first layer convolution kernel information entropy, is the standard deviation of the first layer convolution kernel information entropy; A dynamic threshold pruning method is used to prune the number of convolutional kernels whose importance index is below a threshold; determining the threshold includes: calculating the mean of the information entropy. and standard deviation Then, a parameter related to the pruning cycle is introduced. : ; wherein, is a hyperbolic tangent function, whose domain is the entire real number field, and whose range is (-1, 1); is the total pruning round of the convolution kernel number pruning; the function value is negative when the independent variable is less than 0, and the function value is positive when the independent variable is greater than 0; The pruning threshold of each round is ; For the convolution kernels reserved after the convolution kernel number pruning is completed, the L1 norm of each vector in the convolution kernels is calculated; Based on the calculated L1 norm, the convolution kernels are pruned based on a preset threshold to complete the convolution kernel size pruning, thereby reducing the dimension of the convolution kernel size.

2. The federated learning oriented model pruning method according to claim 1, characterized in that, An L1 norm of an internal vector of a convolution kernel, including: an L1 norm of a vector within a convolution kernel is calculated according to the following formula: .

3. A federated learning oriented model pruning system, characterized in that, The method comprises the following steps: A first calculation module is configured to calculate an importance index of a convolution kernel in each convolution layer for a neural network model for image processing to be pruned by a client; The importance index of the convolution kernel is determined according to the following calculation steps: calculating the information entropy of each convolution kernel in each convolution layer; The mean and standard deviation of the information entropy of the convolution kernel are calculated to obtain the importance index of the convolution kernel; The information entropy of the convolution kernel is calculated, including that the information entropy in the convolution kernel is calculated according to the following formula: ； wherein, is the weight of the th convolution kernel in the th convolution layer, is the number of input channels of the layer, i.e., the number of convolution kernels, is the size of the convolution kernel; the element values in are evenly divided into intervals, and then the frequency corresponding to each interval is calculated, represents the ratio of the number of elements in the th interval to the total number of elements. Assume the probability distribution of the convolution kernel weights approximately follows a Gaussian normal distribution; given the probability distribution of the weights, the... The first convolutional layer The information entropy of each convolutional kernel is transformed into the difference entropy of a continuous random variable; the discrete weight information of the convolutional kernel is transformed into information entropy calculated using a continuous probability distribution; the mean and standard deviation of the information entropy of the convolutional kernel are calculated to obtain the importance index of the convolutional kernel. The mean and standard deviation of the information entropy of the convolutional kernel in each convolutional layer are calculated according to the following formula: ； ； wherein, is the current pruning round, is the first layer convolution kernel information entropy of the first layer convolution kernel in the first round of pruning, is the average of the first layer convolution kernel information entropy, is the standard deviation of the first layer convolution kernel information entropy; The first pruning module is configured to prune the convolution kernel with an importance index lower than a threshold by using a dynamic threshold model pruning method; the threshold is determined by: calculating the mean value of information entropy and the standard deviation , and then introducing a parameter related to the pruning round : . wherein, is a hyperbolic tangent function, whose domain is the entire real number field, and whose range is (-1, 1); is the total pruning round of the convolution kernel number pruning; the function value is negative when the independent variable is less than 0, and the function value is positive when the independent variable is greater than 0; The pruning threshold of each round is ; A second calculation module is configured to calculate the L1 norm of each vector in the convolution kernels reserved after the convolution kernel number pruning is completed; A second pruning module is configured to prune the convolution kernels based on the calculated L1 norm and based on a preset threshold to complete the convolution kernel size pruning, thereby reducing the dimension of the convolution kernel size.

4. A non-transitory computer-readable storage medium, comprising, The non-transitory computer-readable storage medium is configured to store computer instructions, which, when executed by a processor, implement the model pruning method for federated learning according to claim 1 or 2.

5. A computer program product, characterised in that, The computer program, when running on one or more processors, is configured to implement the model pruning method for federated learning according to claim 1 or 2.

6. An electronic device, comprising: The method comprises the following steps: A processor, a memory and a computer program are provided, wherein the processor is connected to the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute instructions for implementing the model pruning method for federated learning according to claim 1 or 2.

Images