Machine learning model compression method based on federated learning and mean iteration

Abstract

The present invention discloses a machine learning model compression method based on federated learning and mean iteration, comprising the following steps: a user performs federated learning on a local end to obtain the machine learning model of this time; the machine learning model obtained by the user through federated learning is compressed using a mean iteration model compression algorithm; the compressed machine learning model is uploaded to a central server, the compressed machine learning model is restored and the model is aggregated, and the aggregated machine learning model is subjected to precision detection. The present invention dynamically adjusts the boundary and optimization value through a mean iteration compression algorithm to obtain the optimal value and the optimal boundary interval of the quantization interval, thereby ensuring the convergence of the model and the accuracy of the model at a high compression rate.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a mean iteration model compression method based on federated learning. Background Art

[0002] In the field of artificial intelligence, data is the foundation of machine learning. However, in most industries, data often exists in the form of isolated islands due to industry competition, privacy security, and complex administrative procedures. Even the integration of data between different departments of the same company faces many obstacles. In reality, it is almost impossible to integrate data scattered across various places and institutions, or the cost is huge. With the further development of artificial intelligence, paying attention to data privacy and security has become a global trend.

[0003] Federated learning is a machine learning architecture that aims to help multiple institutions use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations, so as to achieve better learning results and privacy protection performance. Compared with the traditional distributed machine learning architecture, federated learning exchanges trained models instead of raw data before training, thereby ensuring the privacy of private data without reducing the learning effect. Therefore, federated learning allows each user to participate in joint learning while maintaining data privacy and collaborate to achieve common goals and tasks. Due to the distributed structure of federated learning, each user needs to transmit the local training model to the central server. Considering that the existing model has an increasing trend of change, the communication overhead between users and the central server will become larger and larger. By compressing the model, the communication overhead can be reduced, thereby improving the communication efficiency of the system.

[0004] Compression technology has important application value in the rapidly developing wireless communication transmission. Data compression refers to a technical method that reduces the amount of data to reduce storage space, improves its transmission, storage and processing efficiency, or reorganizes data according to a certain algorithm to reduce data redundancy and storage space without losing useful information. Compression is divided into two categories: lossless compression and lossy compression. Lossless compression is generally achieved by encoding the original file in some way, and then restoring a percentage of the data of the source file before and after transmission. However, due to the 100% data restoration characteristic of lossless compression itself, it is difficult to achieve a high compression rate; lossy compression loses some information, so that a high compression rate can be achieved through a certain algorithm.

[0005] In distributed machine learning systems, due to the characteristics of machine learning model parameters, the use of lossless compression cannot significantly reduce the model size. Since the machine model has a strong ability to resist external interference, the use of lossy compression can give the model a high compression ratio, so it has many applications in distributed machine learning systems. According to the FetchSGD scheme proposed in the literature 1 (Rothchild D, Panda A, Ullah E, et al. FetchSGD: Communication-Efficient Federated Learning with Sketching [J]. 2020.), lossy compression with high compression rate can be achieved, but the scheme is based on hash tables to achieve parameter compression and restoration. The randomness of the scheme during compression and decompression makes it impossible for the scheme to provide a convergence guarantee. When using this scheme, it takes a lot of time to adjust the length and width structure of the hash table and the final size of the hash table according to the actual model, and finally it is possible to achieve a high compression rate. The QSGD scheme proposed in document 2 (D Alistarh, D Grubic, Li J, et al. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding [J]. 2016.) can achieve convergence. This scheme is based on the boundary and quantization level to achieve one-time quantization for compression. When the quantization level is high (i.e., the compression rate is low), the accuracy problem can be guaranteed. However, at a high compression rate, due to the one-time quantization problem of its own algorithm, the error between its quantized parameters and original parameters is too large at a high compression rate, and its accuracy test effect is low. However, the traditional federated learning architecture, such as the FedAvg architecture proposed by Google, is difficult to achieve a good balance between compression rate and model test accuracy after compressing the user-trained model. Summary of the invention

[0006] The purpose of the present invention is to provide a machine learning model compression method based on federated learning and mean iteration that can balance the compression rate and model testing accuracy, reduce the size of the model uploaded by the user, and reduce the communication overhead of the central server.

[0007] The technical solution to achieve the purpose of the present invention is: a machine learning model compression method based on federated learning and mean iteration, comprising the following steps:

[0008] Step 1: The user performs federated learning on the local end to obtain the machine learning model;

[0009] Step 2: Compress the machine learning model obtained by the user's federated learning using the mean iteration model compression algorithm;

[0010] Step 3: Upload the compressed machine learning model to the central server, restore and aggregate the compressed machine learning model, and perform accuracy testing on the aggregated machine learning model.

[0011] Furthermore, federated learning is a machine learning framework that enables multiple units to use data and perform machine learning modeling while meeting the requirements of user privacy protection, data security and government regulations. Federated learning is any of vertical federated learning, horizontal federated learning, transfer federated learning, centralized federated learning, and decentralized federated learning.

[0012] Furthermore, the user refers to a device capable of performing computing, which may be any one of a mobile terminal, a computer, and an edge router.

[0013] Furthermore, a machine learning model is a set of data calculation processes, which is mathematically expressed as a function and physically expressed as a piece of code. After the data is operated by this code, the corresponding output results are obtained, including the model parameters of the support vector machine, the model parameters of the multilayer perceptron, the model parameters of the neural network, and the model parameters of reinforcement learning.

[0014] Furthermore, in step 1, the user performs federated learning on the local side to obtain the machine learning model of this time, which is represented by g i .

[0015] Furthermore, the machine learning model obtained by performing federated learning on the user in step 2 is compressed using a mean iteration model compression algorithm, as follows:

[0016] Step 21: Set the quantization level M and use the user-trained model g i , sorted into a key-value pair sequence with a model length of L, sorted from small to large, and the formula is expressed as:

[0017] g sort_i ={θ 1 ,θ 2 ,......θ L-1 ,θ L}

[0018] g keys_i ={key 1 ,key 2 ,......,key L-1 ,key L}

[0019] Among them, g sort_i represents the model sorted by user i, {θ 1 ,θ 2 ,......θL-1 ,θ L} represents user g sort_i Parameters after internal sorting of the model; g keys_i represents the index sequence sorted by user i; {key 1 ,key 2 ,......,key L-1 ,key L} represents the index value after user sorting;

[0020] In order to transmit the quantized binary code at the end, a quantized code sequence of length L is defined, and the mathematical formula is expressed as: Code i ={c 1 ,c 1 ,......,c L-1 ,c L}, where Code i Represents the encoded model, {c 1 ,c 1 ,......,c L-1 ,c L} represents the binary coded value corresponding to the model parameter of user i;

[0021] Step 22: Set the number of boundary value samples K, satisfying the condition K = M + 1. In the initial state, K boundary values ​​are randomly given, mathematically expressed as: D i ={d 1 ,d 2 ,......,d K-1 ,d K}, the initial boundary value satisfies: θ 1 =d 1 <d 2 ,......,d K-1 <d K =θ L ; The number of quantized coding values ​​set inside the boundary is M, mathematically expressed as: Coded i = {v 1 ,v 2 ,......,v M-1 ,v M}, where Coded i It represents the quantized code value of user i;

[0022] Step 23: Calculate the mean of the model parameters inside each adjacent boundary, and use the mean as the quantized code value for this iteration update. i The coding model is used for quantization coding, and the error and Q corresponding to each coding value are calculated. err ={err 1,err 2 ,.....,err M-1 ,err M}, the solution process is as follows:

[0023]

[0024]

[0025] …

[0026]

[0027]

[0028] Among them, v M Represents the quantized code value, Len(θ L ) means that under the constraint d M ≤θ L ≤d M+1 The number of parameters when c L Represents the binary code, err M represents the error sum of each boundary, d M ≤θ L ≤d M+1 is the parameter θ L Constraints of

[0029] Step 24: Calculate the total quantization error Q of this model all , mathematically expressed as:

[0030] Q all =∑err m ,m=1,2,......,M

[0031] Step 25: Update the boundary value by quantizing the code value, which is mathematically expressed as:

[0032]

[0033] According to the above steps, the quantization error Q is solved. all ;

[0034] Repeat steps 23 to 25 to calculate the next quantization error Q next :If the errors are not equal, perform iterative operations and repeat steps 23 to 25; if they are equal, the optimal solution has been found, and g keys_i ={key 1 ,key 2 ,......,key L-1 ,key L} and Code i ={c1 ,c 2 ,......,c L-1 ,c L}, solve the binary code Code of the original model sorting orig_i ={s 1 ,s 2 ,......,s L-1 ,s L}, output model binary code Code orig_i ={s 1 ,s 2 ,......,s L-1 ,s L} and quantized coded value Coded i = {v 1 ,v 2 ,......,v M-1 ,v M}.

[0035] Furthermore, in step 3, the compressed machine learning model is uploaded to the central server, the compressed machine learning model is restored and the model is aggregated, and the accuracy of the aggregated machine learning model is tested, as follows:

[0036] Model restoration refers to restoring the compressed model uploaded by the user on the central server using mathematical methods;

[0037] Model aggregation means that after restoring the compressed models uploaded by each user to the central server, the parameters corresponding to each restored model are summed and averaged to finally obtain the aggregated global model;

[0038] Accuracy testing refers to the accuracy testing of the global model after model aggregation, including accuracy pre-testing on the central server and accuracy pre-testing on the user.

[0039] Furthermore, the step 3 is specifically as follows:

[0040] Step 31: User i compresses the model Code orig_i and Coded i Upload to the central server;

[0041] Step 32: The central server restores the compressed models uploaded by each user, and then aggregates them on the central server to obtain an aggregated global model;

[0042] Step 33: Perform accuracy test on the aggregated global model.

[0043] Compared with the prior art, the present invention has the following significant advantages: (1) it adopts the mean iteration model compression method, which can ensure system convergence while ensuring a high compression rate; (2) it has high test accuracy, which reduces the communication overhead of the central server and improves the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the machine learning model compression method based on federated learning and mean iteration of the present invention.

[0045] Figure 2 It is a system schematic diagram of model compression transmission in federated learning of the present invention.

[0046] Figure 3 It is a comparison chart of communication performance between the compressed model and the uncompressed model in the embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention discloses a machine learning model compression method based on federated learning and mean iteration. The method refers to performing model training on users in federated learning, compressing the trained model using a mean iteration model compression algorithm, uploading the compressed model to a central server, restoring and aggregating the model, and then performing accuracy detection to reduce the communication overhead from the user to the central server, and includes the following steps:

[0048] Step 1: The user performs federated learning on the local end to obtain the machine learning model;

[0049] Step 2: Compress the machine learning model obtained by the user's federated learning using the mean iteration model compression algorithm;

[0050] Step 3: Upload the compressed machine learning model to the central server, restore and aggregate the compressed machine learning model, and perform accuracy testing on the aggregated machine learning model.

[0051] Furthermore, users of federated learning conduct model training, which is characterized by the fact that federated learning is a machine learning framework that can effectively help multiple units use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations, including vertical federated learning, horizontal federated learning, transfer federated learning, centralized federated learning, and decentralized federated learning.

[0052] Furthermore, users refer to devices that can perform computing, including mobile terminals, computers, and edge routers.

[0053] Furthermore, a machine learning model is a set of data calculation process methods, which is mathematically expressed as a function and physically expressed as a piece of code. After the data is operated by this code (such as addition, subtraction, multiplication, division, or other operation steps), the corresponding output results can be obtained, including the model parameters of the support vector machine, the model parameters of the multilayer perceptron, the model parameters of the neural network, and the model parameters of reinforcement learning.

[0054] Furthermore, the step 1 is specifically as follows:

[0055] The user trains the model in this round, and the trained model is mathematically expressed as: g i .

[0056] Furthermore, mean iteration model compression is a data compression method for learning models. It is mathematically expressed as a function and physically expressed as a piece of code. After the data is operated by this code (such as addition, subtraction, multiplication, division, or other operation steps), the corresponding output result can be obtained. Mean iteration model compression includes the model parameters of support vector machines, the model parameters of multilayer perceptrons, the model parameters of neural networks, and the model parameters of reinforcement learning.

[0057] The machine learning model obtained by performing federated learning on the user in step 2 is compressed using the mean iteration model compression algorithm, as follows:

[0058] Step 21: Set the quantization level M and use the user-trained model g i , sorted into a key-value pair sequence with a model length of L, sorted from small to large, and the formula is expressed as:

[0059] g sort_i ={θ 1 ,θ 2 ,......θ L-1 ,θ L}

[0060] g keys_i ={key 1 ,key 2 ,......,key L-1 ,key L}

[0061] Among them, g sort_i represents the model sorted by user i, {θ 1 ,θ 2 ,......θ L-1 ,θ L} represents user g sort_i Parameters after internal sorting of the model; g keys_irepresents the index sequence sorted by user i; {key 1 ,key 2 ,......,key L-1 ,key L} represents the index value after the user sorts;

[0062] In order to transmit the quantized binary code at the end, a quantized code sequence of length L is defined, and the mathematical formula is expressed as: Code i ={c 1 ,c 1 ,......,c L-1 ,c L}, where Code i Represents the encoded model, {c 1 ,c 1 ,......,c L-1 ,c L} represents the binary coded value corresponding to the model parameter of user i.

[0063] Step 22: Set the number of boundary value samples K, satisfying the condition K = M + 1. In the initial state, K boundary values ​​are randomly given, mathematically expressed as: D i ={d 1 ,d 2 ,......,d K-1 ,d K}, the initial boundary value satisfies: θ 1 =d 1 <d 2 ,......,d K-1 <d K =θ L ; The number of quantized coding values ​​set inside the boundary is M, mathematically expressed as: Coded i = {v 1 ,v 2 ,......,v M-1 ,v M}, where Coded i It represents the quantized code value of user i.

[0064] Step 23: Calculate the mean of the model parameters inside each adjacent boundary, and use the mean as the quantized code value for this iteration update. i The coding model is used for quantization coding, and the error and Q corresponding to each coding value are calculated. err ={err 1 ,err 2 ,.....,err M-1 ,err M}, the solution process is as follows:

[0065]

[0066]

[0067] …

[0068]

[0069]

[0070] Among them, v M Represents the quantized code value, Len(θ L ) means that under the constraint d M ≤θ L ≤d M+1 The number of parameters when c L Represents the binary code, err M represents the error sum of each boundary, d M ≤θ L ≤d M+1 is the parameter θ L constraints.

[0071] Step 24: Calculate the total quantization error Q of this model all , mathematically expressed as:

[0072] Q all =∑err m ,m=1,2,......,M

[0073] Step 25: Update the boundary value by quantizing the code value, which is mathematically expressed as:

[0074]

[0075] According to the above steps, the quantization error Q is solved. all ;

[0076] Repeat steps 23 to 25 to calculate the next quantization error Q next :If the errors are not equal, perform iterative operations and repeat steps 23 to 25; if they are equal, the optimal solution has been found, and g keys_i ={key 1 ,key 2 ,......,key L-1 ,key L} and Code i ={c 1 ,c 2 ,......,c L-1 ,c L}, solve the binary code Code of the original model sorting orig_i ={s 1 ,s 2 ,......,s L-1 ,s L}, output model binary code Code orig_i ={s 1 ,s 2 ,......,s L-1 ,s L} and quantized coded value Coded i = {v 1 ,v 2 ,......,v M-1 ,v M}.

[0077] Furthermore, in step 3, the compressed machine learning model is uploaded to the central server, the compressed machine learning model is restored and the model is aggregated, and the accuracy of the aggregated machine learning model is tested, as follows:

[0078] Model restoration refers to the use of specific mathematical methods to restore the compressed model uploaded by the user on the central server. Mathematically, it is a function, and physically it is a piece of code. After the data is operated by this code (such as addition, subtraction, multiplication, division, or other operation steps), the corresponding output result can be obtained.

[0079] Model aggregation is to restore the compressed models uploaded by each user to the central server, sum and average the parameters corresponding to each restored compressed model, and finally obtain the aggregated global model.

[0080] Accuracy testing refers to the accuracy testing of the global model after model aggregation, including accuracy pre-testing on the central server and accuracy pre-testing on the user.

[0081] Furthermore, the step 3 is specifically as follows:

[0082] Step 31: User i compresses the model Code orig_i and Coded i Upload to the central server;

[0083] Step 32: The central server restores the compressed models uploaded by each user, and then aggregates them on the central server to obtain an aggregated global model;

[0084] Step 33: Perform accuracy test on the aggregated global model.

[0085] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by method personnel in this field all fall within the scope defined by the claims attached to this application.

[0086] Example 1

[0087] In a decentralized federated learning architecture, each user needs to upload his or her trained model to the central server, which then aggregates the models to generate a global model. Under such a framework process, as the learning model becomes more and more complex, the size of the model also becomes larger and larger. At this time, for the central server, the communication overhead component becomes a major problem in a communication system. The mean iteration compression model method proposed in the present invention saves communication resources and reduces communication overhead. For each user who uploads the model, while ensuring that the system converges and the accuracy meets the requirements, the process of this implementation is as follows: Figure 1 As shown in the figure, the user compresses the uploaded model locally. After the user sets the compression level and uses the mean iteration optimization algorithm, the encoded values ​​of the model parameters and the encoded equivalent parameters are obtained after compression. After transmitting the compressed model to the central server, the user's compressed model is restored and then aggregated to obtain the global model after this training optimization. The global model is then broadcast to each user. At this point, a complete process of mean iteration compression and model restoration is completed. The model transmission system implemented in this embodiment is as follows Figure 2 As shown, user 1 to user i are trained to obtain models 1 to model n respectively, and then the models are compressed and uploaded to the central server for restoration and aggregation. The central server performs accuracy tests on the aggregated models. After the test is completed, the global model tested by the central server is broadcast to users 1 to n respectively, thus completing a complete process of model compression upload, compression, accuracy test and global model distribution. After using the design of the present invention, the communication overhead can be reduced to a very low level, while also ensuring that the test accuracy and the accuracy of the aggregated model before compression remain almost unchanged. The results of this implementation are shown in FIG. Figure 3 As shown:

[0088] Figure 3 In the figure, we set up 32 times, 16 times compression and uncompressed models to communicate with the central server 50 times. It can be seen from the figure that the system model still converges when the original model and the compressed model are transmitted. From the results, the method of the present invention can ensure that the model still converges after compression when the accuracy is within the expected range, significantly reducing the communication overhead between users and the central server, improving the overall performance of the system, and has a very broad application prospect.

Claims

1. A machine learning model compression method based on federated learning and mean iteration, characterized in that: The following steps are involved: Step 1: The user performs federated learning on the local side to obtain the machine learning model, which is represented by g i ; Step 2: Compress the machine learning model obtained by the user's federated learning using the mean iteration model compression algorithm, as follows: Step 21: Set the quantization level M and use the user-trained model g i , sorted into a key-value pair sequence with a model length of L, sorted from small to large, and the formula is expressed as: g sort_i ={θ1,θ2,......θ L-1 ,i L } g keys_i ={key1,key2,......,key L-1 ,key L } Among them, g sort_i represents the model sorted by user i, {θ1,θ2,......θ L-1 ,θ L } represents user g sort_i Parameters after internal sorting of the model; g keys_i represents the index sequence sorted by user i; {key1, key2, ..., key L-1 ,key L } represents the index value after the user sorts; In order to transmit the quantized binary code at the end, a quantized code sequence of length L is defined, and the mathematical formula is expressed as: Code i ={c1,c1,......,c L-1 ,c L }, where Code i represents the encoded model, {c1,c1,......,c L-1 ,c L } represents the binary coded value corresponding to the model parameter of user i; Step 22: Set the number of boundary value samples K, satisfying the condition K = M + 1. In the initial state, K boundary values ​​are randomly given, mathematically expressed as: D i ={d1,d2,......,d K-1 ,d K }, the initial boundary value satisfies: θ1=d1<d2,......,d K-1 <d K =θ L ; The number of quantized coding values ​​set inside the boundary is M, mathematically expressed as: Coded i ={v1,v2,......,v M-1 ,v M }, where Coded i It represents the quantized code value of user i; Step 23: Calculate the mean of the model parameters inside each adjacent boundary, and use the mean as the quantized code value for this iteration update. i The coding model is used for quantization coding, and the error and Q corresponding to each coding value are calculated. err ={err1,err2,.....,err M-1 ,err M }, the solution process is as follows: … Among them, v M Represents the quantized code value, Len(θ L ) means that under the constraint d M ≤θ L ≤d M+1 The number of parameters when c L Represents the binary code, err M represents the error sum of each boundary, d M ≤θ L ≤d M+1 is the parameter θ L Constraints of Step 24: Calculate the total quantization error Q of this model all , mathematically expressed as: Q all =∑err m ,m=1,2,......,M Step 25: Update the boundary value by quantizing the code value, which is mathematically expressed as: According to the above steps, the quantization error Q is solved. all ; Repeat steps 23 to 25 to calculate the next quantization error Q next :If the errors are not equal, perform iterative operations and repeat steps 23 to 25; if they are equal, the optimal solution has been found, and g keys_i ={key1,key2,......,key L-1 ,key L } and Code i ={c1,c2,......,c L-1 ,c L }, solve the binary code Code of the original model sorting orig_i ={s1,s2,......,s L-1 ,s L }, output model binary code Code orig_i ={s1,s2,......,s L-1 ,s L } and quantized coded value Coded i ={v1,v2,......,v M-1 ,v M }; Step 3: Upload the compressed machine learning model to the central server, restore and aggregate the compressed machine learning model, and perform accuracy testing on the aggregated machine learning model.

2. The method for compressing a machine learning model based on federated learning and mean iteration according to claim 1, characterized in that: Federated learning is a machine learning framework that enables multiple units to use data and perform machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations. Federated learning is any of vertical federated learning, horizontal federated learning, transfer federated learning, centralized federated learning, and decentralized federated learning.

3. The machine learning model compression method based on federated learning and mean iteration according to claim 1, characterized in that: A user refers to a device capable of computing, which can be any of a mobile terminal, a computer, or an edge router.

4. The method for compressing a machine learning model based on federated learning and mean iteration according to claim 1, characterized in that: A machine learning model is a set of data calculation processes, which is mathematically expressed as a function and physically expressed as a piece of code. After the data is operated by this code, the corresponding output result is obtained, including the model parameters of the support vector machine, the model parameters of the multilayer perceptron, the model parameters of the neural network, and the model parameters of reinforcement learning.

5. The method for compressing a machine learning model based on federated learning and mean iteration according to claim 1, characterized in that: In step 3, the compressed machine learning model is uploaded to the central server, the compressed machine learning model is restored and the model is aggregated, and the accuracy of the aggregated machine learning model is tested as follows: Model restoration refers to restoring the compressed model uploaded by the user on the central server using mathematical methods; Model aggregation means that after restoring the compressed models uploaded by each user to the central server, the parameters corresponding to each restored model are summed and averaged to finally obtain the aggregated global model; Accuracy testing refers to the accuracy testing of the global model after model aggregation, including accuracy pre-testing on the central server and accuracy pre-testing on the user.

6. The method for compressing a machine learning model based on federated learning and mean iteration according to claim 5, characterized in that: The step 3 is as follows: Step 31: User i compresses the model Code orig_i and Coded i Upload to the central server; Step 32: The central server restores the compressed models uploaded by each user, and then aggregates them on the central server to obtain an aggregated global model; Step 33: Perform accuracy test on the aggregated global model.

Images