A Distributed Machine Learning Model Transmission Compression Method Based on Model Decomposition

Abstract

This invention discloses a distributed machine learning model transmission compression method based on model decomposition. By using a compression algorithm to decompose and compress the model gradient layer by layer, the method compresses the amount of data transmitted between the worker node and the parameter server node in each iteration during the training of the distributed machine learning model. This effectively reduces the amount of communication data, achieves a trade-off between communication overhead and training model accuracy, alleviates the communication bottleneck problem in distributed training, and improves the model training speed while ensuring training accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of model transmission compression in distributed machine learning, and particularly relates to a distributed machine learning model transmission compression method based on model decomposition. BACKGROUND

[0002] In recent years, machine learning has achieved good application effects in the fields of image recognition, natural language processing, automatic driving and the like. With the increase of training data volume and the complication of machine learning models, the storage resources and computing resources of a single machine cannot complete the training task in an effective time. Distributed machine learning, as a new training structure for accelerating machine learning, has emerged as the times require and received extensive attention from the academic and industrial circles.

[0003] Distributed machine learning allocates computing tasks to multiple nodes in a computer cluster, utilizes the computing resources and storage resources of multiple nodes for parallel training, accelerates the training process and reduces the time required for training. The parameter server architecture (PS architecture) is a common distributed machine learning architecture. The parameter server architecture divides the nodes in the computing cluster into parameter server nodes and worker nodes according to functions, the parameter server is responsible for storing model parameters, and the worker node is responsible for model parameter update calculation. In the training process, the worker node receives the global model parameters from the parameter server node, trains according to the local training data using the latest global model, and calculates the model gradient, and then the worker node transmits the model gradient to the parameter server node. After the parameter server node receives the latest model gradient from each worker node, it converges and updates the global model parameters, and then distributes the updated global model to each worker node for the next iteration.

[0004] In the distributed training based on the parameter server, with the continuous increase of the number of computing nodes and the continuous increase of the size of the training model, in multiple iterations, the communication data volume of model transmission between the worker node and the parameter server node is too large, and the limited communication bandwidth cannot transmit data in time, which will cause network congestion and delay, which will consume more communication resources and time, so that the communication problem becomes the bottleneck of distributed machine learning, the training process cannot be effectively carried out, and the scalability of distributed machine learning is affected. Therefore, communication has become the main bottleneck in distributed machine learning.

[0005] Researchers have proposed various methods to alleviate the communication bottleneck problem of distributed training. Among the gradient transmission compression methods, gradient quantization and gradient sparsification are widely studied. Gradient quantization refers to representing the gradient with high precision in low precision. However, the data compression rate of this method is limited and cannot meet the compression needs of large-scale networks. Moreover, this method may lead to a decrease in the accuracy of the training model. Gradient sparsification refers to transmitting only the gradients that have a relatively large contribution to the model update in each iteration, and not transmitting the gradients with a small contribution to the model update. The non-transmitted gradients can be used for error accumulation. This method has a higher compression rate and is more flexible than the gradient quantization method. However, it also brings a large computational overhead. When the additional computational overhead exceeds the reduced communication overhead, it will result in a longer distributed training time. Moreover, the non-transmitted gradients will affect the subsequent training, and therefore, gradient sparsification may cause fluctuations in the accuracy of the model.

[0006] Therefore, in order to alleviate the communication bottleneck problem in distributed training and reduce the amount of communication transmission parameters, the model transmission compression method will affect the convergence process and the accuracy of the training model to some extent. The existing gradient transmission compression methods, such as gradient quantization and gradient sparsification, do not consider the characteristics of the data itself, and therefore may lead to problems such as a decrease in convergence speed or a decrease in model accuracy. SUMMARY

[0007] Therefore, the present application provides a distributed machine learning model transmission compression method based on model decomposition, which realizes model training with high convergence speed and accuracy.

[0008] The distributed machine learning model transmission compression method based on model decomposition provided by the present application comprises the following steps:

[0009] Step 1: The gradient tensors of each layer of the model of the worker node are matrixed and expanded into gradient tensor initial matrices according to their sizes;

[0010] Step 2: The gradient tensor initial matrices are decomposed and compressed into compressed matrices by using a compression algorithm;

[0011] Step 3: The worker node sends the compressed matrices formed in the current iteration to the server node;

[0012] Step 4: The server node reconstructs the compressed matrices to obtain reconstructed gradients, and updates the global model according to the learning rate, the reconstructed gradients of all worker nodes and the current global model parameters to obtain updated global model parameters, and finally sends the updated global model parameters to all worker nodes;

[0013] Step 5: If the current iteration number is less than a threshold value, step 1 is performed; otherwise, the training of the distributed machine learning model is completed, and the process ends.

[0014] Further, the step 1 is to splice the bias layer in the worker node model with the corresponding weight layer and then perform matrix expansion.

[0015] Further, when the compression algorithm in the step 2 is PCA, the process of decomposing and compressing the gradient tensor initial matrix into a compressed matrix in the step 2 is as follows:

[0016] Step 1.1, if the number of rows of the gradient tensor initial matrix is less than the number of columns, transpose the gradient tensor initial matrix and then execute step 1.2; otherwise, execute step 1.2.

[0017] Step 1.2, center the gradient tensor initial matrix to obtain the gradient tensor initial matrix mean Calculate the covariance matrix, decompose the covariance matrix to obtain the eigenvalue and the eigenvector;

[0018] Step 1.3, arrange the eigenvalue and the corresponding eigenvector in descending order according to the eigenvalue value;

[0019] Step 1.4, multiply the preset eigenvector number ratio a and the number of eigenvectors to obtain the projection matrix column number l, and take the first l columns of the sorted eigenvectors to generate the projection matrix W.

[0020] Step 1.5, subtract the gradient tensor initial matrix mean from the projection matrix W to obtain the following calculation:

[0021]

[0022] Obtain the dimensionality reduction gradient Y.

[0023] Step 1.6, form the compressed matrix from the gradient tensor initial matrix mean, the dimensionality reduction gradient and the projection matrix.

[0024] Further, the step 1.4 is to determine the projection matrix column number according to the preset eigenvalue proportion threshold value: calculate whether the proportion of the sum of the first i sorted eigenvalues in the sum of all eigenvalues exceeds the set threshold value, when the threshold value is exceeded, the first i eigenvalues corresponding eigenvectors are selected to form the projection matrix W, otherwise, the number of eigenvalues participating in the summation is increased until the proportion exceeds the threshold value, and the eigenvalues corresponding eigenvectors participating in the summation are selected to generate the projection matrix W.

[0025] Further, when the compression algorithm in the step 2 is SVD, the process of decomposing and compressing the gradient tensor initial matrix into a compressed matrix in the step 2 is as follows:

[0026] Step 2.1, decompose the matrix X TX is decomposed to obtain eigenvalues and eigenvectors, wherein X is the gradient tensor initial matrix;

[0027] Step 2.2, if the number of rows m of the gradient tensor initial matrix X is greater than the number of columns n, step 2.3 is performed, if m is not greater than n, step 2.6 is performed;

[0028] Step 2.3, the eigenvalues and the corresponding eigenvectors are arranged in descending order according to the numerical value of the eigenvalues to form a right singular value matrix V;

[0029] Step 2.4, the eigenvalues are square root processed to obtain singular values to generate a singular value matrix σ;

[0030] Step 2.5, U=XV T σ is calculated to obtain a left singular value matrix U, and step 2.9 is performed;

[0031] Step 2.6, the eigenvalues and the corresponding eigenvectors are arranged in descending order according to the numerical value of the eigenvalues to form a left singular value matrix U;

[0032] Step 2.7, the eigenvalues are square root processed to obtain singular values to generate a singular value matrix σ;

[0033] Step 2.8, V=σU T X is calculated to obtain a right singular value matrix V;

[0034] Step 2.9, the singular value matrix σ, the left singular value matrix U and the right singular value matrix V constitute a compression matrix.

[0035] Further, when the compression algorithm in step 2 is KLT, the process of decomposing and compressing the gradient tensor initial matrix into a compression matrix in step 2 is as follows:

[0036] Step 3.1, the autocorrelation matrix R=E[XX T ] of the gradient tensor initial matrix X is calculated, and the autocorrelation matrix is decomposed to obtain eigenvalues and eigenvectors;

[0037] Step 3.2, the eigenvalues and the corresponding eigenvectors are arranged in descending order according to the numerical value of the eigenvalues;

[0038] Step 3.3, the eigenvectors corresponding to the first d eigenvalues are selected to constitute a matrix U;

[0039] Step 3.4, Y=UX is calculated to obtain a transformed matrix Y, and the matrix Y and the matrix U constitute a compression matrix.

[0040] Beneficial effects:

[0041] This invention employs a compression algorithm to decompose and compress the model gradient layer by layer, thereby compressing the amount of data transmitted between worker nodes and parameter server nodes in each iteration of the distributed machine learning model training process. This effectively reduces the amount of communication data, achieves a trade-off between communication overhead and training model accuracy, alleviates the communication bottleneck problem in distributed training, and improves model training speed while ensuring training accuracy. Attached Figure Description

[0042] Figure 1 The flowchart illustrates a distributed machine learning model transmission compression method based on model decomposition provided by this invention.

[0043] Figure 2 The flowchart shows the PCA compression algorithm used in the distributed machine learning model transmission compression method based on model decomposition provided by this invention. Detailed Implementation

[0044] The following examples illustrate the invention in detail.

[0045] Distributed machine learning can employ neural network models. Classic neural network models include fully connected neural networks, convolutional neural networks, and residual networks. Commonly used neural network models include LeNet, ResNet, and AlexNet. Neural network models can be functionally divided into three parts: input layer, hidden layer, and output layer. This invention provides a distributed machine learning model transmission compression method based on model decomposition. The basic idea is as follows: On the worker nodes, after obtaining the model gradient in each round of training, a compression algorithm is used to compress the model gradient, decomposing it into multiple small matrices. These decomposed small matrices are then transmitted to the parameter server to reduce the amount of communication data in each iteration. After receiving the compressed data from each worker node, the parameter server node approximates the original gradient from the compressed data, thereby significantly reducing the amount of model gradient transmission and obtaining a high-performance training model.

[0046] This invention provides a distributed machine learning model transmission compression method based on model decomposition, which specifically includes the following steps:

[0047] Step 1: Expand the gradient tensors of each layer of the working node model into an initial gradient tensor matrix based on their size.

[0048] In this invention, during distributed training under a parameter server architecture, the components of the machine learning neural network include input layers, convolutional layers, fully connected layers, residual blocks, etc. Furthermore, the neural network model contains different types of layers such as weight layers and bias layers. Some layers have higher orders, while others are first-order tensors, such as bias layers. Therefore, different layers need to be processed differently during dimensionality reduction. To reduce the amount of transmitted data, this invention, when performing decompression and compression operations on the model gradients of the worker nodes, needs to process mostly high-order tensors. Existing compression algorithms generally process matrices. Therefore, to achieve decompression and compression of the model gradients of the worker nodes, the model gradients first need to be preprocessed to convert them from high-order tensors to low-order tensors.

[0049] Furthermore, since the bias layer has a relatively small amount of data, its gradient can be directly transmitted, or the bias layer can be concatenated with the corresponding weight layer before matrix expansion. During training, after each worker node completes its current iteration, the gradient tensor of each layer in the neural network is successively expanded into a matrix according to its size.

[0050] Furthermore, since the gradient scales of different layers are different, the number of rows and columns of the expanded matrix are also not fixed. In order to obtain a higher data compression rate, when the number of rows of the initial matrix of the expanded gradient tensor is less than the number of columns, it is transposed first; otherwise, no transpose operation is required.

[0051] Step 2: Use a compression algorithm to decompose the initial gradient tensor matrix into multiple smaller matrices to form a compressed matrix.

[0052] Existing compression algorithms include PCA (Principal Component Analysis), SVD (Singularly Valuable Decomposition), and KLT (Karhunen-Loeve Transform). Among these, the classic dimensionality reduction algorithm PCA can reduce data dimensionality while preserving data features, yielding compressed data matrices and projection matrices. The SVD algorithm can be used for eigenvalue decomposition, which represents complex matrices by multiplying smaller matrices. The KLT feature extraction method performs orthogonal transformations on the data to extract features and obtain decomposed matrices. Many of the algorithms listed above can effectively compress data volume. By selecting a compression algorithm to decompose the expanded matrix, the resulting smaller matrices become smaller than the initial gradient tensor matrix.

[0053] Step 3: The worker node sends the compressed matrix formed during the current round of training to the server node.

[0054] Step 4: The server node reconstructs the gradient based on the received compression matrix, then updates the global model based on the learning rate, the reconstructed gradients of all worker nodes, and the current global model parameters to obtain the updated global model parameters. Finally, the updated global model parameters are sent to all worker nodes.

[0055] During training, the initial learning rate (lr) is usually set based on experience. As training progresses, the learning rate will decay and gradually approach zero. The server node sums the reconstruction gradients of all worker nodes, calculates the average gradient, and updates the global model using mini-batch stochastic gradient descent. The inputs are the learning rate, reconstruction gradients, and current global model parameters, and the output is the updated global model parameters.

[0056] Step 5: If the current iteration count is less than the threshold, proceed to step 1; otherwise, the distributed machine learning model training is complete, and the process ends.

[0057] In distributed machine learning, this invention provides a distributed machine learning model transmission compression method based on model decomposition. By selecting a suitable compression algorithm, the initial gradient tensor matrix obtained during training is decomposed and compressed into smaller matrices. Each worker node then transmits these smaller matrices to the server node. This method compresses the amount of data transmitted during distributed machine learning model training by transmitting the smaller matrices, thereby reducing the amount of communication data per iteration, reducing distributed training communication overhead, and accelerating model training. The compression algorithm can be selected from PCA, SVD, KLT, etc.

[0058] Example 1:

[0059] This embodiment describes how, when PCA is selected as the compression algorithm, a distributed machine learning model transmission compression method based on model decomposition provided by this invention is used to compress the amount of data transmitted for model training. Specifically, it includes the following steps:

[0060] Step 1.1: Expand the gradient tensor of a certain layer of the working node model into an initial gradient tensor matrix based on its size.

[0061] Step 1.2: The initial gradient tensor matrix is ​​decomposed into multiple smaller matrices using the PCA compression algorithm to form a compressed matrix, including the following steps:

[0062] Step 1.2.1: If the number of rows in the initial matrix of the gradient tensor is less than the number of columns, transpose it and then proceed to step 1.2.2; otherwise, proceed to step 1.2.2.

[0063] Step 1.2.2: Decenter the initial gradient tensor matrix and calculate the covariance matrix; decompose the covariance matrix to obtain eigenvalues ​​and eigenvectors.

[0064] Step 1.2.3: Sort the eigenvalues ​​and their corresponding eigenvectors in descending order of eigenvalue values.

[0065] Step 1.2.4: Determine the number of columns in the projection matrix based on the preset proportion of eigenvectors α or the eigenvalue proportion threshold η, and generate the projection matrix from the eigenvectors. If the number of columns in the projection matrix is ​​determined based on the preset proportion of eigenvectors, multiply the number of eigenvectors by the proportion α to obtain the number of columns l in the projection matrix, and take the first l columns of the sorted eigenvectors to generate the projection matrix W. If the number of columns in the projection matrix is ​​determined based on the preset eigenvalue proportion threshold η, calculate whether the proportion of the sum of the first i sorted eigenvalues ​​in the sum of all eigenvalues ​​exceeds the set threshold. If it exceeds the threshold, take the eigenvectors corresponding to the first i eigenvalues ​​to form the projection matrix W. Otherwise, increase the number of eigenvalues ​​to be summed until their proportion exceeds the threshold, and then generate the projection matrix W from the eigenvectors corresponding to the eigenvalues ​​involved in the summation.

[0066] Step 1.2.5: Calculate the dimensionality-reduced gradient based on the projection matrix. This is derived from the mean of the initial matrix of the decentralized gradient tensor. The dimensionality reduction gradient is calculated using the projection matrix W. The initial matrix mean of the gradient tensor, the dimensionality-reduced gradient, and the projection matrix are the small matrices obtained after decomposing the model gradient, i.e., the compression matrix.

[0067] Step 1.3: The worker node sends the compressed matrix formed during the current round of training to the server node.

[0068] Step 1.4: The server node reconstructs the gradient based on the received compression matrix, then updates the global model based on the learning rate, the reconstructed gradients of all worker nodes, and the current global model parameters to obtain the updated global model parameters. Finally, the updated global model parameters are sent to all worker nodes.

[0069] Step 1.5: If the current iteration count is less than the threshold, proceed to step 1.1; otherwise, the distributed machine learning model training is complete, and this process ends.

[0070] Example 2:

[0071] This embodiment describes how, when SVD is selected as the compression algorithm, a distributed machine learning model transmission compression method based on model decomposition provided by this invention is used to compress the amount of data transmitted for model training. Specifically, it includes the following steps:

[0072] Step 2.1: Expand the gradient tensor of a certain layer of the working node model into an initial gradient tensor matrix based on its size.

[0073] Step 2.2: The initial gradient tensor matrix is ​​decomposed into multiple smaller matrices to form a compressed matrix using the SVD compression algorithm, including the following steps:

[0074] Step 2.2.1: Determine the number of rows m and columns n of the initial matrix X of the gradient tensor. If m is greater than n, proceed to step 2.2.2. If m is not greater than n, proceed to step 2.2.6.

[0075] Step 2.2.2: For matrix XX T Decompose the eigenvalues ​​and eigenvectors.

[0076] Step 2.2.3: Sort the eigenvalues ​​and their corresponding eigenvectors in descending order of eigenvalue values. The sorted eigenvectors form the right singular value matrix V.

[0077] Step 2.2.4: Perform square root processing on the eigenvalues ​​to obtain singular values ​​and generate the singular value matrix σ.

[0078] Step 2.2.5, from U = XV T σ is used to calculate the left singular value matrix U, and then step 2.3 is executed.

[0079] Step 2.2.6, for matrix X T X is decomposed to obtain eigenvalues ​​and eigenvectors.

[0080] Step 2.2.7: Sort the eigenvalues ​​and their corresponding eigenvectors in descending order of eigenvalue values. The sorted eigenvectors form the left singular value matrix U.

[0081] Step 2.2.8: Perform square root processing on the eigenvalues ​​to obtain singular values ​​and generate the singular value matrix σ.

[0082] Step 2.2.9, from V = σU T X calculates the right singular value matrix V.

[0083] Step 2.3: The singular value matrix σ, the left singular value matrix U, and the right singular value matrix V are decomposed into smaller matrices, i.e., compressed matrices; the worker node sends the compressed matrices formed during the current round of training to the server node.

[0084] Step 2.4: The server node reconstructs the gradient based on the received compression matrix, then updates the global model based on the learning rate, the reconstructed gradients of all worker nodes, and the current global model parameters to obtain the updated global model parameters. Finally, the updated global model parameters are sent to all worker nodes.

[0085] Step 2.5: If the current iteration count is less than the threshold, proceed to step 2.1; otherwise, the distributed machine learning model training is complete, and this process ends.

[0086] Example 3:

[0087] This embodiment describes how, when KLT is selected as the compression algorithm, a distributed machine learning model transmission compression method based on model decomposition provided by this invention is used to compress the amount of data transmitted for model training. Specifically, it includes the following steps:

[0088] Step 3.1: Expand the gradient tensor of a certain layer of the working node model into an initial gradient tensor matrix based on its size.

[0089] Step 3.2: The initial gradient tensor matrix is ​​decomposed into multiple smaller matrices using the KLT compression algorithm to form a compressed matrix, including the following steps:

[0090] Step 3.2.1: Calculate the autocorrelation matrix R = E[XX] of the expansion matrix. T The autocorrelation matrix is ​​decomposed to obtain eigenvalues ​​and eigenvectors.

[0091] Step 3.2.2: Sort the eigenvalues ​​and their corresponding eigenvectors in descending order of eigenvalue values.

[0092] Step 3.2.3: Select the eigenvectors corresponding to the first d eigenvalues ​​to form matrix U.

[0093] Step 3.2.4: Calculate the transformed matrix Y using Y = UX. Matrix Y and matrix U are small matrices to be transmitted to the parameter server node, i.e., the compressed matrices.

[0094] Step 3.3: The worker node sends the compressed matrix formed during the current round of training to the server node.

[0095] Step 3.4: The server node reconstructs the gradient based on the received compression matrix, then updates the global model based on the learning rate, the reconstructed gradients of all worker nodes, and the current global model parameters to obtain the updated global model parameters. Finally, the updated global model parameters are sent to all worker nodes.

[0096] Step 3.5: If the current iteration count is less than the threshold, proceed to step 3.1; otherwise, the distributed machine learning model training is complete, and this process ends.

[0097] The performance of this invention was evaluated experimentally. Image recognition accuracy was tested using the AlexNet network on the MNIST dataset. With PCA compression algorithm, an eigenvalue proportion threshold of 0.8 was set, and the number of worker nodes was 4. The MNIST dataset was divided across 4 worker nodes for training. After distributed machine learning training began, each worker node trained according to the initial AlexNet network parameters, calculating the model gradient for one iteration. At each worker node, the gradient was matrix-expanded layer by layer to obtain the initial gradient tensor matrix. It was then determined whether the initial gradient tensor matrix needed to be transposed to ensure the number of columns was less than the number of rows. Next, the mean of each column vector in the initial gradient tensor matrix was calculated. The mean was subtracted from each column element to obtain a decentralized initial gradient tensor matrix. The covariance matrix was then calculated row-wise from the decentralized matrix, and the covariance matrix was decomposed to obtain eigenvalues ​​and eigenvectors. The eigenvalues ​​and corresponding eigenvectors were then sorted in descending order of eigenvalue value. The projection matrix was determined using the eigenvalue proportion threshold of 0.8, and the dimension-reduced gradient matrix was calculated. Following the steps above, the gradients of the remaining layers (excluding the bias layer) are compressed. The mean of the initial gradient tensor matrix, the reduced gradient matrix, and the projection matrix after this round of compression and decomposition are then sent to the parameter server. Upon receiving the data, the parameter server reconstructs the gradients and updates the global model using mini-batch stochastic gradient descent based on the reconstructed gradients of all working nodes, the learning rate, and the current global model. The parameter server then distributes the updated global model to each working node for the next iteration of training. Training is complete when the number of iterations reaches a set value. Experimental results show that compared to algorithms that do not compress the transmitted gradients, when using PCA as the compression algorithm, the accuracy of this invention decreases slightly by 0.3%, while the data transmission volume is reduced by approximately 98%, significantly reducing the distributed training communication overhead.

[0098] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A model decomposition based distributed machine learning model transmission compression method, characterized in that, The method comprises the following steps: Step 1, the gradient tensor of each layer of the worker node model is matrixed and expanded according to its size into a gradient tensor initial matrix; Step 2, a compression algorithm is used to decompose and compress the gradient tensor initial matrix into a compressed matrix; Step 3, the worker node sends the compressed matrix formed in the current iteration to the server node; Step 4, the server node reconstructs the compressed matrix to obtain a reconstructed gradient, and updates the global model according to the learning rate, the reconstructed gradient of all worker nodes and the current global model parameter to obtain an updated global model parameter, and finally sends the updated global model parameter to all worker nodes; Step 5, if the current iteration number is less than a threshold value, step 1 is executed; Otherwise, the training of the distributed machine learning model is completed, and the process ends. In step 1, the bias layer and the corresponding weight layer in the worker node model are spliced and then matrixed and expanded.

2. The distributed machine learning model transmission compression method of claim 1, wherein, When the compression algorithm in step 2 is PCA, the process of decomposing and compressing the gradient tensor initial matrix into a compressed matrix in step 2 is as follows: Step 1.1, if the number of rows of the gradient tensor initial matrix is less than the number of columns, transpose the gradient tensor initial matrix and execute step 1.2; otherwise, execute step 1.2; Step 1.2, decentering the initial matrix of gradient tensor to obtain the mean value of the initial matrix of gradient tensor The covariance matrix is calculated, and the eigenvalues and eigenvectors are obtained by decomposing the covariance matrix; Step 1.3, sort the eigenvalues and corresponding eigenvectors in descending order according to the eigenvalue values; Step 1.4, multiply the preset eigenvector quantity ratio α by the eigenvector quantity to obtain the projection matrix column number l, and take the first l columns of the sorted eigenvectors to generate the projection matrix W; Step 1.5, Mean of gradient tensor initial matrix and the projection matrix W are computed as follows: Step 1.5, obtain the dimensionality reduction gradient Y; Step 1.6, form the compressed matrix from the gradient tensor initial matrix mean, the dimensionality reduction gradient and the projection matrix.

3. The distributed machine learning model transmission compression method of claim 2, wherein, In step 1.4, the projection matrix column number is determined according to the preset eigenvalue proportion threshold η: whether the proportion of the sum of the first i sorted eigenvalues in the sum of all eigenvalues exceeds the set threshold value is calculated, when the threshold value is exceeded, the projection matrix W is constructed by the first i eigenvalues corresponding to the eigenvectors, otherwise, the number of eigenvalues participating in the summation is increased until the proportion exceeds the threshold value, and the projection matrix W is generated by selecting the eigenvalues corresponding to the eigenvectors participating in the summation.

4. The distributed machine learning model transmission compression method of claim 1, wherein, When the compression algorithm in step 2 is SVD, the process of decomposing and compressing the gradient tensor initial matrix into a compressed matrix in step 2 is as follows: Step 2.1, decompose the matrix X T X to obtain eigenvalues and eigenvectors, wherein X is a gradient tensor initial matrix; Step 2.2, if the number of rows m of the gradient tensor initial matrix X is greater than the number of columns n, execute step 2.3, if m is not greater than n, execute step 2.6; Step 2.3, sort the eigenvalues and corresponding eigenvectors in descending order according to the eigenvalue values to form the right singular value matrix V; Step 2.4, square root processing is performed on the eigenvalues to obtain singular values and generate the singular value matrix σ; Step 2.5, compute left singular matrix U from X = U V T σcompute left singular matrix U, perform Step 2.9; Step 2.6, sort the eigenvalues and corresponding eigenvectors in descending order according to the eigenvalue values to form the left singular value matrix U; Step 2.7, square root processing is performed on the eigenvalues to obtain singular values and generate the singular value matrix σ; Step 2.8, V = σU T X computes the right singular matrix V; Step 2.9, the compressed matrix is composed of the singular value matrix σ, the left singular value matrix U and the right singular value matrix V.

5. The distributed machine learning model transmission compression method according to claim 1, characterized in that, When the compression algorithm in step 2 is KLT, the process of decomposing and compressing the gradient tensor initial matrix into a compressed matrix in step 2 is as follows: Step 3.1, compute the autocorrelation matrix R = E[XX T ] of the initial matrix X of the gradient tensor; decompose the autocorrelation matrix to obtain eigenvalues and eigenvectors; Step 3.2, arranging the eigenvalues and corresponding eigenvectors in descending order according to the eigenvalue values; Step 3.3, selecting the eigenvectors corresponding to the first d eigenvalues to form a matrix U; Step 3.4, calculating a transformed matrix Y by Y=UX, wherein the matrix Y and the matrix U form a compression matrix.

Images