A Distributed Machine Learning Method and System Based on Gradient Compression and Quantization

Abstract

The present invention discloses a distributed machine learning method and system based on gradient compression and quantization, which relates to the technical field of communication resource computing. The method includes the steps of: the server performs data compression processing on the gradient information of the client, wherein the client gradient information is preliminarily compressed by using discretized quantization processing, and the preliminarily compressed information is further compressed by using an encoding method; the compressed gradient information is distributed to the client; the client adaptively calculates and selects the gradient threshold by using an edge computing method to obtain the updated gradient information; the client uploads the updated information to the server. In this application, when the server distributes gradient information, a method combining discrete quantization and encoding is used to compress the gradient distributed by the server, and then on the basis of discrete quantization, an encoding method is used to further compress the data, so as to achieve efficient compression of the gradient.

Description

Technical Field

[0001] The present invention relates to the technical field of communication resource computing, and particularly to a distributed machine learning method and system based on gradient compression and quantization. Background Art

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] With the rapid development of Internet and Internet of Things technologies, the amount of data has increased exponentially. When dealing with such a large amount of data, traditional centralized machine learning methods not only have high computing costs, but also have potential risks in terms of data privacy and security. Distributed machine learning can effectively alleviate these problems by dispersing data and computing tasks to multiple computing nodes. It allows data preprocessing and preliminary learning to be performed on local nodes, and then the learning results are aggregated to a central node for integration, thereby improving learning efficiency and reducing the burden on the central server.

[0004] Although distributed machine learning has many theoretical advantages, in practical applications, especially in environments with limited communication resources, it faces significant challenges. Limited communication resources mean that when transmitting data and model updates between nodes, the limitations of communication bandwidth and latency must be considered. In such an environment, traditional distributed machine learning algorithms often result in huge communication overheads, thereby reducing the overall learning efficiency.

[0005] In distributed machine learning, communication overheads are mainly reflected in the process of transmitting model parameters, gradient information, update information, etc. between nodes. Due to the large amount of data, the transmission of this information often consumes a large amount of communication resources. Especially in a wireless network environment, where the communication bandwidth is limited and the latency is high, the problem of communication overheads is particularly prominent. Therefore, how to reduce communication overheads and improve communication efficiency has become an urgent problem in the field of distributed machine learning.

[0006] In recent years, some progress has been made in the research on the problem of limited communication resources. For example, the FedBCD algorithm repeatedly uses the statistical data of the previous round for multiple iterations to reduce the number of communications. The Flex-VFL algorithm sets a threshold for the interval between communication rounds, allowing each party to perform a different number of local iterations. However, these algorithms usually have certain requirements for the overall training architecture, and the training scenarios are not universal. The Sparse Ternary Compression (STC) framework extends the existing top-k gradient dilution compression technology and introduces a new mechanism that enables weight updates to perform downstream compression, normalization, and optimal Golomb coding. In addition, a new distributed quantization gradient method has been proposed, which has the characteristics of quantized gradient adaptive communication, but it is difficult to determine the gradient of a constant threshold in practice. The Lazily Aggregated Gradient (LAG) algorithm has also been proposed, which can adaptively calculate the gradient and skip part of the gradient communication, thereby reducing the communication bandwidth and alleviating the server pressure. The basic principle is to detect the slowly changing gradient and compress these gradients, but LAG cannot be used to optimize convex problems.

[0007] In summary, how to overcome the defects of poor practicability, low compression efficiency, and unstable gradient threshold in the gradient compression process has become an urgent problem to be solved in the existing technology. Summary of the Invention

[0008] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a distributed machine learning method and system based on gradient compression and quantization. When the server distributes gradient information, a method combining discrete quantization and coding is used to compress the gradient distributed by the server. Then, on the basis of discrete quantization, a coding method is used to further compress the data, so as to achieve efficient compression of the gradient.

[0009] To achieve the above purpose, the present invention is implemented through the following technical solutions:

[0010] The first aspect of the present invention provides a distributed machine learning method based on gradient compression and quantization, including the following steps:

[0011] The server obtains the client gradient information and caches it;

[0012] Perform data compression processing on the client gradient information. Among them, the client gradient information is initially compressed by using discretized quantization processing, and the initially compressed information is further compressed by using a coding method;

[0013] Distribute the compressed gradient information to the client;

[0014] After the client receives the compressed gradient information, it adaptively calculates and selects the gradient threshold using an edge computing method to obtain the updated gradient information;

[0015] The client uploads the updated information to the server.

[0016] Furthermore, the specific steps for the server to obtain and cache the client gradient information are as follows:

[0017] Before distributing the gradient information to the clients, first, based on the gradient information of the corresponding client in the previous round cached locally, its distribution is statistically analyzed, and the mean and variance are calculated to obtain the threshold interval to be quantized.

[0018] Furthermore, the specific steps for data compression processing of the client gradient information include:

[0019] Iterate on the gradient information, and perform quantization processing on the gradient information before each round of iteration to achieve data compression;

[0020] Use Huffman coding to further compress the information volume of the quantized gradient information to obtain the optimal variable-length coding.

[0021] Even further, the specific steps for quantizing the gradient information are as follows:

[0022] Obtain the threshold interval to be quantized, evenly divide the threshold interval into P sub-intervals to form P + 1 endpoint values;

[0023] Approximate each element in the gradient information to the nearest endpoint.

[0024] Even further, after each round of iteration of the gradient information, the server will re-record and statistically analyze the original uncompressed gradient distribution of each client to achieve real-time dynamic update of the quantization interval and optimize the compression effect.

[0025] Furthermore, when distributing the compressed gradient information to the clients, concatenate the encoded values of all elements in the gradient information bit by bit to form a bit string, and distribute the bit string and the gradient mapping code table to the corresponding clients.

[0026] Furthermore, the specific steps for adaptively calculating and selecting the gradient threshold using an edge computing method are as follows:

[0027] The edge server first distributes the current global model to all subordinate device nodes;

[0028] Subsequently, each device node calculates the local parameter gradient and uploads it to the edge side, and the edge server obtains the total gradient by weighted summation;

[0029] Set the trust threshold through an adaptive algorithm, and use the trust threshold to evaluate the trust of all gradients to obtain a set of trustworthy communication nodes.

[0030] The second aspect of the present invention provides a distributed machine learning system based on gradient compression and quantization, including:

[0031] An information acquisition module, configured to enable the server to acquire client gradient information and cache it;

[0032] An information compression module, configured to perform data compression processing on the client gradient information. Among them, the client gradient information is preliminarily compressed by using discretized quantization processing, and the preliminarily compressed information is further compressed by using an encoding method;

[0033] An information distribution module, configured to distribute the compressed gradient information to the client;

[0034] A local processing module, configured to, after the client receives the compressed gradient information, adaptively calculate and select the gradient threshold by using an edge computing method to obtain updated gradient information;

[0035] An information feedback module, configured to enable the client to upload the updated information to the server.

[0036] The third aspect of the present invention provides a medium, on which a program is stored, and when the program is executed by a processor, the steps in the distributed machine learning method based on gradient compression and quantization described in the first aspect of the present invention are implemented.

[0037] The fourth aspect of the present invention provides a device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the distributed machine learning method based on gradient compression and quantization described in the first aspect of the present invention are implemented.

[0038] The above one or more technical solutions have the following beneficial effects:

[0039] The present application discloses a distributed machine learning method and system based on gradient compression and quantization. When the server distributes gradient information, a method combining discrete quantization and encoding is used to compress the gradients distributed by the server. By discretizing continuous gradient values into a finite number of levels, the data volume of the gradients can be significantly reduced. This can reduce the space and time required for storing and transmitting gradients, thereby reducing the communication burden on the server. Subsequently, on the basis of discrete quantization, an encoding method can be used to further compress the data. After discrete quantization and encoding processing, when the server distributes gradient values, it only needs to concatenate the encoded values of all elements in the gradient bit by bit to form a bit string, and transmit this string and the gradient mapping code table, thereby achieving efficient compression of the gradients.

[0040] This application uses Huffman coding. This coding may perform optimal variable-length coding on the gradient information after discrete processing according to its occurrence frequency. This coding method greatly shortens the overall coding length, thereby improving the communication efficiency. At the same time, it is a lossless compression algorithm that can maintain the integrity of the data while compressing the data. That is, during the decoding process, the original gradient data can be completely restored without introducing any errors or distortions, improving a certain error tolerance rate.

[0041] After the model is trained well on the client side and the gradient information is uploaded to the server while reducing the amount of data distributed by the server, this application adopts the method of adaptive threshold gradient compression. This method can adaptively select which gradient information needs to be transmitted and which can be discarded or delayed, and can reduce the amount of data to be transmitted, which can not only ensure the transmission of key gradient information, but also reduce the transmission of redundant information, thereby maintaining the convergence speed and accuracy of the model.

[0042] Advantages of additional aspects of the present invention will be partly given in the following description, partly will become obvious from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0044] Figure 1 It is a flowchart of the distributed machine learning method based on gradient compression and quantization in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof;

[0047] Embodiment 1:

[0048] Currently, when distributed machine learning compresses data for the problem of limited communication resources, there are the following difficulties:

[0049] 1. It is difficult to achieve efficient communication of the algorithm during the gradient compression process: Some gradient compression methods may cause partial loss of information. Although the communication is efficient, it will reduce the final model performance. At the same time, it is also necessary to consider whether the method used for compressing the gradient will generate high computing resources, etc.

[0050] 2. It is difficult to achieve accurate gradient quantization. Gradient quantization will introduce certain errors, and such errors may affect the training effect and convergence speed of the model.

[0051] 3. It is difficult to select the encoding. Selecting the correct encoding method can improve the compression rate as much as possible on the premise of ensuring data accuracy. The encoding and decoding processes may introduce additional computational overhead, which may affect the real-time performance of gradient update. The selected encoding method should have a certain degree of adaptability and flexibility, and be able to be adjusted and optimized according to different scenarios. At the same time, the encoding method should ensure good compatibility and efficiency at the decoding end.

[0052] Based on the above difficulties, Embodiment 1 of the present invention provides a distributed machine learning method based on gradient compression and quantization, as Figure 1 shown, including two parts: the server distributes gradients to the client and the client uploads gradients to the server. The cloud server aggregates the gradient information of the clients, generates a gradient distribution table for n clients, and performs gradient quantization encoding, so as to achieve the compression of gradient data. During the communication interaction process, the cloud server encodes the compressed gradient information, aggregates it through the edge server, and distributes it to the client. The client performs local gradient information accumulation by adaptively screening gradients. The specific process is as follows: Determine whether the node after training is trustworthy. If it is trustworthy, upload the gradient information; otherwise, skip this round of communication for local accumulation. After the client model is trained, the trustworthy gradient information is uploaded to the cloud server through the edge server.

[0053] Specifically, it includes the following steps:

[0054] Step 1: The server obtains the client gradient information and caches it.

[0055] Step 1.1: The server caches a corresponding gradient information for each client locally, as shown in the gradient distribution table in Figure 1.

[0056] Before distributing the gradient information to the client, first statistically analyze its distribution according to the gradient information of the corresponding client in the previous round cached locally, and calculate the mean and variance .

[0057] Step 1.2: The distribution of the gradient information obtained in the previous round is a normal distribution. Most of the data will be concentrated near the mean value, and the probability of data farther from the mean value is smaller. To remove some extreme gradient information and facilitate quantization by constraining the boundary of the gradient value, the threshold interval of the gradient information to be distributed in the current round is set according to the 3σ rule as , and the elements outside this interval in the gradient are set to zero to obtain the threshold interval to be quantized.

[0058] Step 2: Perform data compression processing on the client gradient information. Among them, the client gradient information is initially compressed using discretized quantization processing, and the initially compressed information is further compressed using an encoding method.

[0059] Step 2.1: Iterate on the gradient information, and perform quantization processing on the gradient information before each round of iteration to achieve data compression.

[0060] Step 2.1.1: Obtain the threshold interval to be quantized, and evenly divide the threshold interval into sub-intervals to form endpoint values.

[0061] Step 2.1.2: Approximate each element in the gradient information to the nearest endpoint.

[0062] Step 2.1.3: After discrete quantization processing, all gradient values within the threshold interval can be represented by these interval endpoint values, and the values outside the interval are 0.

[0063] Step 2.2: Use Huffman coding to further compress the information volume of the quantized gradient information to obtain an optimal variable-length code.

[0064] To further compress the information volume, in this embodiment, Huffman coding is used to perform optimal variable-length coding on a total of possible values according to their occurrence frequencies after processing. The specific process is as follows:

[0065] 1. First, according to the 3σ rule and the quantization steps, all gradient values have been mapped to a finite set of discrete values (the set includes P + 1 quantization interval endpoint values and the values outside the interval (set to 0)).

[0066] 2. Secondly, count the occurrence frequency of each quantized value, that is, the number of times each discretized gradient value appears in the current training round.

[0067] 3. Then, based on this frequency information, construct a Huffman tree; starting from the root node of the Huffman tree, assign binary codes according to the structure of the tree (usually assign 0 to the left branch and 1 to the right branch), and each leaf node (i.e., the quantized gradient value) will ultimately be assigned a binary code.

[0068] 4. Finally, use the generated Huffman codes to compress the quantized gradients. Each quantized value (including 0) will be replaced by its corresponding Huffman code.

[0069] Taking the threshold interval and the number of quantization intervals both being 2 as an example, the gradient mapping code table is shown in Table 1.

[0070] Table 1. Gradient mapping code table

[0071]

[0072] After each round of discrete and coded compression iteration of the gradient information, the server will re-record and statistically analyze the uncompressed gradient distribution of each client before the next iteration to achieve real-time dynamic update of the quantization interval and optimize the compression effect.

[0073] Step 3: Distribute the compressed gradient information to the clients.

[0074] After discrete quantization and coding processing, when distributing the compressed gradient information as the global model to the clients, only need to concatenate the coded values of all elements in the gradient information bit by bit to form a bit string, and distribute the bit string and the gradient mapping code table to the corresponding clients, so as to achieve efficient compression of the gradients and reduce the amount of data distributed by the server.

[0075] Step 4: After the client receives the compressed gradient information, the corresponding edge server of the client uses edge computing methods to train the local model, and obtains the updated gradient information by adaptively calculating and selecting the gradient threshold.

[0076] Step 4.1: Assume that a single edge server performs gradient update with the th client, and represents the set of all device nodes under this edge server.

[0077] Step 4.2: In the th local iteration, the edge server first distributes the current global model to all subordinate device nodes.

[0078] Step 4.3: Subsequently, each device node calculates the local parameter gradient and uploads it to the edge side, and the edge server obtains the total gradient by weighted summation .

[0079] The most important thing in this step is to determine whether a communication node is worthy of communication. Nodes that are not worthy of communication are represented by and nodes that are worthy of communication are represented by All client node sets , and the total gradient can be expressed as:

[0080] (1).

[0081] Among them, represents the total gradient, represents the total gradient of nodes that are not worthy of communication, represents the total gradient of nodes that are worthy of communication.

[0082] Step 4.4: Set the trust threshold through an adaptive algorithm, and use the trust threshold to evaluate the trust of all gradients to obtain a set of trustworthy communication nodes.

[0083] When uploading gradient information, the selection of the threshold for adaptive threshold gradient compression is very important. Due to the characteristics of distributed learning, the data of clients is usually non-independent and identically distributed, and the data volume may be unbalanced, which may lead to significant differences in the gradients of some clients from those of other clients, affecting the selection of the threshold. At the same time, the selection of the threshold needs to balance the accuracy and compression rate of the gradient. Too high a threshold may cause excessive loss of gradient information and affect the training effect of the model; while too low a threshold may not achieve the expected compression effect. In this embodiment, the trust threshold is determined through the following process:

[0084] For nodes that are not worthy of communication, they are defined here as:

[0085] (2).

[0086] Update rule of the client local model: After the client device updates the local model, calculate the average gradient parameter:

[0087] (3).

[0088] Among them, is the model parameter, is the th client, is the learning rate, is the total number of local model iterations, represents the gradient calculated in the previous round, is the th client, , is the total number of clients

[0089] Substituting Equation (3) into (2) gives:

[0090] (4).

[0091] Wherein, represents the round parameters of the server, that is, the gradient Therefore, the sum of the information of the round gradient is expressed as:

[0092] (5).

[0093] Wherein, represents the gradient of the device node at round, is the set of device points under a single edge server, .

[0094] From (5), by the mean value inequality, we can get the following:

[0095] (6).

[0096] After the above analysis, if the client node is an untrusted node, that is, m ∈ , when the node satisfies Equation (7), the updated parameters must satisfy the gradient descent algorithm .

[0097] (7).

[0098] Next, a proportionality coefficient is needed to measure the ratio of untrusted nodes to the total number of nodes in the client node, as follows:

[0099] (8).

[0100] A large proportionality coefficient means that there are more untrusted nodes in this round of training process, that is, more device nodes skip this round of communication, and vice versa; in practical applications, it is dynamically adjusted. In the initial stage, a smaller can be selected to ensure that most nodes participate in communication and help the model converge quickly; in the subsequent stage, as the model gradually stabilizes, can be appropriately increased to reduce the communication volume and improve the training efficiency.

[0101] Substituting Equation (8) into Equation (7) gives:

[0102] (9).

[0103] Given that when updating the parameters, it is not easy to obtain, but the parameter tends to be smooth during model training, so it is approximated as:

[0104] (10).

[0105] In the above formula and D are both set constant coefficients. For the convenience of calculation, in this embodiment, is selected, . Substituting formula (8) into formula (10) can obtain the client node gradient screening formula as follows:

[0106] (11).

[0107] This formula comprehensively considers the gradient information of the previous rounds and uses the weighted average method to smooth the estimation of the previous round gradients, which helps to more accurately judge the change trend of the gradients. By introducing the historical gradient information, it is also possible to more smoothly estimate the current gradient change and avoid misjudgment caused by extremely large or small gradients in a certain round.

[0108] The principle of the above node gradient screening formula is based on the analysis of the gradient magnitude and change trend, and judges whether a node needs to participate in communication by setting a certain threshold. Specifically, when the gradient change of a node is small and meets certain conditions, the node is considered an untrusted node and this round of communication can be skipped, thus reducing the communication overhead. When the gradient change of a node is large, the node is considered a trusted node and must participate in communication to update the global model.

[0109] According to formula (11), the device nodes can be adaptively judged. When a node satisfies this formula, this round of communication is skipped, the local gradient is accumulated, and the next round of training is continued. If a node does not satisfy this formula, that is, it is judged as a trustworthy node, then the node participates in this round of communication with the server. Otherwise, it is judged as an untrustworthy node. When an untrustworthy node is judged as a trustworthy node during subsequent training, it can be uploaded.

[0110] Step 5: The client uploads the updated information to the server.

[0111] Embodiment 2:

[0112] Embodiment 2 of the present invention provides a distributed machine learning system based on gradient compression and quantization, including:

[0113] An information acquisition module, configured to acquire client gradient information by the server and cache it;

[0114] An information compression module, configured to perform data compression processing on client gradient information, wherein the client gradient information is initially compressed by using discretization quantization processing, and the initially compressed information is further compressed by using an encoding method;

[0115] An information distribution module, configured to distribute the compressed gradient information to the clients;

[0116] A local processing module, configured to adaptively calculate and select a gradient threshold by using an edge computing method after the client receives the compressed gradient information, so as to obtain updated gradient information;

[0117] An information feedback module, configured to upload the updated information by the client to the server.

[0118] Embodiment III:

[0119] Embodiment III of the present invention provides a medium, on which a program is stored, and when the program is executed by a processor, the steps in the distributed machine learning method based on gradient compression and quantization as described in Embodiment I of the present invention are implemented.

[0120] Embodiment IV:

[0121] Embodiment IV of the present invention provides a device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the distributed machine learning method based on gradient compression and quantization as described in Embodiment I of the present invention are implemented.

[0122] The steps involved in Embodiments II, III, and IV above correspond to those in Method Embodiment I. For specific implementation manners, reference may be made to the relevant description part of Embodiment I. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0123] Those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to be implemented. The present invention is not limited to any specific combination of hardware and software.

[0124] Although the specific implementation manners of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A distributed machine learning method based on gradient compression and quantization, characterized in that, It includes the following steps: The server obtains the client gradient information and caches it; Perform data compression processing on the client gradient information. Among them, use discretization quantization processing to perform preliminary compression on the client gradient information, and use coding methods to further compress the preliminarily compressed information; The specific steps for performing data compression processing on the client gradient information include: Iterate on the gradient information, and perform quantization processing on the gradient information before each round of iteration to achieve data compression; Use Huffman coding to further compress the information volume of the quantized gradient information to obtain the optimal variable-length coding; The specific steps for performing quantization processing on the gradient information are: Obtain the threshold interval to be quantized, evenly divide the threshold interval into P sub-intervals, and form P + 1 endpoint values; Approximate each element in the gradient information to the nearest endpoint; After each round of iteration of the gradient information, the server will re-record and statistically analyze the original uncompressed gradient distribution of each client to achieve real-time dynamic update of the quantization interval and optimize the compression effect; Distribute the compressed gradient information to the client; After receiving the compressed gradient information, the client uses the edge computing method to adaptively calculate and select the gradient threshold to obtain the updated gradient information; The client uploads the updated information to the server.

2. The distributed machine learning method based on gradient compression and quantization according to claim 1, characterized in that The specific steps for the server to obtain the client gradient information and cache it are: Before distributing the gradient information to the client, first statistically analyze its distribution according to the gradient information of the corresponding client in the previous round cached locally, and calculate the mean and variance to obtain the threshold interval to be quantized.

3. The distributed machine learning method based on gradient compression and quantization according to claim 1, characterized in that When distributing the compressed gradient information to the client, concatenate the encoded values of all elements in the gradient information bit by bit to form a bit string, and distribute the bit string and the gradient mapping code table to the corresponding client.

4. The distributed machine learning method based on gradient compression and quantization according to claim 1, characterized in that, The specific steps for using the edge computing method to adaptively calculate and select the gradient threshold are: The edge server first distributes the current global model to all subordinate device nodes; Subsequently, each device node calculates the local parameter gradient and uploads it to the edge side, and the edge server obtains the total gradient by weighted summation; Set the trust threshold through the adaptive algorithm, and use the trust threshold to evaluate the trust of all gradients to obtain a set of trustworthy communication nodes.

5. A distributed machine learning system based on gradient compression and quantization, characterized in that, It includes: An information acquisition module, configured to enable the server to obtain the client gradient information and cache it; An information compression module, configured to perform data compression processing on the client gradient information. Among them, use discretization quantization processing to perform preliminary compression on the client gradient information, and use coding methods to further compress the preliminarily compressed information; The specific steps for performing data compression processing on the client gradient information include: Iterate on the gradient information, and perform quantization processing on the gradient information before each round of iteration to achieve data compression; Use Huffman coding to further compress the information volume of the quantized gradient information to obtain the optimal variable-length coding; The specific steps for performing quantization processing on the gradient information are: Obtain the threshold interval to be quantized, evenly divide the threshold interval into P sub-intervals, and form P + 1 endpoint values; Approximate each element in the gradient information to the nearest endpoint; After each round of iteration of the gradient information, the server will re-record and statistically analyze the original uncompressed gradient distribution of each client to achieve real-time dynamic update of the quantization interval and optimize the compression effect; An information distribution module, configured to distribute the compressed gradient information to the client; A local processing module, configured to adaptively calculate and select the gradient threshold by using an edge computing method after the client receives the compressed gradient information to obtain the updated gradient information; An information feedback module, configured to upload the updated information from the client to the server.

6. A computer-readable storage medium, characterized in that, It stores multiple instructions, and the instructions are suitable for being loaded and executed by the processor of the terminal device to perform the distributed machine learning method based on gradient compression and quantization according to any one of claims 1-4.

7. A terminal device, characterized in that, It includes a processor and a computer-readable storage medium. The processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to perform the distributed machine learning method based on gradient compression and quantization according to any one of claims 1-4.

Images