Gradient compression method, device, equipment and storage medium

Abstract

The present application relates to artificial intelligence and provides a gradient compression method, apparatus, device and storage medium based on federated learning, that is, the gradient data whose gradient value in the gradient data to be transmitted is not less than a preset gradient threshold is used as the first gradient data; the gradient data other than the first gradient data in the gradient data to be transmitted is used as the second gradient data, and each gradient data in the second gradient data is compressed according to a 2-bit compression strategy or a 4-bit compression strategy; the first gradient data and the compressed second gradient data are uploaded to a server. The present invention screens out the first gradient data with high importance according to the size of the gradient value, uploads the gradient data with high importance in full, and compresses the second gradient data with low importance according to the corresponding compression strategy. While ensuring the accuracy of model modeling, the amount of data transmitted by the gradient is reduced, and the efficiency of gradient transmission is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a gradient compression method, apparatus, device, and computer-readable storage medium based on federated learning. Background Art

[0002] Federated learning is a learning method in which data is distributed across different entities. In a federated learning system, data is distributed across different clients. The federated server and clients initialize the same model (e.g., a neural network model) with the same initial model parameters. The client first trains on a local dataset to obtain the gradient of the model update (the gradient of the model parameters). Each client then sends the gradient to the federated server. The federated server collects the updated gradients from all client entities, averages them, and returns the global gradient to each client entity, allowing each client entity to train the model.

[0003] The emergence of federated learning offers a new solution to the dilemma between data sharing and privacy protection, and has therefore attracted increasing attention. However, during the federated learning modeling process, as the number of clients increases, the amount of gradient data required to be transmitted between the federated server and the clients also increases. Therefore, addressing the low gradient transmission efficiency caused by the large amount of gradient data transmitted during federated learning modeling has become a pressing technical challenge. Summary of the Invention

[0004] The main purpose of the present invention is to provide a gradient compression method, device, equipment and computer-readable storage medium based on federated learning, aiming to solve the technical problem of low gradient transmission efficiency caused by the large amount of gradient data transmitted during the federated learning modeling process.

[0005] To achieve the above objectives, the present invention provides a gradient compression method based on federated learning, comprising: obtaining gradient data to be transmitted, and using gradient data in the gradient data to be transmitted whose gradient values ​​are not less than a preset gradient threshold as first gradient data; using gradient data in the gradient data to be transmitted other than the first gradient data as second gradient data, and compressing each gradient data in the second gradient data according to a 2-bit compression strategy or a 4-bit compression strategy; and uploading the first gradient data and the compressed second gradient data to a server.

[0006] In addition, to achieve the above-mentioned objectives, the present invention also provides a gradient compression device based on federated learning, comprising: a gradient data acquisition module for acquiring gradient data to be transmitted, and using gradient data in the gradient data to be transmitted whose gradient values ​​are not less than a preset gradient threshold as first gradient data; a gradient data compression module for using gradient data in the gradient data to be transmitted other than the first gradient data as second gradient data, and compressing each gradient data in the second gradient data according to a 2-bit compression strategy or a 4-bit compression strategy; and a gradient data uploading module for uploading the first gradient data and the compressed second gradient data to a server.

[0007] Furthermore, to achieve the above-mentioned objectives, the present invention further provides a federated learning-based gradient compression device, comprising a processor, a memory, and a federated learning-based gradient compression program stored in the memory and executable by the processor. When the federated learning-based gradient compression program is executed by the processor, the steps of the above-mentioned federated learning-based gradient compression method are implemented.

[0008] In addition, to achieve the above-mentioned objectives, the present invention further provides a computer-readable storage medium, on which a gradient compression program based on federated learning is stored. When the gradient compression program based on federated learning is executed by a processor, the steps of the gradient compression method based on federated learning are implemented.

[0009] The present invention provides a gradient compression method based on federated learning, the method comprising: obtaining gradient data to be transmitted, and taking the gradient data whose gradient value is not less than a preset gradient threshold in the gradient data to be transmitted as the first gradient data; taking the gradient data other than the first gradient data in the gradient data to be transmitted as the second gradient data, and compressing each gradient data in the second gradient data according to a 2-bit compression strategy or a 4-bit compression strategy; uploading the first gradient data and the compressed second gradient data to a server. In the above manner, the present invention screens out the first gradient data with high importance according to the size of the gradient value, and then uploads the gradient data with high importance in full, and compresses the second gradient data with low importance according to a 2-bit compression strategy or a 4-bit compression strategy. Thus, while ensuring the accuracy of model modeling, the amount of data transmitted for gradients is reduced, the efficiency of gradient transmission is improved, and the technical problem of low gradient transmission efficiency caused by the large amount of gradient data transmitted in the existing federated learning modeling process is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Schematic diagram of the hardware structure of a gradient compression device based on federated learning involved in an embodiment of the present invention;

[0011] Figure 2 This is a flow chart of the first embodiment of the gradient compression method based on federated learning of the present invention;

[0012] Figure 3 This is a schematic diagram of the functional modules of the first embodiment of the gradient compression device based on federated learning of the present invention.

[0013] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0014] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0015] The federated learning-based gradient compression method involved in the embodiments of the present invention is mainly applied to a federated learning-based gradient compression device, which can be a device with display and processing functions such as a PC, a portable computer, or a mobile terminal.

[0016] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the hardware structure of a federated learning-based gradient compression device involved in an embodiment of the present invention. In this embodiment of the present invention, the federated learning-based gradient compression device may include a processor 1001 (e.g., a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components; the user interface 1003 may include a display and an input unit such as a keyboard; the network interface 1004 may optionally include a standard wired interface or a wireless interface (e.g., a Wi-Fi interface); and the memory 1005 may be a high-speed RAM memory or a non-volatile memory such as a disk storage device. The memory 1005 may also be a storage device independent of the processor 1001.

[0017] Those skilled in the art will understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation on the gradient compression device based on federated learning, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0018] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, and a gradient compression program based on federated learning.

[0019] exist Figure 1 In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the gradient compression program based on federated learning stored in the memory 1005 and execute the gradient compression method based on federated learning provided in the embodiment of the present invention.

[0020] An embodiment of the present invention provides a gradient compression method based on federated learning.

[0021] Reference Figure 2 , Figure 2 2 is a flow chart of the first embodiment of the gradient compression method based on federated learning of the present invention.

[0022] In this embodiment, the gradient compression method based on federated learning includes the following steps:

[0023] Step S10, obtaining gradient data to be transmitted, and taking gradient data with a gradient value not less than a preset gradient threshold in the gradient data to be transmitted as first gradient data;

[0024] In order to solve the problem of low gradient transmission efficiency caused by the large amount of gradient data transmitted during the federated learning modeling process due to uploading the complete gradient, this embodiment screens out important gradient data, i.e., the first gradient data, based on the gradient value of the gradient data, and uploads the important gradient data in full, thereby improving the accuracy of modeling, and compresses the remaining non-important gradient data, i.e., the second gradient data, according to the compression strategy, thereby reducing the amount of data transmitted for the gradient.

[0025] Specifically, the gradient data corresponding to different models, or the gradient data corresponding to different network layers in the same model, is used to obtain the gradient data corresponding to a network layer of a model as the gradient data to be transmitted. The gradient value of each gradient data in the data to be transmitted, that is, the absolute value of the gradient data, is calculated, and the gradient value of each gradient data is compared with a preset gradient threshold. The gradient threshold can be determined based on the gradient value corresponding to the important gradient data of the actual model. The gradient data in the gradient data to be transmitted whose gradient value is not less than the preset gradient threshold, and the important gradient data, are used as the first gradient data. In this way, the important gradient data and the unimportant gradient data are screened out by the gradient value of the gradient data.

[0026] Exemplarily, after the step of obtaining the gradient data to be transmitted and taking the gradient data whose gradient value is not less than a preset gradient threshold in the gradient data to be transmitted as the first gradient data, the method further includes:

[0027] When the data amount of the first gradient data exceeds the data amount threshold, sorting each gradient data in the gradient data to be transmitted according to the gradient value;

[0028] Gradient data of a target data amount is obtained from each of the sorted gradient data as updated first gradient data, wherein the target data amount is not greater than the data amount threshold.

[0029] In this embodiment, to prevent the first gradient data from being too large, after determining the first gradient data, the data volume of the data to be transmitted is compared with a pre-set data volume threshold. The data volume threshold is set based on actual transmission resources. When transmission resources are sufficient, the data volume threshold can be set higher, while when transmission resources are limited, the data volume threshold can be set lower. When the data volume of the first gradient exceeds the data volume threshold, the individual gradient data items in the first gradient data can be sorted in descending or ascending order of gradient value using a Top-K method. When sorting the individual gradient data items in the first gradient data in descending order of gradient value (i.e., from largest to smallest gradient value) or in ascending order of gradient value (i.e., from smallest to largest gradient value), based on the target data volume, the k gradient data items with the largest gradient values ​​are obtained from the sorted individual gradient data items. The data volume of the k gradient data items with the largest gradient values ​​is no greater than the target data volume. Thus, the first gradient data that needs to be fully uploaded is updated, preventing the excessive volume of the first gradient data from increasing modeling overhead. When the data volume of the first gradient data is less than the data volume threshold, there is no need to filter the gradient data, and the first gradient data can be directly uploaded in its entirety.

[0030] Step S20: taking the gradient data other than the first gradient data in the gradient data to be transmitted as second gradient data, and compressing each gradient data in the second gradient data according to a 2-bit compression strategy or a 4-bit compression strategy;

[0031] In this embodiment, the gradient data other than the first gradient data in the gradient data to be transmitted is used as the second gradient data, i.e., non-significant gradient data. According to the compression strategy, each gradient data in the first gradient data is compressed to 2 bits (bits) or 4 bits. The 2-bit compression strategy includes 3 thresholds, including 0, a set positive threshold, and its opposite. Gradient data less than the set threshold is compressed to 0, and gradient data not less than the threshold is compressed to the threshold. The 4-bit compression strategy includes 15 thresholds, including 0, 7 set positive thresholds, and their opposites (e.g., -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7). Each gradient data is compressed to the corresponding compression threshold. Thus, some accuracy of the non-significant gradient data is sacrificed to reduce communication overhead.

[0032] Exemplarily, the step of compressing each gradient data in the second gradient data according to the 2-bit compression strategy or the 4-bit compression strategy specifically includes:

[0033] When the variance of the second gradient data is not less than a preset variance threshold, compressing each gradient data in the second gradient data according to the 4-bit compression strategy;

[0034] When the variance of the second gradient data is less than the variance threshold, each gradient data in the second gradient data is compressed according to the 2-bit compression strategy.

[0035] In this embodiment, when the variance of the second gradient data is less than the variance threshold, it means that the volatility of the second gradient data is small, and a 2-bit compression strategy can be used to compress each gradient data in the second gradient data. The 2 bits are divided into a sign bit and a value bit, with the first bit being the sign bit and the other bit being the value bit. In the sign bit, 0 represents a positive number and 1 represents a negative number. When the variance of the second gradient data is not less than the preset variance threshold, it means that the volatility of the second gradient data is large. However, the 2-bit compression strategy only includes 3 thresholds, compressing the second gradient data to 0 or a set threshold greatly reduces the accuracy of the gradient data. Therefore, when the variance of the second gradient data is not less than the variance threshold, a 4-bit compression strategy (including 15 thresholds) is used to compress each gradient data in the second gradient data. The 4 bits are divided into a sign bit and a value bit, with the first bit being the sign bit and the rest being the value bits. In the sign bit, 0 represents a positive number and 1 represents a negative number, and the 3 value bits store the compressed value. These 4 bits are the compressed value after gradient compression.

[0036] Exemplarily, the step of compressing each gradient data in the second gradient data according to the 2-bit compression strategy includes:

[0037] The average gradient value of the second gradient data is used as a first compression threshold, and each gradient data in the second gradient data is compressed to 0, the first compression threshold, or the inverse of the first compression threshold, thereby completing the compression of each gradient data in the second gradient data.

[0038] In this embodiment, if the variance of the second gradient data is less than the variance threshold, it indicates that the individual gradient data in the second gradient data are relatively close. Therefore, the average gradient value of the second gradient data can be used as the compression threshold for the 2-bit compression strategy, i.e., the first compression threshold. First, it is determined whether the gradient data is positive or negative. If it is positive, the gradient data is compared with the corresponding set positive threshold (the first compression threshold). If it is less than the first compression threshold, the gradient data is compressed to 0. If it is not less than the first compression threshold, the gradient data is compressed to the first compression threshold. If it is negative, the gradient data is compared with the opposite of the set positive threshold (the opposite of the first compression threshold). If it is less than the opposite of the first compression threshold, the gradient data is compressed to the opposite of the first compression threshold. If it is not less than the opposite of the first compression threshold, the gradient data is compressed to 0.

[0039] Exemplarily, the step of compressing each gradient data in the second gradient data according to the 4-bit compression strategy specifically includes:

[0040] determining a compression threshold group corresponding to the second gradient data according to an average gradient value of the second gradient data;

[0041] Each gradient data in the second gradient data is compressed to a corresponding compression threshold in the compression threshold group, thereby completing the compression of each gradient data in the second gradient data.

[0042] In this embodiment, a set of compression thresholds corresponding to a 4-bit compression strategy is determined before and after the average gradient value of the second gradient data, i.e., 15 compression thresholds. A gradient data point in the second gradient data is compared with each of the 15 compression thresholds, and the compression threshold closest to the gradient data point is determined from among the 15 compression thresholds. The compression threshold closest to the gradient data point can be the compression threshold with the smallest difference from the gradient data point, or can be greater than the minimum value among the compression thresholds for the gradient data point, or can be less than the maximum value among the compression thresholds for the gradient data point. This process is repeated in this manner, thereby completing the compression of each gradient data point in the second gradient data point.

[0043] Exemplarily, the step of determining the compression threshold value group corresponding to the second gradient data according to the average gradient value of the second gradient data specifically includes:

[0044] determining, according to a preset difference value and the average gradient value, respective second compression thresholds corresponding to the second gradient data;

[0045] The compression threshold group is generated according to each second compression threshold and the opposite number corresponding to each second compression threshold.

[0046] In this embodiment, the variance of the second gradient data is not less than the variance threshold, which means that the individual gradient data in the second gradient data are significantly different. Based on the difference between the average gradient value of the second gradient data and the minimum gradient value in the second gradient data, and the difference between the average gradient value of the second gradient data and the maximum gradient value in the second gradient data, a second compression threshold corresponding to the 4-bit compression strategy (i.e., 7 positive compression thresholds and their opposites, and 15 compression thresholds consisting of 0, the 7 positive compression thresholds, and their opposites) can be determined as the compression threshold group for the second gradient data.

[0047] Exemplarily, the step of compressing each gradient data in the second gradient data to a corresponding compression threshold in the compression threshold group, and completing the compression of each gradient data in the second gradient data, includes:

[0048] using one gradient data in the second gradient data as target gradient data;

[0049] Comparing the target gradient data with the second compression thresholds arranged in sequence in the compression threshold group in sequence, and determining a target compression threshold corresponding to the target gradient data in each of the second compression thresholds;

[0050] The target gradient data is compressed to the target compression threshold, and the next gradient data in the second gradient data is obtained as the target gradient data. The target gradient data is then sequentially compared with each of the second compression thresholds arranged in sequence in the compression threshold group, and a target compression threshold corresponding to the target gradient data is determined in each of the second compression thresholds until compression of each gradient data in the second gradient data is completed.

[0051] In this embodiment, the gradient data in the second gradient data is traversed. First, the positive and negative values ​​of a gradient data are determined. Then, the gradient data is compared with the second compression threshold values ​​from small to large respectively. Then, the gradient data is compressed to the smaller value. The compressed 4-bit data is the compressed value corresponding to the gradient data.

[0052] The compression process of each gradient data in the second gradient data is completed as follows:

[0053] Obtain one gradient data from the second gradient data as the target gradient data, and compare the target gradient data with each of the second compression thresholds arranged in order (from large to small or from small to large) in the compression threshold group. For example, if the target gradient data is A, and the second compression thresholds are -X7, -X6, -X5, -X4, -X3, -X2, -X1, 0, X1, X2, X3, X4, X5, X6, X7 respectively, where X1 - X7 increase in sequence. If A > X3 and A < X4, then compress A to X3.

[0054] Obtain each gradient data in the second gradient data as the target gradient data in sequence, and repeat the above steps until the compression of each gradient data in the second gradient data is completed.

[0055] It can be understood that the first compression threshold and the second compression threshold can be preset by the user according to actual needs, or can be calculated and determined by the system according to the average gradient value of the second gradient data (and can further combine the differences between the minimum gradient value and the maximum gradient value of the second gradient data and the average gradient value).

[0056] Step S30: Upload the first gradient data and the compressed second gradient data to the server.

[0057] In this embodiment, the first gradient data with greater importance is uploaded to the server completely, and then the second gradient data is compressed by gradient and uploaded to the server. Thus, a gradient data that occupies a large memory is converted into a gradient data that occupies a small memory, thereby reducing the communication overhead occupied during the transmission of each gradient data.

[0058] In addition, an embodiment of the present invention further provides a gradient compression device based on federated learning.

[0059] Refer to Figure 3 , Figure 3 which is a schematic diagram of the functional modules of the first embodiment of the gradient compression device based on federated learning of the present invention.

[0060] In this embodiment, the gradient compression device based on federated learning includes:

[0061] A gradient data acquisition module, configured to acquire the gradient data to be transmitted, and use the gradient data in the gradient data to be transmitted whose gradient value is not less than the preset gradient threshold as the first gradient data;

[0062] A gradient data compression module, configured to use the gradient data other than the first gradient data in the gradient data to be transmitted as the second gradient data, and compress each gradient data in the second gradient data according to the 2-bit compression strategy or the 4-bit compression strategy; <000014​The gradient data uploading module is used to upload the first gradient data and the compressed second gradient data to the server.

[0064] Furthermore, the gradient data compression module specifically includes:

[0065] a 4-bit compression unit, configured to compress each gradient data in the second gradient data according to the 4-bit compression strategy when the variance of the second gradient data is not less than a preset variance threshold;

[0066] A 2-bit compression unit is configured to compress each gradient data in the second gradient data according to the 2-bit compression strategy when the variance of the second gradient data is less than the variance threshold.

[0067] Furthermore, the 2-bit compression unit specifically includes:

[0068] The 2-bit compression subunit is configured to use the average gradient value of the second gradient data as a first compression threshold, and compress each gradient data in the second gradient data to 0, the first compression threshold, or the inverse of the first compression threshold, thereby completing compression of each gradient data in the second gradient data.

[0069] Furthermore, the 4-bit compression unit specifically includes:

[0070] a threshold group determining subunit, configured to determine a compression threshold group corresponding to the second gradient data according to an average gradient value of the second gradient data;

[0071] The gradient data compression subunit is configured to compress each gradient data in the second gradient data into a corresponding compression threshold in the compression threshold group, thereby completing the compression of each gradient data in the second gradient data.

[0072] Furthermore, the threshold group determination subunit is further configured to:

[0073] determining, according to a preset difference value and the average gradient value, respective second compression thresholds corresponding to the second gradient data;

[0074] The compression threshold group is generated according to each second compression threshold and the opposite number corresponding to each second compression threshold.

[0075] Furthermore, the gradient data compression subunit is further configured to:

[0076] using one gradient data in the second gradient data as target gradient data;

[0077] Comparing the target gradient data with the second compression thresholds arranged in sequence in the compression threshold group in sequence, and determining a target compression threshold corresponding to the target gradient data in each of the second compression thresholds;

[0078] The target gradient data is compressed to the target compression threshold, and the next gradient data in the second gradient data is obtained as the target gradient data. The target gradient data is then sequentially compared with each of the second compression thresholds arranged in sequence in the compression threshold group, and a target compression threshold corresponding to the target gradient data is determined in each of the second compression thresholds until compression of each gradient data in the second gradient data is completed.

[0079] Furthermore, the gradient compression device based on federated learning also includes:

[0080] a gradient data sorting module, configured to sort each gradient data in the gradient data to be transmitted according to a gradient value when the data amount of the first gradient data exceeds the data amount threshold;

[0081] The gradient data updating module is configured to obtain gradient data of a target data amount from each of the sorted gradient data as updated first gradient data, wherein the target data amount is not greater than the data amount threshold.

[0082] Among them, each module in the above-mentioned gradient compression device based on federated learning corresponds to each step in the above-mentioned embodiment of the gradient compression method based on federated learning, and their functions and implementation processes are not further described here.

[0083] In addition, an embodiment of the present invention also provides a computer-readable storage medium.

[0084] The computer-readable storage medium of the present invention stores a gradient compression program based on federated learning, wherein when the gradient compression program based on federated learning is executed by a processor, the steps of the gradient compression method based on federated learning are implemented.

[0085] Among them, the method implemented when the gradient compression program based on federated learning is executed can refer to the various embodiments of the gradient compression method based on federated learning of the present invention, and will not be repeated here.

[0086] The present invention provides a gradient compression method, apparatus, device and computer-readable storage medium based on federated learning, the method comprising: obtaining gradient data to be transmitted, and taking the gradient data whose gradient value is not less than a preset gradient threshold in the gradient data to be transmitted as the first gradient data; taking the gradient data other than the first gradient data in the gradient data to be transmitted as the second gradient data, and compressing each gradient data in the second gradient data according to a 2-bit compression strategy or a 4-bit compression strategy; uploading the first gradient data and the compressed second gradient data to a server. In the above manner, the present invention screens out the first gradient data with high importance according to the size of the gradient value, and then uploads the gradient data with high importance in full, and compresses the second gradient data with low importance according to a 2-bit compression strategy or a 4-bit compression strategy. Thus, while ensuring the accuracy of model modeling, the amount of data transmitted for gradients is reduced, the efficiency of gradient transmission is improved, and the technical problem of low gradient transmission efficiency caused by the large amount of gradient data transmitted in the existing federated learning modeling process is solved.

[0087] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0088] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0089] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0090] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A gradient compression method based on federated learning, characterized in that: The gradient compression method based on federated learning includes the following steps: Obtaining gradient data to be transmitted from the client, and taking gradient data whose gradient value is not less than a preset gradient threshold in the gradient data to be transmitted as first gradient data; using the gradient data other than the first gradient data in the gradient data to be transmitted as second gradient data; When the variance of the second gradient data is not less than a preset variance threshold, determining, based on the client using a 4-bit compression strategy and an average gradient value of the second gradient data, a compression threshold group corresponding to the second gradient data; When the variance of the second gradient data is not less than the variance threshold, it indicates that the second gradient data has a large volatility, and the 4-bit compression strategy is used to compress the second gradient data with a large volatility. The 4-bit compression strategy includes 0, 7 set positive thresholds and their opposites, so as to avoid reducing the compression accuracy of the gradient data based on multiple compression thresholds. Based on the client compressing each gradient data in the second gradient data to a corresponding compression threshold in the compression threshold group, completing compression of each gradient data in the second gradient data; When the variance of the second gradient data is less than the variance threshold, compressing each gradient data in the second gradient data using a 2-bit compression strategy based on the client; When the variance of the second gradient data is less than the variance threshold, it indicates that the volatility of the second gradient data is small, and each gradient data in the second gradient data is compressed using the 2-bit compression strategy, where the 2-bit compression strategy includes 0, a set positive threshold, and its opposite, thereby sacrificing some accuracy of non-important gradient data to reduce communication overhead. The client uploads the first gradient data and the compressed second gradient data to the server, so as to convert the second gradient data occupying a larger memory into compressed gradient data occupying a smaller memory, thereby reducing communication overhead when the client transmits the second gradient data; Wherein, after obtaining the gradient data to be transmitted from the client and taking the gradient data with a gradient value not less than a preset gradient threshold in the gradient data to be transmitted as the first gradient data, the method further includes: When the amount of the first gradient data exceeds a data amount threshold, sorting the gradient data in the first gradient data in descending or ascending order of gradient values ​​according to a Top-K method based on the client; Based on the client, k gradient data with the largest gradient values ​​are obtained from the sorted gradient data according to the target data volume as the updated first gradient data, wherein the target data volume is not greater than the data volume threshold, and the data volume of the k gradient data with the largest gradient values ​​is not greater than the target data volume.

2. The gradient compression method based on federated learning according to claim 1, characterized in that The step of compressing each gradient data in the second gradient data using the 2-bit compression strategy based on the client includes: The client uses the 2-bit compression strategy to use an average gradient value of the second gradient data as a first compression threshold, and the client compresses each gradient data in the second gradient data to 0, the first compression threshold, or the inverse of the first compression threshold, thereby completing compression of each gradient data in the second gradient data.

3. The gradient compression method based on federated learning according to claim 1, characterized in that The step of determining, based on the average gradient value of the second gradient data by the client, a compression threshold group corresponding to the second gradient data specifically includes: Determining, based on the client using a preset difference value and the average gradient value, respective second compression thresholds corresponding to the second gradient data; The compression threshold group is generated based on the client using each second compression threshold and the opposite number corresponding to each second compression threshold.

4. The gradient compression method based on federated learning according to claim 3, characterized in that The step of compressing each gradient data in the second gradient data to a corresponding compression threshold in the compression threshold group based on the client comprises: Based on the client taking one gradient data in the second gradient data as target gradient data; Comparing the target gradient data with the second compression thresholds arranged in order in the compression threshold group in sequence based on the client, and determining a target compression threshold corresponding to the target gradient data in each of the second compression thresholds; The target gradient data is compressed to the target compression threshold based on the client, and next gradient data in the second gradient data is obtained as the target gradient data, and the following steps are performed: the target gradient data is sequentially compared with each second compression threshold arranged in sequence in the compression threshold group, and a target compression threshold corresponding to the target gradient data is determined in each second compression threshold until compression of each gradient data in the second gradient data is completed.

5. A gradient compression device based on federated learning, characterized in that: The gradient compression device based on federated learning includes: a gradient data acquisition module, configured to acquire gradient data to be transmitted from a client, and use gradient data having a gradient value not less than a preset gradient threshold value in the gradient data to be transmitted as first gradient data; a gradient data compression module, configured to use the gradient data other than the first gradient data in the gradient data to be transmitted as second gradient data, and compress each gradient data in the second gradient data using a 2-bit compression strategy or a 4-bit compression strategy according to the client; A gradient data uploading module, configured to upload the first gradient data and the compressed second gradient data to a server based on the client; Wherein, the gradient data compression module is further used for: When a variance of the second gradient data is not less than a preset variance threshold, determining, based on the client using the 4-bit compression strategy and an average gradient value of the second gradient data, a compression threshold group corresponding to the second gradient data; When the variance of the second gradient data is not less than the variance threshold, it indicates that the second gradient data has a large volatility, and the 4-bit compression strategy is used to compress the second gradient data with a large volatility. The 4-bit compression strategy includes 0, 7 set positive thresholds and their opposites, so as to avoid reducing the compression accuracy of the gradient data based on multiple compression thresholds. Based on the client compressing each gradient data in the second gradient data to a corresponding compression threshold in the compression threshold group, completing compression of each gradient data in the second gradient data; When the variance of the second gradient data is less than the variance threshold, compressing each gradient data in the second gradient data using the 2-bit compression strategy based on the client; When the variance of the second gradient data is less than the variance threshold, it indicates that the volatility of the second gradient data is small, and each gradient data in the second gradient data is compressed using the 2-bit compression strategy, where the 2-bit compression strategy includes 0, a set positive threshold, and its opposite, thereby sacrificing some accuracy of non-important gradient data to reduce communication overhead. Wherein, the gradient data acquisition module is further used for: When the amount of the first gradient data exceeds a data amount threshold, sorting the gradient data in the first gradient data in descending or ascending order of gradient values ​​according to a Top-K method based on the client; Based on the client, k gradient data with the largest gradient values ​​are obtained from the sorted gradient data according to the target data volume as the updated first gradient data, wherein the target data volume is not greater than the data volume threshold, and the data volume of the k gradient data with the largest gradient values ​​is not greater than the target data volume.

6. A gradient compression device based on federated learning, characterized in that: The federated learning-based gradient compression device includes a processor, a memory, and a federated learning-based gradient compression program stored in the memory and executable by the processor. When the federated learning-based gradient compression program is executed by the processor, the steps of the federated learning-based gradient compression method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a gradient compression program based on federated learning, wherein when the gradient compression program based on federated learning is executed by a processor, the steps of the gradient compression method based on federated learning are implemented as described in any one of claims 1 to 4.

Images