Gradient compressor and gradient compression method and device based on information consistency driving

Abstract

The application provides a gradient compressor and a gradient compression method and device based on information consistency driving, and relates to the technical field of distributed learning.The application is used for gradient compression between a plurality of clients and a server in a federated learning system, an inter-information constraint is taken as a core target of gradient compression through an inter-information constraint modeling module, and an optimal compression operator is obtained by minimizing a Lagrange objective function;a multi-scale sparse representation module performs multi-scale sparse processing on the original gradient by using wavelet transform, so that redundant data is reduced, a sparse gradient representation is obtained, and parameters are dynamically adjusted in combination with the inter-information constraint and the optimal compression operator;and a Bayesian adaptive quantization module estimates a probability distribution of the sparse gradient representation to dynamically adjust a quantization bit width, and generates compressed gradients to be transmitted back to the server.The application significantly reduces communication overhead in distributed deep learning through multi-level optimization, while maximum gradient information is retained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed learning and data compression, and particularly relates to a gradient compressor and a gradient compression method and device based on information consistency driving. BACKGROUND

[0002] Distributed deep learning significantly improves the efficiency and privacy protection capability of model training by distributing computing tasks to multiple client devices, and is widely used in federated learning, edge computing and other scenarios. However, in the process of distributed training, high-dimensional feature gradients need to be frequently transmitted between the client and the server, and the communication overhead becomes the main bottleneck restricting the system performance, especially in low-bandwidth or high-latency network environments. For example, when training a deep neural network, the gradient tensor usually contains millions or even tens of millions of floating-point parameters, resulting in a large amount of data in a single communication, increasing the delay and energy consumption. In addition, client devices such as mobile devices or Internet of Things nodes often have limited computing and storage resources, further exacerbating the need for efficient communication.

[0003] In order to solve this problem, existing technologies propose a series of gradient compression methods to reduce data transmission. These methods include gradient sparsification, quantization, low-rank decomposition and differential coding. Gradient sparsification methods such as Top-k sparsification retain the top k% elements by sorting the gradient absolute value, or random sparsification discards elements with a fixed probability to reduce dimensionality; gradient quantization techniques such as SignSGD only transmit gradient signs and cooperate with scaling factors to approximate the original value, and QSGD uses a fixed bit width to discretize the gradient into a finite value set; low-rank decomposition methods such as PowerSGD approximate the gradient tensor as the product of low-dimensional matrices through singular value decomposition, thereby reducing the communication volume; differential coding methods such as EF-SGD utilize the temporal correlation of gradients, only transmitting the difference between the current round and the previous round, and combining an error feedback mechanism to compensate for compression loss.

[0004] Although these techniques alleviate the communication bottleneck to some extent, there are still significant limitations in balancing high compression rate and model performance. For example, Top-k or random sparsification methods lack adaptability, which may discard key small-amplitude gradients or introduce excessive noise, resulting in decreased convergence speed or impaired performance; fixed-bit-width quantization cannot adapt to changes in gradient distribution, and low-bit quantization causes information loss, especially when high precision is required during early training, while high-bit quantization is insufficient in compression rate; low-rank decomposition has high computational overhead and is not suitable for resource-constrained devices, and the low-rank approximation may not capture the complex structure of the gradient, affecting convergence. In addition, existing methods lack systematic multi-level integration, and the trade-off between compression rate and information preservation is not ideal, making it difficult to meet the needs of diverse distributed learning scenarios.

[0005] Therefore, the present application is proposed. Summary of the Invention

[0006] This invention aims to provide a gradient compressor, gradient compression method, and device based on information consistency to solve the problem of excessive communication overhead in distributed deep learning in the prior art. Through multi-level optimization, it significantly reduces the amount of communication data between the client and the server, while preserving gradient information to the maximum extent, ensuring that the performance of distributed training is comparable to that of centralized training.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A gradient compressor based on information consistency is proposed for gradient compression between several clients and servers in a federated learning system. It includes a mutual information constraint modeling module, a multi-scale sparse representation module, and a Bayesian adaptive quantization module.

[0009] The mutual information constraint modeling module is used to receive the original gradients generated by the client's model during training, and uses mutual information constraints as the core objective of gradient compression, obtaining the optimal compression operator by minimizing the Lagrange objective function;

[0010] The multi-scale sparse representation module is used to perform multi-scale sparsification processing on the original gradient using wavelet transform to reduce redundant data and obtain a sparse gradient representation. It is then evaluated in conjunction with the mutual information constraint and the optimal compression operator to dynamically adjust the sparsification parameters.

[0011] The Bayesian adaptive quantization module is used to estimate the probability distribution of the sparse gradient representation using the Bayesian method, dynamically adjust the quantization bit width according to the probability distribution of the gradient, generate a compressed gradient, transmit it back to the server, decompress and aggregate it, and then send it to the client for client model updates.

[0012] Preferably, in the mutual information constraint modeling module, the mutual information is the difference between entropy and conditional entropy, and the mutual information constraint is taken as the core objective of compression, expressed as:

[0013] ;

[0014] ;

[0015] in, Represents the original gradient; This represents the reconstructed gradient, i.e., the approximate gradient after decompression; Compression indicates the target data transmitted between the client and the server; Indicates being bound by;

[0016] represents a compression operator, i.e. a mapping function from the original gradient to the compressed representation ; represents an expectation operator, i.e. taking the average over the data distribution; represents the original gradient and the reconstructed gradient ; represents the L2 norm squared;

[0017] represents the mutual information, i.e. the information about the original gradient that is preserved in the compressed representation; represents the information preservation threshold; represents the entropy, i.e. the overall uncertainty of the original gradient; represents the conditional entropy, i.e. the uncertainty of the gradient that still exists given the compressed representation.

[0018] Preferably, the constrained optimization problem is transformed into an unconstrained form by the Lagrangian objective function and a balancing term is introduced to flexibly adjust the importance of the reconstruction error and the information preservation, which is expressed as:

[0019] ;

[0020] wherein, represents the Lagrangian objective function; represents the Lagrangian multiplier to balance the error term and the information term; represents the expected reconstruction error term; represents the information difference.

[0021] Preferably, the expression of the optimal compression operator is:

[0022] ;

[0023] wherein, represents the optimal compression operator; represents the minimization operation; represents the Lagrangian objective function; represents the compression operator.

[0024] Preferably, when the original gradient is multi-scale sparsified by the wavelet transform, the original gradient is decomposed into low-frequency and high-frequency parts and sparsified in the multi-scale space, which is expressed as:

[0025] ;

[0026] wherein, represents the original gradient; represents the low-frequency coefficient, i.e. the global trend; denotes a scale function, used to represent low frequency part; denotes a high frequency coefficient, i.e. local detail; denotes a wavelet base function, used to represent high frequency part; , denote scale, position index respectively;

[0027] Meanwhile, an energy preserving constraint is introduced to ensure that low frequency part and part of high frequency components can preserve most of original signal energy, expressed as:

[0028] ;

[0029] wherein, denotes a reserved high frequency set, denotes an energy preserving ratio; denotes L2 norm;

[0030] In the process of sparsification, high frequency components are selectively reserved, expressed as:

[0031] ;

[0032] wherein, denotes a sparsified high frequency coefficient;

[0033] so as to obtain a sparsified gradient representation :

[0034] ;

[0035] wherein, denotes a reserved low frequency coefficient set; denotes a sparsified high frequency set.

[0036] Preferably, the evaluation is combined with the optimal compression operator to dynamically adjust the sparsification parameter, specifically:

[0037] calculate mutual information of the original gradient and the sparsified gradient representation;

[0038] combine mutual information constraint and Lagrange objective function to solve the optimal compression operator;

[0039] when the solved mutual information and the optimal compression operator do not meet the set target, adjust the threshold of wavelet transform or the number of decomposition layers.

[0040] Preferably, the probability distribution of the gradient is obtained through probability modeling, expressed as:

[0041] ;

[0042] wherein, representing the input gradient value probability distribution function; representing the number of components of the Gaussian mixture model; representing the weight of the th Gaussian component, satisfying ; representing the mean of the th component, i.e. the center position; representing the variance of the th component, i.e. the spread of the distribution; representing the Gaussian density function; representing the probability density of the gradient value mean , variance .

[0043] Preferably, when dynamically adjusting the quantization bit width according to the gradient distribution, the optimal bit width is selected by maximum a posteriori inference to find the number of bits that can maximize the posterior probability, so as to ensure that the error in the quantization process is minimized, and the expression is:

[0044] ;

[0045] wherein, optimal bit width, i.e. the number of bits finally selected; representing the bit width value that maximizes the posterior probability; b represents the bit width; representing the th gradient value in the set of gradient representation samples after sparsification; representing the posterior probability of the bit width b under the given set of gradient samples ;

[0046] Then, the scaling factor is calculated based on the standard deviation of the gradient and the bit width size to map the floating point number to the integer space, and the expression is:

[0047] ;

[0048] wherein, scaling factor, used to adjust the mapping ratio of the floating point number to the integer domain; representing the standard deviation of the gradient distribution;

[0049] Finally, each gradient value is mapped to an integer in the quantization process, and the error is reduced by rounding, and the expression is:

[0050] ;

[0051] wherein, quantized integer value of the gradient value , i.e. the compressed gradient; denotes a gradient mean value, used for centering processing; denotes a rounding operation.

[0052] The application further provides a gradient compression method based on information consistency driving, comprising:

[0053] receiving an original gradient generated by a model of a client in a training process;

[0054] constructing a mutual information constraint based on the original gradient and a reconstructed gradient, and obtaining an optimal compression operator by minimizing a Lagrange objective function based on the mutual information constraint;

[0055] based on the original gradient, performing multi-scale sparse processing on the original gradient by using wavelet transform to reduce redundant data, to obtain a sparse gradient representation, and combining the mutual information constraint and the optimal compression operator to evaluate to dynamically adjust a sparse parameter;

[0056] estimating a probability distribution of the sparse gradient representation by a Bayesian method, dynamically adjusting a quantization bit width according to the probability distribution of the gradient, generating a compressed gradient, transmitting the compressed gradient back to a server end, decompressing and aggregating the compressed gradient, and then issuing the compressed gradient to the client end for model updating of the client end.

[0057] The application further provides a gradient compression device based on information consistency driving, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the gradient compression method based on information consistency driving.

[0058] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor of a device where the computer readable storage medium is located to implement the gradient compression method based on information consistency driving.

[0059] Compared with the prior art, the application has the following beneficial effects:

[0060] The application comprises a mutual information constraint modeling module, a multi-scale sparse representation module, and a Bayesian adaptive quantization module, and significantly reduces communication overhead in distributed deep learning through multi-level optimization, while maximizing the preservation of gradient information.

[0061] Specifically, the mutual information constraint modeling module ensures that the compression result is not only numerically approximate, but also maintains strong correlation with the original gradient at the information level, avoiding the risk of information loss in traditional compression methods. The multi-scale sparse representation module uses wavelet decomposition to sparsify at different scales, effectively capturing the global trend and local details of the gradient, significantly reducing the communication volume. The Bayesian adaptive quantization module automatically selects the optimal bit width according to the dynamic characteristics of the gradient distribution, using a higher bit width in the early training stage to ensure that information is not lost, and automatically reducing the bit width in the later training stage to improve the compression ratio.

[0062] The gradient compressor based on information consistency driving provided by the present application has high efficiency, universality and robustness, and can be widely applied to federated learning, edge computing and other scenarios. In particular, the present application significantly reduces the communication overhead through multi-level optimization, while ensuring the integrity of gradient information, solving the problem of balancing compression rate and model performance in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0064] Figure 1 A framework diagram of a gradient compressor based on information consistency driving is provided for embodiment one.

[0065] Figure 2 A flowchart of a gradient compression method based on information consistency driving is provided for embodiment two.

[0066] The present application will be further described in detail below in combination with the drawings and specific embodiments. DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0068] Example 1

[0069] like Figure 1 As shown, an information consistency-driven gradient compressor is used for gradient compression between several clients and servers in a federated learning system. It is characterized by including: a mutual information constraint modeling module, a multi-scale sparse representation module, and a Bayesian adaptive quantization module.

[0070] like Figure 1 As shown, in federated learning, each client (client1, ... clientn) uses its local private data to perform the training process on the client model (forward propagation to calculate the loss, back propagation to solve the parameter gradient), generating the original gradient (i.e. the gradient of the model parameters, reflecting the direction and magnitude of the parameter update), which is then compressed by the gradient compressor and uploaded to the server.

[0071] The gradient compressor framework consists of three modules: mutual information constraint modeling, multi-scale sparse representation, and Bayesian adaptive quantization. First, the mutual information constraint modeling module defines the overall optimization objective of the compression process from an information theory perspective, ensuring that the compression result retains key information. Then, the multi-scale sparse representation module reduces redundant data in the frequency and spatial domains, making transmission more efficient. Next, the Bayesian adaptive quantization module dynamically models the gradient distribution and adaptively selects the optimal bit width, achieving a balance between compression ratio and accuracy. These three modules simultaneously optimize communication efficiency and training accuracy.

[0072] In the compression task, simply minimizing numerical errors is not enough to guarantee the effectiveness of model training. Even if two gradients are very close in value, if the amount of information is insufficient, the update direction of the model may be biased. Therefore, we introduce mutual information constraints to maintain information as the core goal of compression. The mutual information constraint modeling module is used to receive the original gradient generated by the client's model during the training process, and take the mutual information constraint as the core goal of gradient compression, and obtain the optimal compression operator by minimizing the Lagrangian objective function.

[0073] Specifically, in the mutual information constraint modeling module, the mutual information is the difference between entropy and conditional entropy, and the mutual information constraint is taken as the core goal of compression, which is used to measure the information correlation between the original gradient and the compressed representation. The higher the mutual information, the less the information loss in the compression process. Its expression is:

[0074] ;

[0075] ;

[0076] Where, represents the original gradient; represents the reconstructed gradient, i.e. the approximate gradient after decompression; represents the compressed representation, which is the target of transmission between the client and the server; represents the constraint;

[0077] represents the compression operator, i.e. the mapping function from the original gradient to the compressed representation ; represents the expectation operator, i.e. taking the average value on the data distribution; represents the L2 norm square of the original gradient and the reconstructed gradient , which is used to measure the reconstruction error; represents the L2 norm;

[0078] represents the mutual information, i.e. the information about the original gradient preserved in the compressed representation; represents the information retention threshold; represents the entropy, i.e. the overall uncertainty of the original gradient; represents the conditional entropy, i.e. the uncertainty of the gradient still existing after the given compressed representation.

[0079] Mutual information reveals the ability of compressed representation to reduce uncertainty. In other words, if the compression result can greatly reduce the uncertainty of the original gradient, it means that the information is better preserved. This definition not only facilitates theoretical analysis, but also provides a measurement standard for subsequent quantization strategies.

[0080] In this way, the compressed representation not only guarantees numerical approximation, but also maintains strong correlation with the original gradient at the information level. This module provides a theoretical basis for subsequent sparsification and quantization, making the entire compression process have an information-theoretic interpretation.

[0081] Since the constrained optimization problem is difficult to solve directly, the constrained optimization problem is converted into an unconstrained form by the Lagrange objective function, and a balance term is introduced to flexibly adjust the importance of reconstruction error and information preservation, which is expressed as:

[0082] ;

[0083] wherein, denotes the Lagrange objective function; denotes the Lagrange multiplier, used to balance the error term and the information term; denotes the expected reconstruction error term; denotes the information difference.

[0084] This transformation not only simplifies the optimization process, but also adapts to the needs of different training stages, achieving dynamic adjustment.

[0085] Finally, the optimal compression operator is obtained by minimizing the Lagrange objective function, and the expression of the optimal compression operator is:

[0086] ;

[0087] wherein, denotes the optimal compression operator; denotes the minimization operation; denotes the Lagrange objective function; denotes the compression operator.

[0088] This compression operator can balance between numerical error and information preservation, ensuring that the compression process is both efficient and reliable.

[0089] The mutual information constraint provides a theoretical framework for the compression task, and the multi-scale sparse representation is the first step to realize compression. Gradient signals often contain both global trends and local details, and if only compressed at a single scale, key information may be lost. Therefore, a multi-scale sparse representation module is constructed.

[0090] The multi-scale sparse representation module is used to perform multi-scale sparse processing on the original gradient based on the original gradient using wavelet transform to reduce redundant data and obtain a sparse gradient representation, and the sparse gradient representation is evaluated in combination with the mutual information constraint and the optimal compression operator to dynamically adjust the sparsification parameters.

[0091] When the original gradient is processed by wavelet transform for multi-scale sparsification, the original gradient is decomposed into low-frequency and high-frequency parts, and sparsification is performed in the multi-scale space, and the expression is:

[0092] ;

[0093] wherein, represents the original gradient; represents the low-frequency coefficient, i.e. the global trend; represents the scale function, which is used to represent the low-frequency part; represents the high-frequency coefficient, i.e. the local detail; represents the wavelet basis function, which is used to represent the high-frequency part; , respectively represent the scale, position index.

[0094] In this way, a large amount of redundant data in the original gradient is reduced, the proportion of non-zero elements is reduced, and the most important components can be selectively retained, so that the information integrity is maintained while reducing the data amount.

[0095] At the same time, in order to ensure that too much energy is not lost after compression, an energy preservation constraint is introduced to ensure that the low-frequency part and part of the high-frequency component can retain most of the original signal energy, so as to avoid the decline of training performance, and the expression is:

[0096] ;

[0097] wherein, represents the reserved high-frequency set, represents the energy preservation ratio; represents the L2 norm;

[0098] In the sparsification process, the high-frequency components are selectively retained, and the expression is:

[0099] ;

[0100] wherein, represents the high-frequency coefficient after sparsification;

[0101] so as to obtain the gradient representation after sparsification :

[0102] ;

[0103] wherein, represents the reserved low-frequency coefficient set; represents the high-frequency set after sparsification.

[0104] The sparse gradient representation is evaluated in conjunction with the optimal compression operator to dynamically adjust the sparsity parameters, specifically as follows:

[0105] Calculate the mutual information between the original gradient and the sparsed gradient representation;

[0106] The optimal compression operator is obtained by combining mutual information constraints and the Lagrange objective function;

[0107] When the calculated mutual information and the optimal compression operator do not meet the set target, adjust the threshold of the wavelet transform or the number of decomposition layers.

[0108] For example, if the mutual information is lower than a preset threshold (indicating excessive information loss), the threshold of the wavelet transform is lowered to retain more gradient components; if the optimal compression operator does not meet expectations (indicating insufficient redundancy reduction), the threshold is raised to further sparsify the gradient. Through iterative adjustments, the sparse gradient representation satisfies both the compression ratio requirement and ensures information consistency.

[0109] After completing the multi-scale sparse representation, the dimensionality and redundancy of the gradient have been significantly reduced, but further reduction of the communication data volume is still needed. Traditional quantization methods typically use a fixed bit width, such as 8-bit or 16-bit representation, which is not necessarily optimal at different training stages. In the early stages of training, the gradient amplitude is large and the distribution is complex, and a higher bit width helps to preserve information; however, in the later stages of training, the gradient gradually converges, and an excessively high bit width will lead to resource waste. Therefore, we propose a Bayesian adaptive quantization module that dynamically selects the optimal bit width and automatically adjusts the quantization strategy.

[0110] The Bayesian adaptive quantization module is used to estimate the probability distribution (such as Gaussian or Laplace distribution) of the sparse gradient representation based on the Gaussian mixture model (GMM) using Bayesian methods (such as Markov chain Monte Carlo method). The quantization bit width is dynamically adjusted according to the probability distribution of the gradient to generate a compressed gradient, which is transmitted back to the server for decompression and aggregation before being sent to the client for model updates.

[0111] The probability distribution of the gradient is obtained through probability modeling, and its expression is:

[0112] ;

[0113] in, Represents the gradient value of the input. The probability distribution function; This represents the number of components in a Gaussian mixture model; Indicates the first The weights of the Gaussian components satisfy the following conditions: ; Indicates the first the mean of the components, i.e. the center position; the variance of the components, i.e. the spread of the distribution; the Gaussian density function; the probability density under the mean the gradient value the mean , the variance

[0114] When dynamically adjusting the quantization bit width according to the gradient distribution, the optimal bit width is selected by maximum a posteriori inference to find the bit number that can maximize the posterior probability, so as to ensure the minimum error in the quantization process. The expression is:

[0115]

[0116] wherein, represents the optimal bit width, i.e. the finally selected bit number; represents finding the bit width value that maximizes the posterior probability; b represents the bit width; represents the gradient value in the sparse gradient representation sample set; represents the posterior probability of the bit width b under the given gradient sample set

[0117] Then, the scaling factor is calculated based on the standard deviation of the gradient and the bit width size to map the floating point number to the integer space. The expression is:

[0118]

[0119] wherein, represents the scaling factor, which is used to adjust the mapping ratio of the floating point number to the integer domain; represents the standard deviation of the gradient distribution.

[0120] The scaling factor depends on the standard deviation of the gradient and the bit width size, so as to ensure that the quantized value will not overflow.

[0121] Finally, each gradient value is mapped to an integer in the quantization process, and the error is reduced by rounding. The expression is:

[0122]

[0123] wherein, represents the gradient value quantized integer value, i.e. compressed gradient; represents the gradient mean, which is used for centering processing; represents the rounding operation.

[0124] ​​​​​​Each gradient value is mapped to an integer in the quantization process, and the error is reduced by rounding. The result of quantization will be transmitted to the server, along with the scaling factor and mean information for decoding.

[0125] The core idea of the Bayesian adaptive quantization module is to obtain gradient distribution information through probability modeling, and then infer the optimal quantization parameter according to the posterior probability, so as to achieve a dynamic balance between compression rate and accuracy.

[0126] The compressed gradient is transmitted to the server, and the server performs the following operations:

[0127] Decompression: According to the rules when compressing (such as wavelet inverse transform, quantization decoding), the compressed gradient is restored to the form of "approximate original gradient".

[0128] Aggregation: Use aggregation algorithms such as FedAvg algorithm to aggregate the decompressed gradients of each client into global gradients (reflecting the comprehensive update requirements of the model on all client data).

[0129] The server sends the aggregated global gradient to each client, and the client updates the local model parameters based on the global gradient, completing one iteration of federated learning. Subsequently, the client can continue the next round of "generate original gradient → compress → upload → aggregate → update" process based on the updated model until the model converges.

[0130] In summary, compared with existing distributed training compression methods, the present application has the following advantages:

[0131] Firstly, this scheme uses mutual information constraint modeling, which not only considers numerical error in the optimization process, but also ensures the information preservation between compressed representation and original gradient. This design avoids the risk of numerical approximation but information loss in traditional compression methods, so that the model can maintain stable convergence during training.

[0132] Secondly, the present application effectively captures the global trend and local details of the gradient through multi-scale sparse representation. Using wavelet transform decomposition method for sparsification at different scales can reduce redundant information while preserving key structural features, thereby significantly reducing communication volume and ensuring high training accuracy in low bandwidth environment.

[0133] Thirdly, the present application adopts Bayesian adaptive quantization strategy to automatically select the optimal bit width according to the dynamic characteristics of gradient distribution. In the early stage of training, the method tends to use higher bit width to ensure that information is not lost; while in the later stage of training, the method automatically reduces the bit width to improve the compression rate. Compared with traditional methods with fixed bit width, this scheme can achieve adaptive balance between communication efficiency and accuracy at different training stages.

[0134] Embodiment Two

[0135] As Figure 2 shown, the second embodiment of the present application also provides a gradient compression method based on information consistency driving, comprising:

[0136] S1, receiving the original gradient generated by the model of the client in the training process;

[0137] S2, constructing mutual information constraint based on the original gradient and the reconstructed gradient, and obtaining the optimal compression operator by minimizing the Lagrange objective function based on the mutual information constraint;

[0138] S3, using wavelet transform to perform multi-scale sparse processing on the original gradient to reduce redundant data, and obtaining the sparse gradient representation, and combining the mutual information constraint and the optimal compression operator to evaluate the sparse gradient representation to dynamically adjust the sparse parameter;

[0139] S4, estimating the probability distribution of the sparse gradient representation by Bayesian method, dynamically adjusting the quantization bit width according to the probability distribution of the gradient, generating the compressed gradient, transmitting it back to the server side, decompressing and aggregating it, and then issuing it to the client side for model update.

[0140] Embodiment Three

[0141] The third embodiment of the present application also provides a gradient compression device based on information consistency driving, comprising a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to realize the gradient compression method based on information consistency driving as described above.

[0142] Embodiment Four

[0143] The fourth embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor of the device where the computer readable storage medium is located to realize the gradient compression method based on information consistency driving as described above.

[0144] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus and method embodiments described above are only illustrative, for example, the flowchart in the drawings shows the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementation manners, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0145] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0146] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes. It should be noted that in this document, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device that includes the element.

[0147] The terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the description of the embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0148] It should be understood that the term "and / or" as used herein merely describes associated objects, and can exist in three forms, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0149] Depending on context, the word "if" as used herein can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0150] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order of the objects. Understandably, the "first\second" can be interchanged in a specific order or sequence as allowed. It should be understood that the objects distinguished by "first\second" can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0151] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A gradient compressor driven by information consistency, used for gradient compression between several clients and a server in a federated learning system, characterized in that, include: The modules include mutual information constraint modeling, multi-scale sparse representation, and Bayesian adaptive quantization. The mutual information constraint modeling module receives the original gradients generated by the client's model during training and uses mutual information constraints as the core objective of gradient compression, obtaining the optimal compression operator by minimizing the Lagrange objective function. The mutual information is the difference between entropy and conditional entropy, and its expression as the core objective of compression is: ； ； in, Represents the original gradient; This represents the reconstructed gradient, i.e., the approximate gradient after decompression; Compression indicates the target data transmitted between the client and the server; Indicates being bound by; This represents the compression operator, i.e., from the original gradient. To compressed representation The mapping function; This represents the expectation operator, which means taking the average value over the data distribution; Represents the original gradient With reconstructed gradient The squared L2 norm between them is used to measure the reconstruction error; Represents the L2 norm; Mutual information refers to the information about the original gradient that is retained in the compressed representation; Indicates the information retention threshold; Entropy represents the overall uncertainty of the original gradient; It represents the conditional entropy, which is the uncertainty that still exists in the gradient after a given compressed representation; The multi-scale sparse representation module is used to perform multi-scale sparsification processing on the original gradient using wavelet transform to reduce redundant data and obtain a sparse gradient representation. It is then evaluated in conjunction with the mutual information constraint and the optimal compression operator to dynamically adjust the sparsification parameters. The Bayesian adaptive quantization module is used to estimate the probability distribution of the sparse gradient representation using the Bayesian method, dynamically adjust the quantization bit width according to the probability distribution of the gradient, generate a compressed gradient, transmit it back to the server, decompress and aggregate it, and then send it back to the client for client model updates; wherein, the probability distribution of the gradient is obtained through probabilistic modeling, and the expression is: ； in, Represents the gradient value of the input. The probability distribution function; This represents the number of components in a Gaussian mixture model; Indicates the first The weights of the Gaussian components satisfy the following conditions: ; Indicates the first The mean of all components, i.e., the center position; Indicates the first The variance of each component, i.e., the extent of the distribution spread; Represents the Gaussian density function; Represents gradient value mean ,variance The probability density is as follows; When dynamically adjusting the quantization bit width based on the gradient distribution, the optimal bit width is selected by inferring the maximum a posteriori probability to find the number of bits that maximizes the a posteriori probability, ensuring that the error is minimized during quantization. The expression is: ； in, This indicates the optimal bit width, i.e., the final number of bits selected; This indicates the search for the bit width value that maximizes the posterior probability; b represents the bit width. The sparsified gradient represents the gradient of the th sample in the sample set. One gradient value; Represents a given set of gradient samples Below, the posterior probability of bit width b; Next, a scaling factor is calculated based on the standard deviation of the gradient and the bit width to map the floating-point number to the integer space, expressed as: ； in, This represents the scaling factor, used to adjust the mapping ratio from floating-point numbers to integer fields; The standard deviation of the gradient distribution; Finally, during the quantization process, each gradient value is mapped to an integer, and errors are reduced by rounding. The expression is as follows: ； in, Represents gradient value The quantized integer value, i.e., the compressed gradient; This represents the gradient mean, used for centering. This indicates the rounding operation.

2. The gradient compressor based on information consistency driving according to claim 1, characterized in that... The constrained optimization problem is transformed into an unconstrained form using the Lagrangian objective function, and a balance term is introduced to flexibly adjust the importance of reconstruction error and information preservation. Its expression is: ； in, Represent the Lagrange objective function; This represents the Lagrange multiplier, used to balance the error term and the information term; This represents the expected reconstruction error term; This indicates the information difference.

3. A gradient compressor based on information consistency driving according to claim 2, characterized in that... The expression for the optimal compression operator is: ； in, This represents the optimal compression operator; This indicates a minimize operation; Represent the Lagrange objective function; This represents the compression operator.

4. A gradient compressor based on information consistency driving according to claim 1, characterized in that... When performing multi-scale sparsification on the original gradient using wavelet transform, the original gradient is decomposed into low-frequency and high-frequency components, and sparsification is performed in the multi-scale space. The expression is as follows: ； in, Represents the original gradient; This represents low-frequency coefficients, i.e., global trends; The scaling function is used to represent the low-frequency component. This represents high-frequency coefficients, i.e., local details; This represents the wavelet basis function, used to represent the high-frequency components; , These represent scale and location index, respectively. Simultaneously, an energy preservation constraint is introduced to ensure that the low-frequency components and some high-frequency components retain most of the original signal energy, expressed as: ； in, Represents the set of high-frequency values ​​that are retained. This indicates that the energy ratio is maintained. Represents the L2 norm; During the sparsification process, high-frequency components are selectively preserved, as expressed by: ； in, Represents the high-frequency coefficients after sparsification; Thus, the sparsed gradient representation is obtained. : ； in, This represents the set of retained low-frequency coefficients; This represents the high-frequency set after sparsification.

5. A gradient compressor based on information consistency driving according to claim 3, characterized in that... The optimal compression operator is used for evaluation to dynamically adjust the sparsity parameters, specifically: Calculate the mutual information between the original gradient and the sparsed gradient representation; The optimal compression operator is obtained by combining mutual information constraints and the Lagrange objective function; When the calculated mutual information and the optimal compression operator do not meet the set target, adjust the threshold of the wavelet transform or the number of decomposition layers.

6. A gradient compression method based on information consistency for federated learning, characterized in that... ,include: Receive the raw gradients generated by the model during training from the client; Mutual information constraints are constructed based on the original gradient and the reconstructed gradient. Based on these mutual information constraints, the optimal compression operator is obtained by minimizing the Lagrangian objective function. Here, the mutual information is the difference between entropy and conditional entropy. The mutual information constraint is used as the core objective of compression, and its expression is: ； ； in, Represents the original gradient; This represents the reconstructed gradient, i.e., the approximate gradient after decompression; Compression indicates the target data transmitted between the client and the server; Indicates being bound by; This represents the compression operator, i.e., from the original gradient. To compressed representation The mapping function; This represents the expectation operator, which means taking the average value over the data distribution; Represents the original gradient With reconstructed gradient The squared L2 norm between them is used to measure the reconstruction error; Represents the L2 norm; Mutual information refers to the information about the original gradient that is retained in the compressed representation; Indicates the information retention threshold; Entropy represents the overall uncertainty of the original gradient; It represents the conditional entropy, which is the uncertainty that still exists in the gradient after a given compressed representation; Based on the original gradient, wavelet transform is used to perform multi-scale sparsification on the original gradient to reduce redundant data and obtain a sparse gradient representation. The sparse gradient representation is then evaluated in conjunction with the mutual information constraint and the optimal compression operator to dynamically adjust the sparsification parameters. The probability distribution of the sparse gradient representation is estimated using a Bayesian method. The quantization bit width is dynamically adjusted based on the gradient's probability distribution to generate a compressed gradient. This compressed gradient is transmitted back to the server, decompressed, aggregated, and then sent to the client for model updates. The probability distribution of the gradient is obtained through probabilistic modeling, and its expression is: ； in, Represents the gradient value of the input. The probability distribution function; This represents the number of components in a Gaussian mixture model; Indicates the first The weights of the Gaussian components satisfy the following conditions: ; Indicates the first The mean of all components, i.e., the center position; Indicates the first The variance of each component, i.e., the extent of the distribution spread; Represents the Gaussian density function; Represents gradient value mean ,variance The probability density is as follows; When dynamically adjusting the quantization bit width based on the gradient distribution, the optimal bit width is selected by inferring the maximum a posteriori probability to find the number of bits that maximizes the a posteriori probability, ensuring that the error is minimized during quantization. The expression is: ； in, This indicates the optimal bit width, i.e., the final number of bits selected; This indicates the search for the bit width value that maximizes the posterior probability; b represents the bit width. The sparsified gradient represents the gradient of the th sample in the sample set. One gradient value; Represents a given set of gradient samples Below, the posterior probability of bit width b; Next, a scaling factor is calculated based on the standard deviation of the gradient and the bit width to map the floating-point number to the integer space, expressed as: ； in, This represents the scaling factor, used to adjust the mapping ratio from floating-point numbers to integer fields; The standard deviation of the gradient distribution; Finally, during the quantization process, each gradient value is mapped to an integer, and errors are reduced by rounding. The expression is as follows: ； in, Represents gradient value The quantized integer value, i.e., the compressed gradient; This represents the gradient mean, used for centering. This indicates the rounding operation.

7. A gradient compression device based on information consistency driving, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement the gradient compression method based on information consistency as described in claim 6.

Images