A gradient compression method and gradient compressor based on adaptive neural coding

Abstract

This invention provides a gradient compression method and gradient compressor based on adaptive neural coding, applied to a client-side application, and relating to the fields of distributed machine learning and communication compression technology. The invention obtains the original feature gradient, calculates the importance score of the gradient element to adaptively generate sampling probabilities, and generates a sampling mask through a differentiable sampling mechanism to weight-correct the original feature gradient, obtaining a weighted corrected gradient. Then, it inputs the gradient into a lightweight neural encoder based on a multilayer perceptron architecture to encode a low-dimensional latent representation. A learnable quantization codebook is used to perform soft-allocation quantization on the low-dimensional latent representation, and the output discretized representation is transmitted to the server for reconstruction into a compressed gradient by the server-side decoder. This application can effectively retain key gradient information at a high compression ratio, significantly reduce communication overhead, and maintain model convergence speed and accuracy, making it suitable for resource-constrained scenarios such as federated learning and edge computing.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning and communication compression technology, and more specifically, to a gradient compression method and gradient compressor based on adaptive neural coding. Background Technology

[0002] Distributed deep learning, by distributing model training tasks across multiple client devices, effectively improves training efficiency and enhances data privacy protection, and has been widely applied in scenarios such as federated learning and edge computing. However, in such systems, the frequent exchange of high-dimensional gradient information between clients and the central server makes communication overhead a key bottleneck restricting overall performance, especially in low-bandwidth, high-latency, or resource-constrained network environments. The gradient tensor of a typical deep neural network often contains millions or even tens of millions of floating-point parameters, resulting in a massive amount of data in a single communication. This not only significantly increases transmission latency and energy consumption but also poses a severe challenge to the computing and storage capabilities of edge devices such as mobile devices or IoT nodes.

[0003] To alleviate this problem, existing technologies have proposed various gradient compression methods. For example, gradient sparsity strategies such as the Top-k method achieve dimensionality reduction by retaining the gradient elements with the largest absolute values, while random sparsity discards gradient components with a fixed probability; quantization techniques such as SignSGD transmit only the gradient sign and supplement it with a scaling factor, while QSGD discretizes the gradient using a fixed bit width (such as 4 bits or 8 bits); low-rank approximation methods such as PowerSGD use singular value decomposition to compress the gradient into a low-dimensional matrix product; differential coding schemes such as EF-SGD take advantage of the temporal correlation of gradients, transmit only the difference between rounds, and combine an error feedback mechanism to compensate for information loss.

[0004] While the aforementioned methods reduce communication load to some extent, they still have significant limitations: sparsification strategies lack awareness of the semantic importance of gradients, easily losing small gradients that play a crucial role in convergence or introducing noise, affecting model performance; fixed-width quantization is difficult to adapt to the dynamic changes in gradient distribution during training, easily causing information loss in the early stages of high-precision requirements, while failing to achieve sufficient compression ratios in later stages; low-rank decomposition has high computational complexity, making it difficult to deploy on resource-constrained devices, and its linear assumption makes it difficult to characterize the nonlinear structure in gradients; more importantly, existing methods are mostly used in isolation or in simple combinations, lacking systematic and multi-level collaborative design, making it difficult to achieve an ideal balance between high compression ratios and model convergence.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention aims to provide a gradient compression method and gradient compressor based on adaptive neural coding, in order to solve the technical defects of existing gradient compression methods in distributed deep learning, such as the difficulty in balancing compression ratio and model convergence performance, lack of adaptive capability to dynamic changes in gradient distribution, and inability to jointly optimize compression modules end-to-end.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A gradient compression method based on adaptive neural coding, applied to the client, includes:

[0009] Obtain the original feature gradient;

[0010] Based on the original feature gradient, the importance score of the gradient element is calculated to adaptively generate the sampling probability, and a sampling mask is generated through a differentiable sampling mechanism to perform weighted correction on the original feature gradient, so as to obtain the weighted corrected gradient.

[0011] The weighted and corrected gradient input is encoded and mapped into a low-dimensional latent representation using a lightweight neural encoder based on a multilayer perceptron architecture;

[0012] The low-dimensional latent representation is soft-allocated quantized using a learnable quantization codebook, and the output discretized representation is transmitted to the server so that it can be reconstructed into a compressed gradient by the server's decoder.

[0013] Preferably, the formula for calculating the importance score of the gradient element is:

[0014] ;

[0015] in, gradient element Importance score; This is the i-th gradient element of the original feature gradient; Indicates the total number of training rounds; for The number of consecutive rounds without an update; The decay coefficient is t; t is the training step index. It is an exponential function;

[0016] The formula for calculating the adaptive generation sampling probability is:

[0017] ;

[0018] in, gradient element The sampling probability; This is a concentration coefficient used to control the concentration of sampling. gradient element Importance score; The dimension of the original feature gradient;

[0019] Based on the sampling probability, Gumbel noise is introduced during sampling to perform differentiable sampling, resulting in an approximate sampling probability, as shown in the formula:

[0020] ;

[0021] in, gradient element Approximate sampling probability; Indicates compliance with standards Distributed noise; For temperature parameters; gradient element sampling probability;

[0022] Based on the approximate sampling probability, a binary sampling mask is generated using a threshold function to control the selection of elements. The expression is as follows:

[0023] ;

[0024] in, gradient element binary sampling mask; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. This is a threshold function used to return the probability of retaining a sample. The proportion is The threshold; Target retention rate;

[0025] Based on the binary sampling mask, importance sampling correction is performed on the original feature gradient to obtain the weighted corrected gradient. The formula is:

[0026] ;

[0027] in, The original feature gradient The binary sampling mask vector; This is the sampling probability vector; This is element-wise multiplication; This is an element-wise division method.

[0028] Preferably, the expression for encoding mapping using the lightweight neural encoder is:

[0029] ;

[0030] in, For low-dimensional latent representation; The lightweight neural encoder; The gradient is the weighted correction. Activation function; , This is the encoder weight matrix; , This refers to the encoder bias coefficient; It is a real number; Let be the number of dimensions of the low-dimensional latent representation; This represents the dimension of the intermediate hidden layers of the encoder. The dimension of the original feature gradient;

[0031] The constraint parameter is given by the following formula:

[0032] ;

[0033] in, These are the parameter budget coefficients.

[0034] Preferably, the low-dimensional latent representation is soft-assigned quantization using a learnable quantization codebook, specifically as follows:

[0035] First, construct a learnable codebook collection. , is used to discretize continuous latent vectors; where, For codebook size, It is the first in the codebook A discrete code vector;

[0036] Then, weights are calculated using temperature-scaled softmax, and the continuous low-dimensional latent representation is softquantized into a weighted combination of discrete codebooks, outputting the discretized representation, as shown in the formula:

[0037] ;

[0038] ;

[0039] in, It is a discretized representation; To quantify the weights; , The first in the codebook n discrete code vectors; It is an L2 norm; For low-dimensional latent representation; It is an exponential function;

[0040] To quantize the temperature parameter, it is gradually decreased during training to eventually converge to hard quantization, as shown in the formula:

[0041] ;

[0042] in, For the first Quantitative temperature parameters for round training; These are the initial temperature parameters; The attenuation rate; This refers to the total number of training rounds.

[0043] Preferably, it further includes: optimizing the codebook using a rate-distortion objective loss function, and adaptively adjusting the codebook size according to the current compression requirements, as expressed by:

[0044] ;

[0045] in, The rate-distortion objective loss function; The distortion term represents the low-dimensional latent representation. Discretization representation The mean square error; For the weighting factor; Represents the entropy term;

[0046] The information entropy, approximated by the codebook, reflects the average bit overhead of the encoding. The formula is:

[0047] ;

[0048] in, Discrete code vector within a batch Average usage frequency; The size of the codebook;

[0049] Codebook size The compression ratio is dynamically adjusted based on the difference between the target compression ratio and the current compression ratio. The formula is as follows:

[0050] ;

[0051] in, Let t be the codebook size for the t-th round; , These are the lower and upper bounds of the codebook size, respectively; Activation function; To adjust the rate, it is used to control the sensitivity of codebook size to changes in compression ratio; , These represent the target compression ratio and the current compression ratio, respectively.

[0052] Preferably, the method further includes: after receiving the discretized representation sent by the client, the server decodes it using a decoder corresponding to the lightweight neural encoder to obtain the compressed gradient. The expression is:

[0053] ;

[0054] in, , This is the decoder weight matrix; , This refers to the decoder bias coefficient; For decoders.

[0055] Preferably, it further includes: performing joint training by minimizing the compression error between the original feature gradient and the compressed gradient, wherein the formula for the joint training loss function is:

[0056] ;

[0057] in, For joint training loss function; To regularize sparsity; , These are the L1 norm and the L2 norm, respectively. , These are the original feature gradient and the compressed gradient, respectively.

[0058] Preferably, the method further includes: calculating the compression error based on the original feature gradient and the compression gradient, constructing a residual buffer and dynamically updating the residual, so that the residual is compressed and transmitted to the server when a preset condition is met, and the server fuses the compression gradient and the residual to generate global aggregation parameters for broadcast updates.

[0059] Preferably, the process of constructing the residual buffer and dynamically updating the residuals is as follows:

[0060] First, calculate the difference between the original feature gradient and the compressed gradient to obtain the compression error, as shown in the formula:

[0061] ;

[0062] in, Let be the compression error in round t; , Let be the original feature gradient and the compressed gradient of the t-th round, respectively;

[0063] The cumulative residual is calculated based on the compression error, and an adaptive decay factor is introduced to prevent the residual from growing indefinitely. The formula for updating the cumulative residual is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] in, , These are the cumulative residuals for rounds t and t-1, respectively. Let be the decay coefficient in round t; Let be the change in the loss function in round t; , These are the loss function values ​​for rounds t and t-1, respectively. To prevent division by zero of small constants;

[0068] The preset conditions are: when the number of transmissions reaches the set number of transmission rounds K, or when the residual norm exceeds the set threshold, the formula is:

[0069] ;

[0070] in, This indicates a transmission trigger flag, where 1 indicates triggering and 0 indicates no triggering; K is the set fixed transmission round interval. This is the relative threshold coefficient; It is an L2 norm; Representation or operation;

[0071] Finally, the residual is compressed and transmitted to the server before the residual buffer is reset.

[0072] The present invention also provides a gradient compressor based on adaptive neural coding, comprising:

[0073] The data acquisition unit is used to acquire the original feature gradients;

[0074] The sampling correction unit is used to calculate the importance score of the gradient element based on the original feature gradient to adaptively generate the sampling probability, and generate a sampling mask through a differentiable sampling mechanism to perform weighted correction on the original feature gradient to obtain the weighted corrected gradient.

[0075] The mapping unit is used to encode and map the weighted and corrected gradient input into a low-dimensional latent representation based on a lightweight neural encoder of a multilayer perceptron architecture.

[0076] The discrete quantization unit is used to perform soft-allocation quantization on the low-dimensional latent representation using a learnable quantization codebook, and outputs the discrete representation to the server for reconstruction into a compressed gradient by the server's decoder.

[0077] The present invention also provides a gradient compression device based on adaptive neural coding, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement the gradient compression method based on adaptive neural coding as described above.

[0078] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device in which the computer-readable storage medium resides, implement a gradient compression method based on adaptive neural coding as described above.

[0079] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0080] This invention combines dynamic importance score calculation with a differentiable sampling mechanism, and introduces a lightweight neural encoder and a learnable quantization codebook to achieve precise selection and efficient compression of core gradient information, thus maintaining model training convergence while reducing communication load. Specifically, this scheme dynamically evaluates the importance scores of gradient elements and generates sampling probabilities to ensure that key gradient elements are retained first, avoiding the information loss problem caused by the lack of semantic awareness in traditional sparsity methods. In addition, the lightweight neural encoder with a multilayer perceptron architecture can effectively capture the nonlinear structural features in the gradient, making it more suitable for resource-constrained devices compared to linear decomposition methods. Finally, soft-assigned quantization is achieved through a learnable quantization codebook, solving the problem that fixed-bit-width quantization cannot adapt to dynamic changes in gradient distribution, thereby achieving a balance between high compression ratio and model convergence.

[0081] This application solves the problem of high gradient communication overhead between clients and servers in distributed deep learning through the collaborative design of adaptive importance perception and neural coding. At the same time, it overcomes the information loss and convergence problems caused by the lack of semantic awareness, inability to adapt to dynamic gradient changes and excessive computational complexity of existing compression methods. Attached Figure Description

[0082] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0083] Figure 1 This is a flowchart illustrating a gradient compression method based on adaptive neural coding, as provided in Example 1.

[0084] Figure 2 This is a schematic diagram of the framework of a gradient compression method based on adaptive neural coding provided in Example 1.

[0085] Figure 3 This is a schematic diagram of a gradient compressor based on adaptive neural coding, provided in Example 2.

[0086] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0088] Example 1

[0089] Embodiment 1 of the present invention provides a gradient compression method based on adaptive neural coding, which can be implemented by a gradient compression device based on adaptive neural coding (hereinafter referred to as gradient compression device), and in particular, executed by one or more processors within the gradient compression device.

[0090] In this embodiment, the gradient compression device may be an electronic device equipped with a processor, the processor having a computer program for the gradient compression method based on adaptive neural coding and the computer program being executable, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0091] In distributed deep learning systems, after a client device (such as a mobile terminal, edge server, or IoT node) completes a round of model training locally, it generates the original gradient input through backpropagation. This gradient is a high-dimensional tensor, typically with dimensions in the millions or even tens of millions. To reduce communication overhead, the client deploys the gradient compressor proposed in this invention, which executes four core processing stages, S1 to S4, sequentially. A global compression ratio controller coordinates the parameters of each module to adapt to the current network environment and training state.

[0092] like Figures 1-2As shown, a gradient compression method based on adaptive neural coding is applied to a client, which includes steps S1 to S4.

[0093] S1, obtain the original feature gradient.

[0094] The client performs forward and backward propagation of the model locally and calculates the original feature gradient for the current training epoch (denoted as t).

[0095] S2, based on the original feature gradient, calculate the importance score of the gradient element to adaptively generate the sampling probability, and generate a sampling mask through a differentiable sampling mechanism to perform weighted correction on the original feature gradient to obtain the weighted corrected gradient.

[0096] This step involves calculating importance scores, generating sampling probabilities, performing differentiable sampling, and weighted correction. It retains gradient elements that have a significant impact on the loss and removes redundant information.

[0097] Specifically, for each gradient element of the original feature gradient Calculate its sensitivity to the loss function. The formula for calculating the importance score of the gradient element is:

[0098] ;

[0099] in, gradient element Importance score; The original feature gradient The i-th gradient element; Indicates the total number of training rounds; for The number of consecutive rounds without an update; The decay coefficient is t; t is the training step index. It is an exponential function.

[0100] The purpose of importance scores for gradient elements is to accurately assess semantic importance and avoid the shortcomings of gradient loss or noise introduction in existing sparsity strategies. In practical applications, the impact of the number of times an element has not been updated can be balanced by adjusting the decay coefficient β, thereby optimizing the accuracy of importance assessment.

[0101] Then, the importance scores are normalized to obtain the adaptive sampling probability of each gradient element, calculated using the following formula:

[0102] ;

[0103] in, gradient element The sampling probability; To collect a concentration coefficient, used to control the concentration of sampling, for example, let... The larger the value, the more likely it is to select elements of high importance; gradient element Importance score; is the dimension of the original feature gradient.

[0104] Adaptive sampling probability is a dynamic adjustment mechanism that controls the sampling concentration through a concentration coefficient γ, ensuring that important elements are prioritized. Different γ values ​​can be selected to adjust the concentration of the sampling distribution, thereby achieving a balance between information preservation and compression efficiency.

[0105] During sampling, Gumbel noise is introduced for differentiability to obtain an approximate sampling probability, as shown in the formula:

[0106] ;

[0107] in, gradient element Approximate sampling probability; Indicates compliance with standards Distributed noise; For temperature parameters; gradient element The sampling probability.

[0108] Differentiable sampling mechanisms can ensure differentiability of the training process while maintaining sampling randomness, thus solving the problem of non-differentiability in traditional sampling methods. Temperature parameter τ The specific value can be dynamically adjusted according to the needs of the training phase.

[0109] Based on the approximate sampling probability, a binary sampling mask is generated using a threshold function to control the selection of elements and avoid the non-differentiability of discrete sampling. The expression is as follows:

[0110] ;

[0111] in, gradient element binary sampling mask; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. This is a threshold function used to return the probability of retaining a sample. The proportion is The threshold; Target retention rate.

[0112] The purpose of generating a binary sampling mask is to precisely control the sampling rate and reduce information loss. The threshold function can be designed in various ways, such as linear mapping or nonlinear transformation, to suit different application scenarios.

[0113] Based on the binary sampling mask, importance sampling correction is performed on the original feature gradient to obtain the weighted corrected gradient. The formula is:

[0114] ;

[0115] in, The original feature gradient The binary sampling mask vector; This is the sampling probability vector; This is element-wise multiplication; This is an element-wise division method.

[0116] By organically combining the above process with steps such as obtaining the original feature gradient and mapping the gradient after weighted correction, not only are the problems of accurately calculating the importance of gradient elements, realizing differentiable sampling, and controlling concentration solved, but also an ideal balance between high compression ratio and model convergence is achieved through close cooperation between the steps.

[0117] S3 encodes the weighted and corrected gradient input into a low-dimensional latent representation using a lightweight neural encoder based on a multilayer perceptron architecture.

[0118] This step compresses the high-dimensional weighted correction gradient into a low-dimensional continuous vector and learns the optimal nonlinear compression strategy. In practical applications, a lightweight neural encoder refers to an encoder based on a multilayer perceptron architecture, which can be implemented through two fully connected layers. Each layer contains linear transformations and nonlinear activation functions to capture complex patterns in the gradient.

[0119] The expression for encoding mapping using the lightweight neural encoder is:

[0120] ;

[0121] in, For low-dimensional latent representation; The lightweight neural encoder; The gradient is the weighted correction. Activation function; , This is the encoder weight matrix; , This refers to the encoder bias coefficient; It is a real number; Let be the number of dimensions of the low-dimensional latent representation; This represents the dimension of the intermediate hidden layers of the encoder. The dimension of the original feature gradient;

[0122] The constraint parameter is given by the following formula:

[0123] ;

[0124] in, These are the parameter budget coefficients.

[0125] This step maps the weighted and corrected gradients into a low-dimensional latent representation using a lightweight neural encoder, addressing the issue of excessive computational resource consumption caused by an excessive number of encoder parameters. Furthermore, the encoder design fully considers the dynamic changes in gradient distribution, utilizing the nonlinear activation function and weight matrix in a multilayer perceptron architecture to capture the semantic importance of the gradients, avoiding information loss caused by fixed-width quantization or linear assumptions in traditional methods. Simultaneously, by constraining the total number of encoder parameters, the model can run efficiently on resource-constrained client devices. This design not only effectively reduces communication overhead but also ensures information integrity during gradient compression, thus achieving a good balance between high compression ratio and model convergence.

[0126] S4. The low-dimensional latent representation is soft-allocated quantized using a learnable quantization codebook, and the output discretized representation is transmitted to the server so that it can be reconstructed into a compressed gradient by the decoder on the server.

[0127] The low-dimensional latent representation is soft-assigned quantization using a learnable quantization codebook, specifically as follows:

[0128] First, construct a learnable codebook collection. , is used to discretize continuous latent vectors; where, For codebook size, It is the first in the codebook A discrete code vector;

[0129] Then, the weights are calculated using temperature-scaled softmax, and the continuous low-dimensional latent representation is softquantized into a weighted combination of discrete codebooks, as shown in the formula:

[0130] ;

[0131] ;

[0132] in, It is a discretized representation; To quantify the weights; , The first in the codebook n discrete code vectors; It is an L2 norm; For low-dimensional latent representation; It is an exponential function;

[0133] To quantize the temperature parameter, it is gradually decreased during training, eventually converging to hard quantization, as shown in the formula:

[0134] ;

[0135] in, For the first Quantitative temperature parameters for round training; These are the initial temperature parameters; The attenuation rate; This refers to the total number of training rounds.

[0136] The temperature scaling softmax mechanism is a technique for dynamically adjusting the quantization of softness and hardness. Its core lies in using temperature parameters. Controlling the smoothness of the softmax function. In practical applications, temperature parameters... This can be achieved using linear decay, exponential decay, or other non-linear decay methods. The purpose of this mechanism is to ensure that the quantization process can adaptively adjust according to the training phase, preserving more detailed information in the early stages while achieving a more compact representation in the later stages.

[0137] This step not only addresses the information loss that can occur with fixed quantization strategies in gradient compression, but also achieves the goal of preserving key details during dynamic training. Particularly in distributed deep learning scenarios, this approach significantly reduces communication overhead while ensuring model convergence, providing a superior gradient compression solution for resource-constrained devices.

[0138] In a preferred embodiment, a rate-distortion objective loss function is used to optimize the codebook, and the codebook size is adaptively adjusted according to the current compression requirements. The expression is as follows:

[0139] ;

[0140] in, The rate-distortion objective loss function; The distortion term represents the low-dimensional latent representation. Discretization representation The mean square error; For the weighting factor; This represents the entropy term.

[0141] Rate-distortion objective loss function is a mathematical tool that optimizes the codebook by combining distortion and entropy terms. It can be implemented using gradient descent-based optimization algorithms. By introducing rate-distortion objective loss function to optimize the codebook, the problems of large distortion or high bit overhead during quantization are solved.

[0142] The information entropy, approximated by the codebook, reflects the average bit overhead of the encoding. The formula is:

[0143] ;

[0144] in, Discrete code vector within a batch Average usage frequency; The size of the codebook;

[0145] Codebook size The compression ratio is dynamically adjusted based on the difference between the target compression ratio and the current compression ratio. The formula is as follows:

[0146] ;

[0147] in, Let t be the codebook size for the t-th round; , These are the lower and upper bounds of the codebook size, respectively; Activation function; To adjust the rate, it is used to control the sensitivity of codebook size to changes in compression ratio; , These represent the target compression ratio and the current compression ratio, respectively.

[0148] This embodiment, by optimizing the codebook and dynamically adjusting its size, not only effectively reduces information distortion during quantization but also significantly lowers the bit overhead of encoding, while ensuring that the codebook size can flexibly adapt to changes in compression requirements during training. This multi-layered collaborative design achieves an ideal balance between high compression ratio and model convergence, providing a more systematic solution to the communication overhead problem in distributed deep learning.

[0149] The server receives the discretized representation sent by the client and decodes it using the decoder corresponding to the lightweight neural encoder to obtain the compressed gradient. The expression is:

[0150] ;

[0151] in, , This is the decoder weight matrix; , This refers to the decoder bias coefficient; For decoders.

[0152] A decoder is a computational module used to restore a low-dimensional discretized representation to high-dimensional data, and it can be implemented using a multilayer perceptron architecture. In practical applications, the design of the decoder needs to ensure structural symmetry with the encoder to avoid information distortion caused by asymmetric operations.

[0153] In a preferred embodiment, the method further includes joint training by minimizing the compression error between the original feature gradient and the compressed gradient, wherein the formula for the joint training loss function is:

[0154] ;

[0155] in, For joint training loss function; To regularize sparsity; , These are the L1 norm and the L2 norm, respectively. , These are the original feature gradient and the compressed gradient, respectively.

[0156] The joint training loss function is an objective function used to optimize the parameters of the encoder and decoder, with the aim of improving the convergence and training performance of the model by minimizing the compression error.

[0157] Compression error refers to the difference between the original feature gradient and the compressed gradient. It can be calculated by element-wise difference, with the aim of quantifying the information loss during the compression process.

[0158] In a preferred embodiment, the method further includes: calculating the compression error based on the original feature gradient and the compression gradient, constructing a residual buffer and dynamically updating the residual, so that the residual is compressed and transmitted to the server when a preset condition is met, and the server fuses the compression gradient and the residual to generate global aggregation parameters for broadcast updates.

[0159] Specifically, the process of constructing the residual buffer and dynamically updating the residuals is as follows:

[0160] First, calculate the difference between the original feature gradient and the compressed gradient to obtain the compression error, as shown in the formula:

[0161] ;

[0162] in, Let be the compression error in round t; , Let be the original feature gradient and the compressed gradient of the t-th round, respectively;

[0163] The cumulative residual is calculated based on the compression error, and an adaptive decay factor is introduced to prevent the residual from growing indefinitely. The formula for updating the cumulative residual is as follows:

[0164] ;

[0165] ;

[0166] ;

[0167] in, , These are the cumulative residuals for rounds t and t-1, respectively. Let be the decay coefficient in round t; Let be the change in the loss function in round t; , These are the loss function values ​​for rounds t and t-1, respectively. To prevent small constants from being divided by zero.

[0168] The preset conditions are: when the number of transmissions reaches the set number of transmission rounds K, or when the residual norm exceeds the set threshold, the formula is:

[0169] ;

[0170] in, This indicates a transmission trigger flag, where 1 indicates triggering and 0 indicates no triggering; K is the set fixed transmission round interval. This is the relative threshold coefficient; It is an L2 norm;

[0171] After compressing the residuals and transmitting them to the server, the residual buffer is reset (e.g., reset to 0) to prepare for subsequent rounds and avoid interference from residual errors.

[0172] This process not only reduces communication overhead but also ensures the transmission of critical error information. After receiving the residuals, the server fuses them with the compressed gradients to reconstruct more complete gradient information, which is then used to generate global aggregate parameters and broadcast to all clients. This scheme, by dynamically managing compression errors, effectively improves the model's convergence accuracy and efficiency while reducing communication overhead.

[0173] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0174] First, the lightweight neural encoder of this invention learns a compact representation of gradients through nonlinear mapping, overcoming the expressive limitations of traditional linear transformations (such as DCT and PCA) and more efficiently preserving gradient semantic information in low-dimensional space. Second, based on the actual impact of gradients on the loss function, this invention introduces gradient element importance and a differentiable sampling mechanism to dynamically evaluate and allocate encoding resources, avoiding the erroneous deletion of small but critical gradients by fixed strategies such as Top-k. Third, this invention introduces rate-distortion theory from information theory into gradient compression, achieving optimal bit allocation through learnable codebooks and entropy regularization, significantly improving quantization efficiency. Fourth, the adaptive residual compensation mechanism adopted in this invention not only accumulates errors but also adjusts the decay factor and transmission strategy according to the model convergence state, matching error feedback with training dynamics and effectively suppressing long-term bias. Finally, the end-to-end differentiable architecture allows all module parameters to be jointly optimized during training, avoiding the overall performance loss caused by local optima in traditional staged compression methods. The collaborative work of the above modules enables the present invention to maintain a model convergence speed and final accuracy close to the uncompressed baseline even at high compression rates. At the same time, the number of encoder parameters is strictly limited, making it suitable for resource-constrained platforms such as mobile terminals and IoT devices.

[0175] This invention constructs a lightweight, multi-layered, end-to-end differentiable compression-decompression pipeline, which significantly reduces the amount of communication data while dynamically retaining the gradient information most critical to model convergence, thereby achieving efficient and stable distributed training in bandwidth-constrained scenarios such as federated learning and edge computing.

[0176] Example 2

[0177] like Figure 3 As shown, the second embodiment of the present invention also provides a gradient compressor based on adaptive neural coding, comprising:

[0178] The data acquisition unit is used to acquire the original feature gradients;

[0179] The sampling correction unit is used to calculate the importance score of the gradient element based on the original feature gradient to adaptively generate the sampling probability, and generate a sampling mask through a differentiable sampling mechanism to perform weighted correction on the original feature gradient to obtain the weighted corrected gradient.

[0180] The mapping unit is used to encode and map the weighted and corrected gradient input into a low-dimensional latent representation based on a lightweight neural encoder of a multilayer perceptron architecture.

[0181] The discrete quantization unit is used to perform soft-allocation quantization on the low-dimensional latent representation using a learnable quantization codebook, and outputs the discrete representation to the server for reconstruction into a compressed gradient by the server's decoder.

[0182] It also includes: a residual update unit, used to calculate the compression error based on the original feature gradient and the compression gradient, construct a residual buffer and dynamically update the residual, so that when the preset conditions are met, the residual is compressed and transmitted to the server, and the server fuses the compression gradient and the residual to generate global aggregation parameters for broadcast update.

[0183] Example 3

[0184] The third embodiment of the present invention also provides a gradient compression device based on adaptive neural coding, which includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the gradient compression method based on adaptive neural coding as described above.

[0185] Example 4

[0186] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device where the computer-readable storage medium is located, implement the gradient compression method based on adaptive neural coding as described above.

[0187] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A gradient compression method based on adaptive neural coding, applied to a client, characterized in that, The method comprises the following steps: obtaining an original feature gradient; based on the original feature gradient, calculating the importance score of the gradient element to adaptively generate a sampling probability, and generating a sampling mask through a differentiable sampling mechanism to weight and correct the original feature gradient to obtain a weighted and corrected gradient; inputting the weighted and corrected gradient into a lightweight neural encoder based on a multi-layer perception architecture to encode and map into a low-dimensional latent representation; using a learnable quantization codebook to soft-allocate and quantize the low-dimensional latent representation, and outputting a discretized representation to a server end to reconstruct a compressed gradient through a decoder of the server end; wherein the formula for calculating the importance score of the gradient element is: ； wherein, is an importance score for the gradient element is the i-th gradient element of the original feature gradient denotes the total training epoch; is the number of consecutive epochs without update is the decay coefficient; t is the training step index; is the exponential function;​​​ the formula for adaptively generating a sampling probability is: ； wherein, is a sampling probability for the gradient element ; is a concentration coefficient for controlling the concentration of the sampling; is an importance score for the gradient element ; is a dimension number of the original feature gradient; based on the sampling probability, introducing Gumbel noise for differentiable sampling during sampling to obtain an approximate sampling probability; based on the approximate sampling probability, generating a binary sampling mask through a threshold function to control the selection of elements; based on the binary sampling mask, performing importance sampling correction on the original feature gradient to obtain a weighted and corrected gradient.

2. The method of claim 1, wherein the method is based on adaptive neural coding. The formula for the approximate sampling probability is: ； wherein, is an approximation of the sampling probability of the gradient element ; represents noise subject to a standard distribution; is a temperature parameter; is a sampling probability of the gradient element ; The expression of the binary sampling mask is: ； wherein, is a binary sampling mask for gradient elements is an indicator function that takes 1 if the condition is satisfied, otherwise 0; is a threshold function that returns the fraction of the reserved sampling probability with a threshold of is the target reservation rate;​​ The formula for the weighted and corrected gradient is: ； wherein, is the weighted corrected gradient; is the original feature gradient is a binary sampling mask vector; is a sampling probability vector; is an element-wise multiplication; is an element-wise division.

3. The method of claim 2, wherein the method is based on adaptive neural coding. The expression for encoding and mapping through the lightweight neural encoder is: ； wherein, is a low-dimensional latent representation; is the lightweight neural encoder; is a weighted corrected gradient; is an activation function; , is an encoder weight matrix; , is an encoder bias coefficient; is a real number; is a dimension number of the low-dimensional latent representation; is a dimension number of the encoder intermediate hidden layer; is a dimension number of the original feature gradient; For the constraint parameter, the formula is: ； wherein is a parameter budget coefficient.

4. The method of claim 3, wherein the method is based on adaptive neural coding. The soft-allocated and quantized low-dimensional latent representation is obtained by using a learnable quantization codebook, specifically: First, a set of learnable codebooks is constructed for discretizing continuous latent vectors; wherein, is the codebook size, is the th discrete code vector in the codebook; Then, through temperature scaling softmax, the continuous low-dimensional latent representation is soft-quantized into a weighted combination of discrete codebooks, and a discretized representation is output, the formula is: ； ； wherein is a discretized representation; is a quantized weight; , are the first , n discrete code vectors in the codebook, respectively; is the L2 norm; is a low-dimensional latent representation; is the exponential function; To quantize the temperature parameter, it is gradually reduced during training to eventually converge to hard quantization, with the formula: ； wherein, is the initial temperature parameter; is the decay rate; is the initial temperature parameter; is the decay rate; is the total number of training epochs.

5. The method of claim 4, wherein Further comprising: optimizing the codebook using a rate-distortion target loss function, and adaptively adjusting the codebook size according to the current compression requirement, the expression is: ； wherein, is a rate-distortion target loss function; is a distortion term, representing the mean squared error of the low-dimensional latent representation and the discrete representation ; is the mean squared error; is a weighting coefficient; represents an entropy term; The information entropy of the approximate codebook is used to reflect the average bit cost of the encoding, and the formula is: ； wherein, is the average frequency of use of the intra-batch discrete code vector is the average frequency of use of the intra-batch discrete code vector is the codebook size; Codebook size The dynamic adjustment is made according to the difference between the target compression rate and the current compression rate, and the formula is: ； wherein, is the codebook size for the t-th round; , are lower and upper bounds of the codebook size, respectively; is an activation function; is an adjustment rate for controlling the sensitivity of the codebook size to the difference of compression rates; , denote the target compression rate and the current compression rate, respectively.

6. The method of claim 4, wherein Further comprising: receiving, by the server, the discretized representation sent by the client, and decoding, by a decoder corresponding to the lightweight neural encoder, the compressed gradient The expression is: ； wherein, , is a decoder weight matrix; is an activation function; , is a decoder bias coefficient; is a decoder.

7. The method of claim 6, wherein the method is based on adaptive neural coding. Further comprising: jointly training by minimizing the compression error between the original feature gradient and the compressed gradient, the formula of the joint training loss function is: ； wherein, is a joint training loss function; is a sparse regularization sparsity; , are an L1 norm, an L2 norm, respectively; , are an original feature gradient, a compressed gradient, respectively.

8. The method of claim 6, wherein the method is based on adaptive neural coding. Further comprising: calculating the compression error according to the original feature gradient and the compressed gradient, constructing a residual buffer and dynamically updating the residual, and when a preset condition is met, transmitting the compressed residual to the server end, and generating a global aggregated parameter by fusing the compressed gradient and the residual on the server end to broadcast update.

9. The method of claim 8, wherein the method is based on adaptive neural coding. The process of constructing the residual buffer and dynamically updating the residual is as follows: First, calculate the difference between the original feature gradient and the compressed gradient to obtain the compression error, the formula is: ； wherein, is the compression error for the tth round; , are the original feature gradient and the compressed gradient for the tth round, respectively. Based on the compression error, calculate the accumulated residual and introduce an adaptive attenuation factor to prevent the residual from growing indefinitely, the update calculation formula of the accumulated residual is: ； ； ； wherein, , are the cumulative residual errors of the t, t-1 rounds, respectively; is the decay coefficient of the t round; is the change of the t round loss function; , are the loss function values of the t, t-1 rounds, respectively; is a small constant to prevent division by zero; The preset condition is: when the transmission times reach a set transmission round K, or when the residual norm exceeds a set threshold, the formula is: ； wherein, denotes a transmission trigger flag, 1 denotes triggering, and 0 denotes not triggering; K is a fixed transmission round interval set; is a relative threshold coefficient; is an L2 norm; denotes an OR operation; Finally, after transmitting the compressed residual to the server end, reset the residual buffer.

10. A gradient compressor based on adaptive neural encoding for implementing a gradient compression method based on adaptive neural encoding according to any one of claims 1-9, characterized in that, The method comprises the following steps: a data acquisition unit for obtaining an original feature gradient; a sampling correction unit for calculating the importance score of the gradient element based on the original feature gradient to adaptively generate a sampling probability, and generating a sampling mask through a differentiable sampling mechanism to weight and correct the original feature gradient to obtain a weighted and corrected gradient; a mapping unit configured to map the weighted corrected gradient into a low-dimensional latent representation by inputting the weighted corrected gradient into a lightweight neural encoder based on a multi-layer perceptron architecture; a discrete quantization unit configured to soft-assignment quantize the low-dimensional latent representation by using a learnable quantization codebook, and output a discrete representation to a server end to be reconstructed into the compressed gradient by a decoder of the server end.

Images