Adaptive heat map mask and mixed precision based spatio-temporal compression method and device

Abstract

The application provides a space-time compression method and device based on an adaptive heat map mask and mixed precision, and relates to the field of distributed learning. The application obtains original high-dimensional data, projects along the channel dimension, generates a two-dimensional heat map through aggregation operation, and outputs a spatial sparse tensor through generalized Gaussian distribution parameter estimation and dynamic threshold calculation; the spatial sparse tensor is interacted with a historical trend tensor, an exponential moving average update of deviation correction is performed, a residual tensor is calculated, and the updated trend tensor is synchronized to a communication opposite end; then, distribution modeling is performed on the residual tensor to determine a separation threshold, the residual space is divided, mixed precision coding is performed on the divided subspaces, and Lagrange is combined for joint optimization, and compressed data is output for transmission. The application can cooperatively eliminate redundancy from the space, time sequence and numerical value dimensions, significantly reduces the communication load, and guarantees the convergence performance and model precision of the distributed training.

Description

Technical Field

[0001] This invention relates to the field of distributed deep learning technology, and more specifically, to a spatiotemporal compression method and apparatus based on adaptive heatmap masking and mixed precision. Background Technology

[0002] Distributed deep learning, by allocating computational tasks to multiple client devices, improves model training efficiency while enhancing data privacy protection, and has been widely applied in scenarios such as federated learning and edge computing. However, in practical deployments, it faces a severe communication bottleneck: clients and servers need to frequently exchange high-dimensional feature gradients, and these gradient tensors typically contain millions or even tens of millions of floating-point parameters, resulting in an extremely large amount of data in a single communication, significantly increasing transmission latency and energy consumption. This problem is particularly prominent in low-bandwidth, high-latency, or resource-constrained network environments, severely limiting the scalability and practicality of distributed training. Especially when the client is a mobile device or IoT node, its limited computing, storage, and energy resources further exacerbate the urgent need for an efficient communication mechanism.

[0003] To alleviate the aforementioned challenges, existing technologies have proposed various gradient compression strategies. Among them, gradient sparsification methods such as Top-k achieve dimensionality reduction by retaining the k% of elements with the largest absolute values, or use random sparsification to discard some gradients with a fixed probability; quantization techniques such as SignSGD transmit only the gradient sign and supplement it with a scaling factor, while QSGD uses a fixed bit width (such as 4 bits or 8 bits) to discretize the gradient; low-rank decomposition methods such as PowerSGD use singular value decomposition to approximate the gradient as a low-dimensional matrix product; and differential coding methods such as EF-SGD are based on the temporal correlation of gradients, transmitting only the residual between the current round and the historical state, and combining error feedback to compensate for information loss.

[0004] While the aforementioned methods reduce communication overhead to some extent, they still have significant drawbacks. Top-k and randomized sparsity lack adaptive judgment of the semantic importance of gradients, easily leading to the accidental deletion of critical small gradients or the introduction of noise, which impairs model convergence and final performance. Fixed-width quantization struggles to adapt to the dynamic changes in gradient distribution during training; low-width quantization results in significant information loss due to insufficient accuracy in the early stages of training, while high-width quantization leads to low compression efficiency. Although low-rank decomposition can compress data, its high computational complexity makes it unsuitable for edge devices with limited computing power, and the low-rank assumption may not accurately characterize the complex structure of gradients, affecting the optimization trajectory. More critically, existing solutions often focus on a single dimension (such as spatial sparsity or numerical quantization only), lacking collaborative modeling and joint optimization of multi-dimensional information such as spatial redundancy, temporal correlation, and numerical statistical characteristics. This makes it difficult to achieve an ideal balance between high compression ratio and high fidelity, failing to meet the comprehensive requirements of communication efficiency, model accuracy, and system robustness in diverse distributed learning scenarios.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] The present invention aims to provide a spatiotemporal compression method, apparatus, device and medium based on adaptive heatmap mask and mixed precision, so as to solve the problems of huge communication overhead, high transmission latency and high power consumption of terminal devices caused by frequent transmission of high-dimensional feature gradients between clients and servers in existing distributed deep learning systems.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A spatiotemporal compression method based on adaptive heatmap masking and mixed precision is applied to communication compression between multiple clients and servers in a distributed learning system. Its features include:

[0009] S1, Obtain the original high-dimensional data; wherein, the original high-dimensional data includes the original features generated by each client when performing forward propagation calculation using the local dataset, and the original gradients sent down by the server after aggregation.

[0010] S2, the original high-dimensional data is projected along the channel dimension, and a two-dimensional heat map reflecting spatial significance is generated through aggregation operation. The generalized Gaussian distribution parameter is estimated and the dynamic threshold is calculated, and the spatial sparse tensor is output.

[0011] S3, Interact the spatial sparse tensor with the historical trend tensor, perform an exponential moving average update with deviation correction, calculate the residual tensor, and synchronize the updated trend tensor to the communication peer.

[0012] S4. Perform distribution modeling on the residual tensor to determine the separation threshold, and use the separation threshold to divide the residual space, so as to perform mixed precision encoding on the divided subspace, and perform joint optimization in combination with Lagrange, and output compressed data for transmission.

[0013] Preferably, S2 specifically comprises:

[0014] The original high-dimensional data is projected along the channel dimension, and the energy value of each spatial location is calculated by norm aggregation to obtain a two-dimensional heat map that reflects spatial significance.

[0015] The energy distribution of the two-dimensional heatmap is modeled as a generalized Gaussian distribution. After estimating the distribution parameters, the dynamic threshold is solved by integral constraint optimization to ensure the lower limit of information entropy and obtain the optimal cutoff threshold.

[0016] Based on the optimal truncation threshold, a binary mask is generated using an indicator function, and the original high-dimensional data is filtered by broadcast Hadamard product to obtain a denoised spatial sparse tensor.

[0017] Preferably, the energy value of the two-dimensional heatmap is used to quantify the semantic importance of each spatial location, and its expression is:

[0018] ;

[0019] in, Representing coordinates The spatial energy value at that location; The number of channels in the two-dimensional heatmap; In the t-th training round, the c-th channel is located at coordinates The original high-dimensional data values; It is the order of the norm; , These are the height and width of the two-dimensional heatmap, respectively.

[0020] The energy distribution of the two-dimensional heatmap is modeled as a generalized Gaussian distribution, and its expression is:

[0021] ;

[0022] in, Energy value The probability density function; These are scale parameters, shape parameters, and position parameters, respectively. It is a gamma function;

[0023] The expression for the optimal truncation threshold is:

[0024] ;

[0025] in, Let be the optimal cutoff threshold for the t-th training round; To minimize the threshold ; These are the scale parameter, shape parameter, and location parameter estimated based on the current sample, respectively. Information retention factor; The total information entropy of the two-dimensional heatmap; For energy value Integral; Energy value The probability density function; It is the set of positive real numbers; For threshold;

[0026] The expression for the spatially sparse tensor is:

[0027] ;

[0028] in, Let be the spatially sparse tensor after spatial filtering in the t-th round; This represents the original high-dimensional data from the t-th training round; For Hadamah accumulation; For broadcast operation; This is an indicator function.

[0029] Preferably, S3 specifically comprises:

[0030] The current spatially sparse tensor and the historical trend tensor maintained locally are obtained. An exponential moving average mechanism with deviation correction is executed to update the current trend tensor, which is then synchronized to the communicating peer to ensure consistency in trend prediction status between the two parties and avoid prediction deviations. The update formula is:

[0031] ;

[0032] in, For the trend tensor estimation of round t; This is the trend tensor from the previous round; Forgetting factor; Let be the spatially sparse tensor after spatial filtering in the t-th round;

[0033] The difference between the spatially sparse tensor and the historical trend tensor is calculated to obtain the residual tensor after eliminating temporal redundancy. The formula is as follows:

[0034] ;

[0035] in, This is the predicted residual tensor for the final transmission.

[0036] Preferably, it further includes: verifying the reduction effect of differential entropy based on the residual tensor and the spatial sparse tensor, so as to eliminate the redundancy of gradient in the temporal dimension, as expressed by:

[0037] ;

[0038] in, For the final transmitted prediction residual tensor; The differential entropy of a continuous random variable; for The variance; for The variance; Let be the spatially sparse tensor after spatial filtering in the t-th round; It is the base of the natural logarithm; This indicates that it is much smaller than.

[0039] Preferably, S4 specifically comprises:

[0040] First, the numerical distribution of the residual tensor is modeled, and its statistical characteristics of peaked and heavy-tailed states are described using a probability density function, the formula of which is:

[0041] ;

[0042] in, For residual tensor elements in The probability density function; These are scale parameters, shape parameters, and position parameters, respectively. It is a gamma function; For residual tensor Elements in;

[0043] Then, based on the tail features of the residual tensor after modeling, the separation threshold is calculated using the inverse function of the cumulative distribution function, expressed as:

[0044] ;

[0045] ;

[0046] in, It is the cumulative distribution function; The separation threshold; It is the inverse function of the cumulative distribution function; To control the preset proportion of the main value; These are the scale parameter, shape parameter, and location parameter estimated based on the current sample, respectively. Indicates to Integral;

[0047] The residual space is divided into an outlier subspace and a main subspace using the separation threshold, as expressed by:

[0048] ;

[0049] ;

[0050] in, It is the set of outliers; For the main value set;

[0051] High-precision encoding is applied to the outlier subspace, and low-precision uniform quantization is applied to the main subspace to calculate total quantization distortion, expressed as:

[0052] ;

[0053] in, To quantify the distortion; The quantization step size is determined by the quantization bit width. The step size is The uniform quantization function;

[0054] Construct a Lagrange cost function, jointly optimize the separation threshold and the quantization bit width of the main value, solve for the Pareto optimal solution, and output compressed data. The expression is:

[0055] ;

[0056] in, Let Lagrange cost function be used. This indicates finding the minimum value; Low-precision quantization bit width for the main value; Represents the mathematical expectation; The square of the norm; The reconstructed residual tensor; These are Lagrange multipliers used to balance the control rate distortion. High-precision quantization bit width for outliers.

[0057] Preferably, the method further includes: the communication peer receiving the compressed data and decompressing it, specifically:

[0058] Based on the separation threshold of the compression end synchronization, outlier data and main data are distinguished in the compressed data;

[0059] The outlier data is subjected to high-precision decoding to recover the outlier subspace, and the main data is subjected to low-precision uniform dequantization to recover the main subspace.

[0060] The outlier subspace and the main subspace are merged to obtain the residual tensor;

[0061] Based on the residual tensor and the locally maintained historical trend tensor, the spatially sparse tensor of the compression phase is restored, and the local trend tensor is updated according to the exponential moving average.

[0062] Based on the binary mask synchronized with the compressed end of the spatial sparse tensor, the positions in the spatial sparse tensor that were filtered by the mask are filled in, the complete spatial dimension of the data is restored, and the original high-dimensional data is output.

[0063] The present invention also provides a spatiotemporal compression device based on adaptive heatmap masking and mixed precision, comprising:

[0064] The acquisition module is used to acquire raw high-dimensional data; wherein, the raw high-dimensional data includes the raw features generated by each client when performing forward propagation calculation using the local dataset, and the raw gradients sent down by the server after aggregation.

[0065] The adaptive heatmap masking module is used to project the original high-dimensional data along the channel dimension, generate a two-dimensional heatmap reflecting spatial saliency through aggregation operation, and perform generalized Gaussian distribution parameter estimation and dynamic threshold calculation to output a spatially sparse tensor.

[0066] The trend-aware residual compensation module is used to interact with the spatial sparse tensor and the historical trend tensor, perform an exponential moving average update with deviation correction, calculate the residual tensor, and synchronize the updated trend tensor to the communication peer.

[0067] The outlier separation and hybrid quantization module is used to perform distribution modeling on the residual tensor to determine the separation threshold, and to divide the residual space using the separation threshold to perform hybrid precision encoding on the divided subspace. At the same time, it combines Lagrange multiplication for joint optimization and outputs compressed data for transmission.

[0068] The present invention also provides a spatiotemporal compression device based on adaptive heatmap masking and mixed precision, including a processor and a memory. The memory stores a computer program that can be executed by the processor to implement the spatiotemporal compression method based on adaptive heatmap masking and mixed precision as described above.

[0069] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device in which the computer-readable storage medium resides, implement the spatiotemporal compression method based on adaptive heatmap masking and mixed precision as described above.

[0070] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0071] This invention reduces the amount of data to be processed by sparsifying the original high-dimensional data based on energy saliency, eliminating background regions that do not carry key semantic information.

[0072] This invention utilizes a synchronously maintained trend tensor to extract residuals, eliminating temporal redundancy between consecutive rounds, making the transmitted data closer to a white noise distribution, and reducing information entropy.

[0073] This invention implements outlier separation and mixed-precision encoding on residual data, prioritizing the accuracy of key gradient information under a limited bit budget, and avoiding the accuracy collapse caused by uniform quantization.

[0074] This invention eliminates redundancy across different dimensions without information coupling loss, thus achieving a compression efficiency far exceeding that of single-dimensional compression methods. At the computational level, the spatial filtering part avoids global sorting, requiring only linear complexity channel aggregation and threshold comparison; the temporal compensation part involves only tensor addition / subtraction and scalar multiplication; the distribution modeling and threshold solving in the quantization part can be completed through lightweight statistical estimation, making the overall algorithm complexity suitable for resource-constrained edge devices. Regarding training stability, the high-precision preservation of outliers ensures the accuracy of the optimization direction, and the momentum mechanism of bias correction suppresses gradient oscillations, making the distributed training process more robust. Attached Figure Description

[0075] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0076] Figure 1 This is a flowchart illustrating a spatiotemporal compression method based on adaptive heatmap masking and mixed precision, as provided in Example 1.

[0077] Figure 2 This is a structural framework diagram of a spatiotemporal compression method based on adaptive heatmap masking and mixed precision, provided in Example 1.

[0078] Figure 3 This is a schematic diagram of a spatiotemporal compression device based on adaptive heatmap masking and mixing accuracy, provided in Embodiment 2.

[0079] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0081] Example 1

[0082] Embodiment 1 of the present invention provides a spatiotemporal compression method based on adaptive heatmap masking and mixed precision, which can be implemented by a spatiotemporal compression device based on adaptive heatmap masking and mixed precision (hereinafter referred to as compression device), specifically, executed by one or more processors within the compression device.

[0083] In this embodiment, the compression device may be an electronic device equipped with a processor, which carries a computer program for the spatiotemporal compression method based on adaptive heatmap masking and mixed precision, and the computer program can be executed, such as a computer, smartphone, smart tablet, workstation, etc., which are not limited here.

[0084] To address the critical communication bandwidth limitation issue in distributed edge intelligent systems, this embodiment proposes a spatiotemporal compression framework (AHM-Hybrid) based on adaptive heatmap masking and mixed precision. Figures 1-2 As shown, a spatiotemporal compression method based on adaptive heatmap masking and mixed precision is applied to the communication compression between multiple clients and servers in a distributed learning system, which includes steps S1 to S4.

[0085] S1, Obtain the original high-dimensional data; wherein, the original high-dimensional data includes the original features generated by each client when performing forward propagation calculation using the local dataset, and the original gradients sent down by the server after aggregation.

[0086] The purpose of this step is to provide a unified input data source for subsequent spatiotemporal compression, covering the original features generated by local training on the client and the original gradients aggregated and distributed by the server, thus adapting to distributed learning scenarios.

[0087] In practical applications of distributed training, such as when performing personalized federated learning tasks on mobile devices, the client first uses the local dataset for model forward propagation computation. When computation reaches a specific network layer, a high-dimensional intermediate feature tensor is generated, which is one of the core processing objects in the subsequent spatiotemporal compression process.

[0088] After receiving compressed data uploaded by each client, the server first recovers the original local gradients through the corresponding decompression process. Then, it uses aggregation algorithms such as federated averaging to aggregate the original local gradients of all clients, obtaining a high-dimensional global unified gradient. This global gradient is then distributed to each client to synchronously update the client's local model parameters, achieving coordinated advancement of global training progress and personalized local training. This high-dimensional global unified gradient is also one of the core processing objects in the spatiotemporal compression process.

[0089] S2, the original high-dimensional data is projected along the channel dimension, and a two-dimensional heat map reflecting spatial saliency is generated through aggregation operation. The generalized Gaussian distribution parameter is estimated and the dynamic threshold is calculated, and the spatial sparse tensor is output.

[0090] During the training of deep convolutional neural networks, feature maps often exhibit significant spatial sparsity. To maximize the preservation of key semantic information while maintaining low computational cost, we model the spatial filtering problem as an energy maximization problem under sparsity constraints.

[0091] Specifically, this includes steps S21 to S23.

[0092] S21, the original high-dimensional data is projected along the channel dimension, and the energy value of each spatial location is calculated by norm aggregation to obtain a two-dimensional heat map reflecting spatial significance.

[0093] The purpose of this step is to project high-dimensional data along the channel dimension, calculate the energy value of each spatial location through norm aggregation, quantify its semantic importance, and identify key semantic regions.

[0094] The energy value of the two-dimensional heatmap is used to quantify the semantic importance of each spatial location, and its expression is as follows:

[0095] ;

[0096] in, Representing coordinates The spatial energy value at that location; The number of channels in the two-dimensional heatmap; In the t-th training round, the c-th channel is located at coordinates The original high-dimensional data values; It is the order of the norm; , These are the height and width of the two-dimensional heat map, respectively.

[0097] Empirical studies show that the energy distribution of deep feature maps follows a generalized Gaussian distribution, and its probability density function has a variable shape parameter, which can flexibly describe various forms from Laplace distribution to Gaussian distribution.

[0098] The energy distribution of the two-dimensional heatmap is modeled as a generalized Gaussian distribution, and its expression is:

[0099] ;

[0100] in, Energy value The probability density function; These are scale parameters, shape parameters, and position parameters, respectively. This is a gamma function.

[0101] This step reduces the dimensionality of high-dimensional data to a two-dimensional heatmap, which can significantly reduce the computational complexity of subsequent processing. By accurately locating semantically key regions, it provides an objective basis for subsequent spatial filtering and avoids blind compression.

[0102] Compared to fixed distribution assumptions (such as Gaussian distribution), the generalized Gaussian distribution used in this step can flexibly adapt to the energy distribution of different tasks and different training stages, improve the accuracy of distribution modeling, provide a computable probabilistic framework for subsequent threshold optimization, and avoid the subjectivity of empirical threshold setting.

[0103] S22, the energy distribution of the two-dimensional heatmap is modeled as a generalized Gaussian distribution. After estimating the distribution parameters, the dynamic threshold is solved through integral constraint optimization to ensure the lower limit of information entropy, and the optimal truncation threshold expression is obtained as follows:

[0104] ;

[0105] in, Let be the optimal cutoff threshold for the t-th training round; To minimize the threshold ; These are the scale parameter, shape parameter, and location parameter estimated based on the current sample, respectively. Information retention factor; The total information entropy of the two-dimensional heatmap; For energy value Integral; Energy value The probability density function; It is the set of positive real numbers; The threshold value is used.

[0106] This step balances "key information retention" and "reduction of computational load" by solving for the minimum truncation threshold while ensuring that the information entropy of the retained region is not lower than the preset lower limit.

[0107] S23, based on the optimal truncation threshold, a binary mask is generated using an indicator function, and the original high-dimensional data is filtered through a broadcast Hadamard product to obtain a denoised spatial sparse tensor, the expression of which is:

[0108] ;

[0109] in, Let be the spatially sparse tensor after spatial filtering in the t-th round; This represents the original high-dimensional data from the t-th training round; For Hadamah accumulation; For broadcast operation; This is an indicator function.

[0110] This step uses a truncation threshold to generate a binary mask, filtering out background noise and redundant information in the spatial dimension to obtain a denoised spatial sparse tensor. This significantly reduces the spatial dimension of the data and reduces the computational overhead of subsequent temporal processing and quantization. Moreover, it can accurately preserve semantic key regions, avoid precision loss, and provide a "cleaner" input for temporal redundancy elimination.

[0111] S3, the spatial sparse tensor interacts with the historical trend tensor, performs an exponential moving average update with deviation correction, calculates the residual tensor, and synchronizes the updated trend tensor to the communication peer.

[0112] The spatially sparse tensor, after spatial filtering, still exhibits strong autocorrelation across consecutive time steps. To eliminate this temporal redundancy, this step aims to construct an unbiased linear predictor. The temporal evolution of the gradient is treated as a stochastic process with a drift term, and a bias-corrected exponential moving average mechanism is introduced to recursively estimate its first-order moment trend.

[0113] Specifically, the current spatially sparse tensor and the locally stored historical trend tensor are obtained, an exponential moving average mechanism for deviation correction is executed, the current trend tensor is updated, and synchronized to the communication peer to ensure that the trend prediction status of both parties is consistent and to avoid prediction deviation; the update formula is:

[0114] ;

[0115] in, For the trend tensor estimation of round t; This is the trend tensor from the previous round; Forgetting factor; Let be the spatially sparse tensor after spatial filtering in round t.

[0116] The bias correction mechanism can effectively track the temporal trend of the gradient and avoid forgetting historical information or overfitting; the forgetting factor can flexibly adjust the dependence on historical information and adapt to the temporal evolution characteristics of different tasks.

[0117] S32, calculate the difference between the spatially sparse tensor and the historical trend tensor to obtain the residual tensor with temporal redundancy eliminated, using the following formula:

[0118] ;

[0119] in, This is the predicted residual tensor for the final transmission.

[0120] This step removes the deterministic temporal trends of the gradients, retaining only random fluctuations, thus eliminating redundant information in the temporal dimension. The variance of the residual tensor is significantly reduced, and its distribution becomes more concentrated, greatly improving the efficiency of subsequent quantization and compression. After removing temporal redundancy, the transmitted data contains only unpredictable random fluctuations, reducing redundant bits in communication.

[0121] This round of communication only transmits the difference between the current observation and the trend forecast. From an information theory perspective, the core benefit of this operation lies in reducing the differential entropy of the random variable. According to the maximum entropy principle, for distributions with the same variance, the Gaussian distribution has the maximum entropy. By removing the deterministic trend component, the variance of the residuals is significantly reduced, resulting in a strict decrease in differential entropy.

[0122] In a preferred embodiment, the method further includes: verifying the reduction effect of differential entropy based on the residual tensor and the spatially sparse tensor, so as to eliminate the redundancy of gradient in the temporal dimension, as expressed by:

[0123] ;

[0124] in, For the final transmitted prediction residual tensor; The differential entropy of a continuous random variable; for The variance; for The variance; Let be the spatially sparse tensor after spatial filtering in the t-th round; It is the base of the natural logarithm; This indicates that it is much smaller than.

[0125] This step provides quantitative metrics to demonstrate the effectiveness of temporal redundancy elimination, offering theoretical support for improved compression efficiency and ensuring a significant reduction in the differential entropy of the residual tensor, thus laying the foundation for subsequent "low bit rate, low distortion" quantization compression.

[0126] S4. Perform distribution modeling on the residual tensor to determine the separation threshold, and use the separation threshold to divide the residual space, so as to perform mixed precision encoding on the divided subspace, and perform joint optimization in combination with Lagrange, and output compressed data for transmission.

[0127] The numerical distribution of residual tensors typically exhibits a peaked, heavy-tailed characteristic. To minimize the reconstruction error under finite bandwidth constraints, the quantization problem is modeled as a rate-distortion optimization problem.

[0128] Specifically, this includes steps S41 to S45.

[0129] S41, First, the numerical distribution of the residual tensor is modeled, and its peaked and heavy-tailed statistical characteristics are described using a probability density function, the formula of which is:

[0130] ;

[0131] in, For residual tensor elements in The probability density function; These are scale parameters, shape parameters, and position parameters, respectively. It is a gamma function; For residual tensor The elements in.

[0132] This step accurately identifies outliers (a few large values) and main values ​​(a large number of small values) by characterizing the statistical features of the residual tensor’s “peak and heavy tail”, and accurately captures the distribution characteristics of the residuals. This provides an objective basis for heterogeneous precision coding, avoids using uniform precision for all data, and achieves differentiated processing of “high precision for key information and low precision for redundant information”.

[0133] S42, then based on the tail features of the residual tensor after modeling, the separation threshold is calculated through the inverse function of the cumulative distribution function, expressed as:

[0134] ;

[0135] ;

[0136] in, It is the cumulative distribution function; The separation threshold; It is the inverse function of the cumulative distribution function; To control the preset proportion of the main value; These are the scale parameter, shape parameter, and location parameter estimated based on the current sample, respectively. Indicates to Find the integral.

[0137] Based on the tail features of the residual distribution, the separation threshold is automatically determined by the inverse function of the cumulative distribution function, accurately separating outliers from main values. This step can automatically adapt to the tail features of different residual distributions, avoid the bias of manual thresholding, ensure reasonable division between outliers and main values, and balance reconstruction accuracy and compression ratio.

[0138] S43, using the separation threshold, the residual space is divided into an outlier subspace and a main value subspace, as expressed in:

[0139] ;

[0140] ;

[0141] in, It is the set of outliers; The set of main values.

[0142] The residual space is split into two non-overlapping subspaces, providing a clear processing object for heterogeneous precision coding.

[0143] S44, high-precision encoding is applied to the outlier subspace, and low-precision uniform quantization is applied to the main subspace to calculate total quantization distortion, expressed as:

[0144] ;

[0145] in, To quantify the distortion; The quantization step size is determined by the quantization bit width. The step size is The uniform quantization function.

[0146] The goal of this step is to find the optimal quantization strategy that minimizes the total mean square quantization error. By using high-precision encoding for outliers (which have a significant impact on gradient accuracy) and low-precision uniform quantization for main values ​​(which account for a large proportion but have a small impact), a balance is struck between reconstruction accuracy and transmission rate, avoiding the loss of critical gradient information and improving communication efficiency.

[0147] S45, Construct the Lagrange cost function, jointly optimize the separation threshold and the quantization bit width of the main value, solve for the Pareto optimal solution, and output the compressed data. The expression is:

[0148] ;

[0149] in, Let Lagrange cost function be used. This indicates finding the minimum value; Low-precision quantization bit width for the main value; Represents the mathematical expectation; The square of the norm; The reconstructed residual tensor; These are Lagrange multipliers used to balance the control rate distortion. High-precision quantization bit width for outliers.

[0150] To balance transmission bit rate and distortion, a Lagrange cost function is constructed, and the separation threshold and quantization bit width are jointly optimized to find the Pareto optimal solution. This minimizes the weighted sum of "quantization distortion + transmission bit rate", avoiding the limitations of single-objective optimization. By considering both accuracy and bit rate, the optimal compression strategy is found.

[0151] In another preferred embodiment, the method further includes: the communication peer receiving the compressed data and decompressing it, specifically:

[0152] Based on the separation threshold of the compression end synchronization, outlier data and main data are distinguished in the compressed data;

[0153] The outlier data is subjected to high-precision decoding to recover the outlier subspace, and the main data is subjected to low-precision uniform dequantization to recover the main subspace.

[0154] The outlier subspace and the main subspace are merged to obtain the residual tensor;

[0155] Based on the residual tensor and the locally maintained historical trend tensor, the spatially sparse tensor of the compression phase is restored, and the local trend tensor is updated according to the exponential moving average.

[0156] Based on the binary mask synchronized with the compressed end of the spatial sparse tensor, the positions in the spatial sparse tensor that were filtered by the mask are filled in, the complete spatial dimension of the data is restored, and the original high-dimensional data is output.

[0157] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0158] (1) Extreme compression performance through multi-dimensional collaboration. This invention breaks through the limitations of traditional compression methods that only focus on a single dimension, and innovatively eliminates redundancy through three orthogonal dimensions: spatial sparsity, temporal correlation, and numerical statistics. Spatial background noise is accurately removed by channel energy aggregation; temporal inertial components are eliminated by unbiased trend prediction; and numerical redundancy is minimized by heterogeneous precision coding. This three-in-one compression mechanism enables this invention to achieve a compression ratio significantly better than single sparsification or single quantization methods while maintaining model accuracy, greatly reducing communication overhead in edge networks.

[0159] (2) Low computational complexity and edge-friendliness. Addressing the pain point of limited computing resources on edge devices, this invention designs an algorithm flow with extremely low complexity. This invention abandons the global element sorting operation commonly used in existing sparsity techniques (which typically has logarithmic linear complexity), and instead adopts a linear channel aggregation and threshold truncation strategy, significantly reducing computational latency. Simultaneously, this invention only involves simple tensor addition and subtraction and moving average operations, quickly achieving data distribution through statistical quantiles. The lightweight design of the entire framework makes it very suitable for deployment on IoT nodes or mobile terminals with limited computing power, meeting the needs of real-time training.

[0160] (3) High-fidelity gradient information preservation and convergence stability. This invention effectively solves the contradiction between compression ratio and model performance through a refined information preservation strategy. This invention adopts an outlier separation strategy to maintain a high-precision floating-point representation of the "peak" data that carries key gradient information, avoiding the precision collapse problem caused by excessive dynamic range in traditional unified quantization. In addition, by introducing a momentum mechanism for bias correction, the random oscillation of gradients is smoothed, making the transmitted residual data statistically closer to the zero-mean distribution, which not only reduces quantization noise, but also improves the convergence stability and robustness of the distributed training process.

[0161] (4) Adaptive statistical perception capability. This invention possesses powerful data distribution adaptive capability, capable of coping with changes in data features at different training stages and network layers. This invention utilizes the generalized Gaussian distribution assumption to dynamically adjust the spatial truncation threshold, automatically adapting to changes in feature map sparsity; and optimizes the dynamic balance between outlier ratio and quantization bit width through Lagrange rate distortion optimization. This adaptive mechanism ensures that the system can automatically find the optimal balance between communication bandwidth and model accuracy between the early stages (requiring high precision) and later stages (requiring high compression) of training without frequent manual adjustment of hyperparameters.

[0162] Example 2

[0163] like Figure 3 As shown, the second embodiment of the present invention also provides a spatiotemporal compression device based on adaptive heatmap masking and mixing precision, comprising:

[0164] The acquisition module is used to acquire raw high-dimensional data; wherein, the raw high-dimensional data includes the raw features generated by each client when performing forward propagation calculation using the local dataset, and the raw gradients sent down by the server after aggregation.

[0165] The adaptive heatmap masking module is used to project the original high-dimensional data along the channel dimension, generate a two-dimensional heatmap reflecting spatial saliency through aggregation operation, and perform generalized Gaussian distribution parameter estimation and dynamic threshold calculation to output a spatially sparse tensor.

[0166] The trend-aware residual compensation module is used to interact with the spatial sparse tensor and the historical trend tensor, perform an exponential moving average update with deviation correction, calculate the residual tensor, and synchronize the updated trend tensor to the communication peer.

[0167] The outlier separation and hybrid quantization module is used to perform distribution modeling on the residual tensor to determine the separation threshold, and to divide the residual space using the separation threshold to perform mixed precision encoding on the divided subspace. At the same time, it combines Lagrange multiplication for joint optimization and outputs compressed data for transmission.

[0168] Example 3

[0169] The third embodiment of the present invention also provides a spatiotemporal compression device based on adaptive heatmap masking and mixed precision, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the spatiotemporal compression method based on adaptive heatmap masking and mixed precision as described above.

[0170] Example 4

[0171] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, they implement the spatiotemporal compression method based on adaptive heatmap mask and mixed precision as described above.

[0172] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A spatiotemporal compression method based on adaptive heatmap masking and mixed precision, applied to communication compression between multiple clients and servers in a distributed learning system, characterized in that... include: S1, Obtain the original high-dimensional data; wherein, the original high-dimensional data includes the original features generated by each client when performing forward propagation calculation using the local dataset, and the original gradients sent down by the server after aggregation. S2, the original high-dimensional data is projected along the channel dimension, and a two-dimensional heat map reflecting spatial significance is generated through aggregation operation. The generalized Gaussian distribution parameter is estimated and the dynamic threshold is calculated, and the spatial sparse tensor is output. S3, Interact the spatial sparse tensor with the historical trend tensor, perform an exponential moving average update with deviation correction, calculate the residual tensor, and synchronize the updated trend tensor to the communication peer. S4, perform distribution modeling on the residual tensor to determine the separation threshold, and use the separation threshold to divide the residual space, so as to perform mixed precision encoding on the divided subspace, and at the same time perform joint optimization in combination with Lagrange, and output compressed data for transmission; Specifically, S4 is: First, the numerical distribution of the residual tensor is modeled, and its statistical characteristics of peaked and heavy-tailed states are described using a probability density function, the formula of which is: ； in, For residual tensor elements in The probability density function; These are scale parameters, shape parameters, and position parameters, respectively. It is a gamma function; For residual tensor Elements in; Then, based on the tail features of the residual tensor after modeling, the separation threshold is calculated using the inverse function of the cumulative distribution function, expressed as: ； ； in, It is the cumulative distribution function; The separation threshold; It is the inverse function of the cumulative distribution function; To control the preset proportion of the main value; These are the scale parameter, shape parameter, and location parameter estimated based on the current sample, respectively. Indicates to Integral; The residual space is divided into an outlier subspace and a main subspace using the separation threshold, as expressed by: ； ； in, It is the set of outliers; For the main value set; High-precision encoding is applied to the outlier subspace, and low-precision uniform quantization is applied to the main subspace to calculate total quantization distortion, expressed as: ； in, To quantify the distortion; The quantization step size is determined by the quantization bit width. The step size is The uniform quantization function; Construct a Lagrange cost function, jointly optimize the separation threshold and the quantization bit width of the main value, solve for the Pareto optimal solution, and output compressed data. The expression is: ； in, Let Lagrange cost function be used. This indicates finding the minimum value; Low-precision quantization bit width for the main value; Represents the mathematical expectation; The square of the norm; The reconstructed residual tensor; These are Lagrange multipliers used to balance the control rate distortion. High-precision quantization bit width for outliers.

2. The spatiotemporal compression method based on adaptive heatmap masking and mixed precision according to claim 1, characterized in that... S2 specifically refers to: The original high-dimensional data is projected along the channel dimension, and the energy value of each spatial location is calculated by norm aggregation to obtain a two-dimensional heat map that reflects spatial significance. The energy distribution of the two-dimensional heatmap is modeled as a generalized Gaussian distribution. After estimating the distribution parameters, the dynamic threshold is solved by integral constraint optimization to ensure the lower limit of information entropy and obtain the optimal cutoff threshold. Based on the optimal truncation threshold, a binary mask is generated using an indicator function, and the original high-dimensional data is filtered by broadcast Hadamard product to obtain a denoised spatial sparse tensor.

3. The spatiotemporal compression method based on adaptive heatmap masking and mixed precision according to claim 2, characterized in that... The energy value of the two-dimensional heatmap is used to quantify the semantic importance of each spatial location, and its expression is: ； in, Representing coordinates The spatial energy value at that location; The number of channels in the two-dimensional heatmap; In the t-th training round, the c-th channel is located at coordinates The original high-dimensional data values; It is the order of the norm; , These are the height and width of the two-dimensional heatmap, respectively. The energy distribution of the two-dimensional heatmap is modeled as a generalized Gaussian distribution, and its expression is: ； in, Energy value The probability density function; These are scale parameters, shape parameters, and position parameters, respectively. It is a gamma function; The expression for the optimal truncation threshold is: ； in, Let be the optimal cutoff threshold for the t-th training round; To minimize the threshold ; These are the scale parameter, shape parameter, and location parameter estimated based on the current sample, respectively. Information retention factor; The total information entropy of the two-dimensional heatmap; For energy value Integral; Energy value The probability density function; It is the set of positive real numbers; For threshold; The expression for the spatially sparse tensor is: ； in, Let be the spatially sparse tensor after spatial filtering in the t-th round; This represents the original high-dimensional data from the t-th training round; For Hadamah accumulation; For broadcast operation; This is an indicator function.

4. The spatiotemporal compression method based on adaptive heatmap masking and mixed precision according to claim 1, characterized in that... S3 specifically refers to: The current spatially sparse tensor and the historical trend tensor maintained locally are obtained. An exponential moving average mechanism with deviation correction is executed to update the current trend tensor, which is then synchronized to the communicating peer to ensure consistency in trend prediction status between the two parties and avoid prediction deviations. The update formula is: ； in, For the trend tensor estimation of round t; This is the trend tensor from the previous round; Forgetting factor; Let be the spatially sparse tensor after spatial filtering in the t-th round; The difference between the spatially sparse tensor and the historical trend tensor is calculated to obtain the residual tensor after eliminating temporal redundancy. The formula is as follows: ； in, This is the predicted residual tensor for the final transmission.

5. A spatiotemporal compression method based on adaptive heatmap masking and mixed precision according to claim 4, characterized in that... It also includes: verifying the reduction effect of differential entropy based on the residual tensor and the spatial sparse tensor, so as to eliminate the redundancy of gradient in the temporal dimension, as expressed by: ； in, For the final transmitted prediction residual tensor; The differential entropy of a continuous random variable; for The variance; for The variance; Let be the spatially sparse tensor after spatial filtering in the t-th round; It is the base of the natural logarithm; This indicates that it is much smaller than.

6. The spatiotemporal compression method based on adaptive heatmap masking and mixed precision according to claim 1, characterized in that... It also includes: the communication peer receiving the compressed data and decompressing it, specifically: Based on the separation threshold of the compression end synchronization, outlier data and main data are distinguished in the compressed data; The outlier data is subjected to high-precision decoding to recover the outlier subspace, and the main data is subjected to low-precision uniform dequantization to recover the main subspace. The outlier subspace and the main subspace are merged to obtain the residual tensor; Based on the residual tensor and the locally maintained historical trend tensor, the spatially sparse tensor of the compression phase is restored, and the local trend tensor is updated according to the exponential moving average. Based on the binary mask synchronized with the compressed end of the spatial sparse tensor, the positions in the spatial sparse tensor that were filtered by the mask are filled in, the complete spatial dimension of the data is restored, and the original high-dimensional data is output.

7. A spatiotemporal compression device based on adaptive heatmap masking and mixed precision, used to implement the spatiotemporal compression method based on adaptive heatmap masking and mixed precision as described in any one of claims 1-6, characterized in that, include: The acquisition module is used to acquire raw high-dimensional data; wherein, the raw high-dimensional data includes the raw features generated by each client when performing forward propagation calculation using the local dataset, and the raw gradients sent down by the server after aggregation. The adaptive heatmap masking module is used to project the original high-dimensional data along the channel dimension, generate a two-dimensional heatmap reflecting spatial saliency through aggregation operation, and perform generalized Gaussian distribution parameter estimation and dynamic threshold calculation to output a spatially sparse tensor. The trend-aware residual compensation module is used to interact with the spatial sparse tensor and the historical trend tensor, perform an exponential moving average update with deviation correction, calculate the residual tensor, and synchronize the updated trend tensor to the communication peer. The outlier separation and hybrid quantization module is used to perform distribution modeling on the residual tensor to determine the separation threshold, and to divide the residual space using the separation threshold to perform hybrid precision encoding on the divided subspace. At the same time, it combines Lagrange multiplication for joint optimization and outputs compressed data for transmission.

Images