Bandwidth-distortion adaptive multi-layer scalable feature compressor and method, system thereof

Abstract

The application provides a bandwidth distortion adaptive multi-layer scalable feature compressor and a method and system thereof, and relates to the technical field of machine learning.The application dynamically decides the number of layers, the bit width distribution of each layer and the slice size according to the real-time network state training stage and precision requirements through a code rate distortion joint controller; maps input features to a subspace with more concentrated energy and clusters and groups feature channels through a layered learnable orthogonal transform module; allocates different bit widths to coefficients based on importance through a non-uniform quantization module; dynamically adjusts the slice size and the slice rhythm according to the round-trip time and the jitter degree through an RTT adaptive scheduling module; records the cumulative reconstruction error and performs residual coding through an error tracking compensation module; and periodically transmits the anchor layer to clear the cumulative error through a key frame synchronization module.The application significantly improves the robustness, convergence stability and communication efficiency of distributed training in a complex network environment.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning technology, and more specifically, to a bandwidth distortion adaptive multilayer scalable feature compressor and its method and system. Background Technology

[0002] In distributed training scenarios such as federated learning and edge intelligence, balancing communication efficiency and model accuracy is a key challenge. Due to the prevalence of link bandwidth fluctuations and latency constraints, the accuracy requirements vary significantly across different training stages. For example, high reconstruction quality is needed in the early stages of training to ensure stable model convergence, while compression efficiency and throughput are emphasized in the mid-to-late stages, allowing for moderate distortion. However, most existing technologies employ fixed-bit-width or single-layer compression methods, making it difficult to dynamically adjust compression strategies based on task stage and link status. Fixed-bit-width methods may lead to resource waste when bandwidth is abundant, while under-transmission or retransmission can cause low communication efficiency or even irreversible accuracy loss when bandwidth is limited. Single-layer coding methods lack fine-grained quality and bit control capabilities, failing to meet the real-time and accuracy requirements of complex network environments.

[0003] To alleviate communication bottlenecks in distributed learning, researchers have proposed various gradient compression methods. These methods mainly include gradient sparsification, quantization, low-rank decomposition, and differential coding. Gradient sparsification reduces the transmission dimensionality by retaining important gradient elements or randomly discarding some elements, but it lacks adaptability to the training phase and task characteristics, potentially leading to the loss of key information or the introduction of excessive noise. Quantization reduces communication overhead by decreasing the number of bits required to represent the gradient, but fixed-bit-width quantization cannot adapt to dynamic changes in gradient distribution; low bit width may result in insufficient information retention, while high bit width is insufficient for low-bandwidth scenarios. Low-rank decomposition methods rely on singular value decomposition to approximate high-dimensional gradient tensors as low-rank matrix products, thereby reducing communication costs, but its computational overhead is high, and its ability to express complex gradient change patterns is limited. Differential coding methods utilize the temporal correlation of gradients to transmit only the difference and combine it with an error feedback mechanism to compensate for information loss, but its stability and performance remain unsatisfactory in environments with large gradient fluctuations or weak networks.

[0004] In general, existing technologies mostly focus on a single compression dimension, such as sparsity, quantization, or low-rank decomposition, lacking the ability to jointly optimize from multiple perspectives, including spatial, frequency, and time series data. This limitation makes it difficult to achieve an ideal trade-off between compression ratio and information preservation, failing to simultaneously achieve high compression ratio, low latency, and high accuracy. Furthermore, existing methods are not sufficiently applicable to diverse distributed learning scenarios, making it difficult to meet the practical needs of complex environments such as low bandwidth, high latency, and high jitter.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] The present invention aims to provide a bandwidth distortion adaptive multilayer scalable feature compressor and its method and system to solve the problem of difficulty in balancing communication efficiency and model accuracy in existing distributed training.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A bandwidth-distortion adaptive multilayer scalable feature compressor is applied between the client and server sides of a distributed machine learning system to compress the raw data acquired from the client or server side and output compressed data. It includes a rate-distortion joint controller, a hierarchical learnable orthogonal transformation module, a non-uniform quantization module, an RTT adaptive scheduling module, and an error tracking compensation module.

[0009] The rate-distortion joint controller is used to dynamically decide the total number of layers, the bit width allocation of each layer, and the fragment size based on the real-time network status, training stage, and accuracy requirements, so as to dynamically adjust the input raw data and minimize the reconstruction distortion between the raw data and the reconstructed data; wherein the reconstructed data is obtained by decoding and reconstructing the compressed data by the decoder at the receiving end; the network status includes available bandwidth, round-trip time, and jitter.

[0010] The hierarchical learnable orthogonal transformation module is used to map the original data to a more energy-concentrated subspace and cluster the feature channels. During data transmission, the base layer only carries the center coefficients and low-frequency components of each group, while the enhancement layer encodes the high-frequency details and deviation parts of each group layer by layer, performing progressive data reconstruction from coarse to fine, to obtain transmittable hierarchical data.

[0011] The non-uniform quantization module is used to calculate importance scores based on the transformed hierarchical data, so as to dynamically allocate bit width;

[0012] The RTT adaptive scheduling module is used to dynamically adjust the fragment size and fragmentation interval according to the real-time network status.

[0013] The error tracking and compensation module is used to calculate the cumulative reconstruction error of each round based on the residual encoding of the enhancement layer and dynamically adjust the bit width allocation of the next round to obtain compressed data.

[0014] Preferably, the raw data includes: raw features obtained from the client, or global gradients obtained from the server;

[0015] The rate-distortion joint controller includes a network state awareness submodule, a training phase awareness submodule, and an accuracy requirement submodule.

[0016] The network state awareness submodule calculates the effective available bandwidth by monitoring available bandwidth, round-trip time, and jitter, using the following formula:

[0017] ;

[0018] in, Effective available bandwidth; For real-time monitoring of available bandwidth; RTT stands for Round Trip Time. For RTT jitter; This is a conservative coefficient used to address protocol overhead and network fluctuation redundancy.

[0019] The training phase perception submodule is used to calculate the distortion weight coefficient based on the current training round and the total number of rounds, using the following formula:

[0020] ;

[0021] in, The distortion weight coefficient for the current training round t; Total number of rounds; , These represent the maximum and minimum distortion weight coefficients, respectively; The decay rate parameter is used to control the transition speed from high precision to high compression.

[0022] The accuracy requirement submodule is used to minimize the weighted reconstruction distortion of the original data and the reconstructed data by jointly optimizing the total number of layers, the bit width of each layer, and the size of each fragment, based on the available bandwidth and distortion weight coefficient, under a preset latency budget constraint. The objective function formula for the joint optimization is:

[0023] ;

[0024] in, This represents the total number of layers; For the first Layer width; This refers to the size of the slice; Distortion due to weighted reconstruction; For the first Layer weights; The original data; For the first The reconstructed data corresponding to the layer; It is an L2 norm; This indicates a minimize operation;

[0025] The constraints are that the total bit rate does not exceed the bandwidth-delay product, and the actual transmission time does not exceed the delay budget. The formula is:

[0026] ;

[0027] in, For the first Layer bitrate; Budget for delay; This refers to the actual transmission time. Indicates being bound by;

[0028] Total number of layers The determination is based on available communication resources and is dynamically determined using the following formula:

[0029] ;

[0030] in, Base layer fixed bit rate, The average bit rate for a single enhancement layer; This indicates taking the maximum value;

[0031] The optimal bit width is assigned to each layer based on the Lagrange multiplier method, and the formula is as follows:

[0032] ;

[0033] in, For the first Optimal bit width for the layer; For the first Layer weighted reconstruction distortion; For the first Layer in position width The predicted distortion is below; B is the candidate bit width set;

[0034] The fragment size is dynamically adjusted based on the bandwidth-delay product and the number of layers, using the following formula:

[0035] ;

[0036] in, To adjust the fragment size; , These are the lower and upper bounds of the fragment size, respectively.

[0037] Preferably, it also includes an early stop decision mechanism, that is, by evaluating the marginal benefit in real time, if the ratio of the distortion improvement to the bitrate overhead at the current layer is lower than a set threshold. If it is estimated that completing the next layer of transmission will exceed the latency budget, the data transmission of subsequent enhancement layers should be stopped immediately to avoid wasting communication resources when the marginal benefit is extremely low. The expression is:

[0038] ;

[0039] in, , Representing the current layer The next layer Reconstruction distortion after transmission is complete; Indicates the elapsed transmission time; Indicates the estimated transmission time for the next layer; Budget for time delay.

[0040] Preferably, in the hierarchical learnable orthogonal transformation module, a learnable orthogonal transformation method is used to map the original data to a subspace with more concentrated energy, and a partitioning method based on energy thresholds is used to cluster and group the feature channels in the subspace according to the channel entropy characteristics, as shown in the following expression:

[0041] The formula for learnable orthogonal transformation is: ;

[0042] in, The transformed data; The input is the raw data, where C is the number of channels, H is the height, and W is the width. It is a learnable orthogonal transformation matrix; , It is the identity matrix;

[0043] The formula for classifying based on energy thresholds is: , ;

[0044] in, For the first Energy of each channel; For the first Data after transformation of each channel; Represents the L2 norm; For the first The entropy value of each channel; Let be the normalized probability distribution of the channel coefficients; i represents the index of the element in the c-th channel.

[0045] All channels were grouped into clusters based on their energy thresholds. There are several groups, and the formula is:

[0046] ;

[0047] in, For the first Group, containing energy in All channels within the interval; , The first The energy threshold of group k-1; Total number of groups; Indicates the current channel;

[0048] After clustering, the mean of all channel data in each group is calculated, and the low-frequency component is extracted using a low-pass filter as the base layer transmission content. The base layer transmission content is represented as follows:

[0049] ;

[0050] in, Indicates the first The group's base layer transmits content; This indicates a low-pass filter to preserve key low-frequency information; To calculate the within-group centrality coefficient; For the first All channel data for the group;

[0051] Subtract the previous data from all channels in each layer and group. The residuals from the cumulative reconstruction results of each layer are used as the content transmitted in the enhancement layer, and the residuals are encoded layer by layer. The expression is as follows:

[0052] ;

[0053] ;

[0054] in, For the first Layer The group's enhancement layer transmits content; For the first Layer The group's enhancement layer transmits content; Indicates the 0th layer. The enhanced layer of the group transmits the content, namely the first The group's base layer transmits content.

[0055] Preferably, the formula for the importance score is:

[0056] ;

[0057] in, For the first The importance of each channel; For the first Energy of each channel; For the first The entropy value of each channel, The L2 norm of the gradient; For the first The gradient of each channel; These are weighting coefficients, adjusted according to task characteristics;

[0058] Then, for coefficients with high importance scores, a higher bit width is assigned to reduce quantization error; while for coefficients with low importance scores, a lower bit width is used; the bit width allocation formula is as follows:

[0059] ;

[0060] in, For the first The allocated bit width for each channel; , , These represent the high, medium, and low bit width settings, respectively. , These represent the high and low importance rating levels, respectively.

[0061] Simultaneously, by combining the µ-law adaptive non-uniform quantization method, logarithmic compression is used to improve the effective resolution of coefficients with low importance scores under low-bit conditions, as expressed in the following expression:

[0062] ;

[0063] in, Represents the input coefficients to be quantized The result after μ-law quantization; These are compression parameters; This represents the maximum value of the quantization coefficient. For symbolic functions, used to preserve The symbol; It is a logarithmic function, and by using logarithmic compression, the small values ​​of the coefficients to be quantized can obtain higher resolution.

[0064] Preferably, in the RTT adaptive scheduling module, the jitter level of the round-trip time (RTT) is monitored in real time, and the fragment size P and the fragmentation interval are dynamically adjusted. When the network jitter is large, the fragment size is reduced and an acknowledgment interval is inserted to reduce the risk of timeout and packet loss. When the bandwidth is stable and sufficient, the fragment size is increased to reduce header overhead and improve overall communication efficiency, as shown in the following expression:

[0065] The formula for real-time monitoring of RTT jitter is: ;

[0066] in, RTT jitter is used to measure the degree of fluctuation in round-trip time. Indicates standard deviation; This represents the round-trip time delay value obtained from the nth measurement;

[0067] ;

[0068] in, ; , They represent , The size of the slice at any given moment; , These represent the maximum and minimum values ​​of the fragment size, respectively. , These represent the high and low thresholds for RTT jitter, respectively. This indicates the adjustment step size for the slice size, used to adjust the slice size in non-extreme jitter conditions;

[0069] The formula for the transmission interval is: ;

[0070] in, This indicates the transmission interval, which is the time interval between two transmissions of a fragment; This represents the average round-trip time delay. , This is the adjustment coefficient.

[0071] Preferably, in the error tracking compensation module, an error tracking table is established to record the cumulative reconstruction error of different channels and frequency bands, for use in the next round of dynamic bit width allocation and early stop decision, as follows:

[0072] First, calculate the residual coding of the enhancement layer for each layer, using the following formula:

[0073] ;

[0074] in, For the first Layer The residuals of the group; For the first The group's enhancement layer transmits content. For the first Layer Group reconstructed data;

[0075] Next, the cumulative reconstruction error is calculated, which consists of the sum of the squares of the errors from the previous round and the current round, as shown in the formula:

[0076] ;

[0077] in, For the first The first channel in the The cumulative reconstruction error of the wheel; For the first The first channel in the The enhanced layer of the wheel transmits content. For the first The first channel in the Reconstructed data for the wheel;

[0078] The bit width is dynamically adjusted based on the cumulative reconstruction error, using the following formula:

[0079] ;

[0080] in, , They represent the first The first channel in the Wheel width; This is the error threshold; To round down;

[0081] When the maximum error of all channels is lower than the target error threshold, the enhancement layer coding is stopped to avoid invalid transmission. The stopping condition is:

[0082] ;

[0083] in, This represents the maximum cumulative error among all channels in round t; The target error threshold is set.

[0084] Preferably, it also includes a keyframe synchronization module, that is, it introduces a synchronization strategy to clear accumulated reconstruction errors by periodically transmitting keyframes, and the triggering conditions are as follows:

[0085] In the first case, an independently encoded keyframe is sent every N training rounds, and reconstruction is performed without relying on historical residuals. The expression is:

[0086] ;

[0087] in, Indicates a keyframe; Indicates the current round; Indicates the modulo operation; For synchronization period;

[0088] In the second scenario, a sudden surge in innovation energy triggers the transmission of keyframes, namely: the total energy of the current round. Compared to the previous round of energy When the ratio exceeds the set energy surge threshold, a keyframe is triggered to re-establish the baseline, as expressed by:

[0089] ;

[0090] in, The threshold for a sudden increase in energy;

[0091] The third scenario occurs when the cumulative distortion exceeds a set threshold, i.e., when the most recent distortion is calculated. The cumulative reconstruction error of each round; when the cumulative reconstruction error exceeds a set threshold... The expression that triggers the transmission of a keyframe is:

[0092] ;

[0093] in, For the first The average reconstruction error of the wheel; This is the cumulative distortion threshold;

[0094] In the fourth scenario, when packet loss occurs for an extended period, the error tracking state is cleared to ensure that the encoding and decoding ends are resynchronized, thus avoiding accuracy loss due to error drift.

[0095] This invention also provides a bandwidth-distortion-adaptive multilayer scalable feature compression method, comprising:

[0096] Collect real-time network status, training phase, and accuracy requirements; wherein, the network status includes available bandwidth, round-trip latency, and jitter;

[0097] Obtain raw data; wherein, the raw data includes raw features sent by the client or raw gradients sent by the server;

[0098] The original data is mapped to a more energy-concentrated subspace through a learnable orthogonal transformation, and the transformed feature channels are clustered and grouped so that during data transmission, the base layer only carries the center coefficients and low-frequency components of each group, while the enhancement layer encodes the high-frequency details and deviation parts of each group layer by layer with residuals, and performs progressive data reconstruction from coarse to fine to obtain transmittable layered data.

[0099] Importance scores are calculated based on the transformed hierarchical data, with bit widths dynamically allocated.

[0100] Adjust the fragment size and fragmentation interval according to the real-time network status;

[0101] The cumulative reconstruction error for each round is calculated based on the residual coding of the enhancement layer, and the bit width allocation for the next round is dynamically adjusted to obtain compressed data.

[0102] The total number of layers, the bit width allocation of each layer, and the size of the slices are dynamically determined based on the real-time network status, training stage, and accuracy requirements, so as to dynamically adjust the input original data and minimize the reconstruction distortion between the original data and the reconstructed data; wherein, the reconstructed data is obtained by the decoder decoding and reconstructing the compressed data.

[0103] This invention also provides a distributed machine learning system comprising several clients and servers, and a bandwidth-distortion adaptive multilayer scalable feature compressor as described above, located between the clients and the servers. The clients send raw features to the multilayer scalable feature compressor for compression, outputting compressed data. The servers receive and aggregate the compressed data, generating reconstructed data, i.e., global gradients. These global gradients are compressed by the multilayer scalable feature compressor to generate compressed gradients, which are then sent to each client. Each client updates its local model based on the compressed gradients and, based on the updated model, continues local training using the original features until the model converges.

[0104] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device in which the computer-readable storage medium resides, implement a bandwidth-distortion adaptive multilayer scalable feature compression method as described above.

[0105] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0106] This invention proposes a multi-layered scalable feature compression framework with adaptive bandwidth distortion by integrating six modules: a rate-distortion joint controller, a hierarchical learnable orthogonal transform module, a non-uniform quantization module, an RTT adaptive scheduling module, an error tracking compensation module, and a keyframe synchronization module. This significantly improves the robustness, convergence stability, and communication efficiency of distributed training in complex network environments. This invention fully utilizes the energy concentration, spatial redundancy, frequency domain sparsity, and temporal consistency of features, exhibiting good adaptability and scalability, making it particularly suitable for resource-constrained environments such as weak networks, low bandwidth, high latency, and strong jitter.

[0107] Specifically, the rate-distortion joint controller dynamically decides the number of layers, bit width allocation for each layer, and slice size based on real-time network status, training stage, and accuracy requirements to minimize reconstruction distortion under bandwidth constraints. The hierarchical learnable orthogonal transform module employs learnable orthogonal transform (LoT) to map input features to a more energy-concentrated subspace and clusters feature channels based on their entropy characteristics. The intra-layer non-uniform quantization module uses a non-uniform quantization strategy based on importance, allocating different bit widths according to the importance of coefficients. The RTT adaptive scheduling module designs an adaptive on-chip rate adjustment mechanism based on round-trip time (RTT). The error tracking compensation module specifies that the enhancement layer encodes the residuals on the reconstructed results of the base layer or lower layers and establishes an error tracking table to record the accumulated reconstruction error. The keyframe synchronization module clears accumulated errors by periodically transmitting anchor layers (keyframes). Attached Figure Description

[0108] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0109] Figure 1 The system framework diagram of the bandwidth distortion adaptive multilayer scalable feature compression method provided in Example 1 is shown.

[0110] Figure 2 This is a flowchart illustrating a bandwidth distortion adaptive multilayer scalable feature compression method provided in Example 1.

[0111] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0112] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0113] Example 1

[0114] Embodiment 1 of the present invention provides a bandwidth distortion adaptive multilayer scalable feature compression method, which can be implemented by a bandwidth distortion adaptive multilayer scalable feature compression device (hereinafter referred to as feature compression device), and in particular, executed by one or more processors within the feature compression device.

[0115] In this embodiment, the feature compression device may be an electronic device equipped with a processor, the processor having a computer program for the bandwidth distortion adaptive multilayer scalable feature compression method and the computer program being executable, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0116] In existing technologies, balancing communication efficiency and model accuracy in distributed machine learning systems faces significant challenges. Fixed-bit-width or single-layer compression methods struggle to adapt to dynamic network environments and training phase requirements, leading to resource waste or accuracy loss. For example, high accuracy is needed in the early stages of training to ensure model convergence, while compression efficiency becomes more important in the later stages. Existing methods lack adaptive adjustment capabilities and cannot achieve the optimal trade-off under bandwidth fluctuations and latency constraints.

[0117] To address the aforementioned issues, a mechanism is needed to dynamically adjust the compression strategy based on real-time network status and training phases. By analyzing the varying demands for reconstruction quality at different stages, we consider mapping features to an energy-concentrated subspace and employing hierarchical encoding to progressively refine the reconstruction. Furthermore, we introduce dynamic bit-width allocation and slice scheduling mechanisms to handle network fluctuations, while simultaneously reducing the impact of accumulated errors on subsequent training through an error tracking compensation mechanism.

[0118] The core of distributed learning is "parallel training by multiple nodes (such as clients) + global aggregation by a central node (such as a server)". Figure 1 The distributed machine learning system shown has an overall architecture including m clients, a server, and a multi-layer scalable feature compressor based on bandwidth distortion adaptation. The multi-layer scalable feature compressor is used to implement the feature compression process and the gradient compression process. The multi-layer scalable feature compressor compresses the original features obtained from the clients or the global gradients from the server (original features and global gradients are collectively referred to as original data) and outputs compressed data (compressed features or compressed gradients).

[0119] The core idea of ​​this invention is to propose a bandwidth-distortion adaptive multilayer scalable feature compression framework (BD-SLTC), namely a multilayer scalable feature compressor based on bandwidth distortion adaptation. It includes: a rate-distortion joint controller, a hierarchical learnable orthogonal transform module, a non-uniform quantization module, an RTT adaptive scheduling module and an error tracking compensation module, and a keyframe synchronization module. These modules work together to ensure efficient communication and high-precision model training in complex network environments.

[0120] Existing methods often employ fixed compression strategies or single-dimensional optimization, making it difficult to adapt to dynamic network environments and training requirements. For example, fixed-bit-width quantization cannot be flexibly adjusted when bandwidth fluctuates, and single-layer encoding lacks progressive reconstruction capabilities. This scheme combines multi-layer scalable encoding with dynamic parameter adjustment. When bandwidth is limited, it prioritizes the transmission of the base layer to ensure basic accuracy, and gradually enhances details when bandwidth is ample. At the same time, it reduces cumulative distortion through an error compensation mechanism, significantly improving the balance between communication efficiency and model accuracy in complex network environments.

[0121] (a) Rate Distortion Joint Controller

[0122] In this framework, the encoder first introduces a rate-distortion joint controller (RD-Controller). By sensing network bandwidth, round-trip latency, and accuracy requirements during training in real time, it jointly determines the number of layers, the bit width of each layer, and the size of each slice, thereby minimizing overall distortion under constraints. Specifically, the encoder adopts a layered structure of "base layer + multiple enhancement layers." The base layer carries the most important low-frequency components and dominant features to ensure usable reconstruction accuracy even with minimal communication. The enhancement layers compensate for the detailed information of different frequency bands or groups step-by-step through inter-layer incremental residuals, enabling the model to improve reconstruction quality layer by layer when bandwidth conditions permit.

[0123] The rate-distortion joint controller is used to dynamically decide the total number of layers, the bit width allocation of each layer, and the fragment size based on the real-time network status, training stage, and accuracy requirements, so as to dynamically adjust the input raw data and minimize the reconstruction distortion between the raw data and the reconstructed data; wherein, the reconstructed data is obtained by the decoder decoding and reconstructing the compressed data; the network status includes available bandwidth, round-trip time, and jitter.

[0124] First, the input feature or gradient data undergoes dynamic parameter adjustment by a rate-distortion co-controller. This controller senses three types of information in real time: network status, training phase, and accuracy requirements. The network status awareness module calculates the effective available bandwidth by monitoring available bandwidth, round-trip time (RTT), and jitter. Available bandwidth is the actual amount of data that can be transmitted per unit time (e.g., 5 Mbps), reflecting the upper limit of transmission capacity. Round-trip time (RTT) is the time it takes for data to travel from the sender to the receiver and back (e.g., 100 ms), reflecting network response speed. Jitter is the fluctuation value of RTT (e.g., ±20 ms), reflecting network stability.

[0125] The rate-distortion joint controller includes a network state awareness submodule, a training phase awareness submodule, and an accuracy requirement submodule.

[0126] The network state awareness submodule calculates the effective available bandwidth by monitoring available bandwidth, round-trip time, and jitter, using the following formula:

[0127] ;

[0128] in, Effective available bandwidth; For real-time monitoring of available bandwidth; RTT stands for Round Trip Time. For RTT jitter; This is a conservative coefficient used to address protocol overhead and network fluctuation redundancy.

[0129] The training phase perception submodule is used to calculate the distortion weight coefficient based on the current training round and the total number of rounds, using the following formula:

[0130] ;

[0131] in, The distortion weight coefficient for the current training round t; Total number of rounds; , These represent the maximum and minimum distortion weight coefficients, respectively; This is the decay rate parameter, used to control the transition speed from high precision to high compression.

[0132] The accuracy requirement submodule is used to minimize the weighted reconstruction distortion of the original data and the reconstructed data by jointly optimizing the total number of layers, the bit width of each layer, and the size of each fragment, based on the available bandwidth and distortion weight coefficient, under a preset latency budget constraint. The objective function formula for the joint optimization is:

[0133] ;

[0134] in, This represents the total number of layers; For the first Layer width; This refers to the size of the slice; Distortion due to weighted reconstruction; For the first Layer weights; The original data; For the first The reconstructed data corresponding to the layer; It is an L2 norm; This indicates a minimize operation.

[0135] The constraints are that the total bit rate does not exceed the bandwidth-delay product, and the actual transmission time does not exceed the delay budget. The formula is:

[0136] ;

[0137] in, For the first Layer bitrate; Budget for delay; This refers to the actual transmission time. It indicates that one is bound by something.

[0138] Total number of layers The determination is based on available communication resources and is dynamically determined using the following formula:

[0139] ;

[0140] in, Base layer fixed bit rate, The average bit rate for a single enhancement layer; This indicates taking the maximum value.

[0141] The optimal bit width is assigned to each layer based on the Lagrange multiplier method, and the formula is as follows:

[0142] ;

[0143] in, For the first Optimal bit width for the layer; For the first Layer weighted reconstruction distortion; For the first Layer in position width The predicted distortion is below; B is the candidate bit-width set.

[0144] The fragment size is dynamically adjusted based on the bandwidth-delay product and the number of layers, using the following formula:

[0145] ;

[0146] in, To adjust the fragment size; , These are the lower and upper bounds of the fragment size, respectively.

[0147] Compared to existing technologies, traditional methods using a fixed number of layers and uniform quantization bit width cannot adapt to dynamic network environments and training phase requirements. For example, a fixed 3-layer encoding may exceed the latency budget when bandwidth suddenly drops, while a fixed 8-bit quantization may waste bandwidth resources in the later stages of training. This scheme dynamically adjusts the layer structure and bit width allocation through joint optimization of the model. For example, when bandwidth is ample, enhancement layers are added to improve accuracy, while the number of layers is reduced to ensure real-time performance when bandwidth is limited. Compared to single-layer encoding methods, the multi-layer progressive reconstruction mechanism can reduce reconstruction errors at the same bit rate. For example, the base layer transmits the main energy information, and the enhancement layers gradually supplement details.

[0148] Through the above technical solution, this application achieves adaptive feature compression based on network status and training stage. In scenarios with network fluctuations, transmission reliability is ensured by dynamically adjusting the number of layers and the size of fragments; for example, fragments are automatically reduced to mitigate packet loss when jitter intensifies. During different training stages, the balance between accuracy and compression is adjusted through distortion weight coefficients; for example, a high bit width is maintained in the early stages of training to ensure model convergence stability, while the bit width is appropriately reduced in later stages to improve throughput. This solution effectively solves the problem of insufficient adaptability of fixed compression strategies in complex network environments, improving the communication efficiency and model accuracy co-optimization capability of distributed machine learning systems.

[0149] To adapt to unstable network environments, this invention designs a segmentation mechanism based on a delay window. This ensures that data is transmitted incrementally with small granularities and controllable delays. An early stop mechanism is triggered when the target distortion is reached or the delay budget is exhausted, thereby avoiding unnecessary communication overhead. The receiving end can perform a flexible reconstruction at any receiving level, meeting both the minimum accuracy requirements under weak network conditions and gradually improving training performance in high-bandwidth scenarios.

[0150] The early termination mechanism assesses marginal benefits in real time, determining whether the distortion improvement relative to the bitrate overhead falls below a set threshold. If it is estimated that completing the next layer of transmission will exceed the latency budget, the data transmission of subsequent enhancement layers should be stopped immediately to avoid wasting communication resources when the marginal benefit is extremely low, and to ensure that high-value information is transmitted first within the latency constraint. The expression is:

[0151] ;

[0152] in, , Representing the current layer The next layer Reconstruction distortion after transmission is complete; Indicates the elapsed transmission time; Indicates the estimated transmission time for the next layer; For latency budgeting. Compared with existing technologies, traditional methods usually adopt a fixed number of layers or a static stopping strategy, which cannot dynamically adjust the number of transmission layers according to real-time network fluctuations and task requirements. This can easily lead to resource waste or timeout packet loss when bandwidth drops suddenly or latency increases suddenly. In contrast, the early stopping decision mechanism, by introducing the dual constraints of marginal benefit assessment and latency budget, can adaptively terminate inefficient transmissions and achieve more refined resource control in complex network environments.

[0153] Through the above technical solution, this application effectively solves the problems of low transmission efficiency and resource waste in the existing technology in dynamic network environment. It can intelligently terminate redundant data transmission according to real-time channel conditions and task requirements while ensuring basic reconstruction quality, thereby improving the utilization rate of communication resources and reducing the risk of transmission timeout.

[0154] (ii) Hierarchical learnable orthogonal transformation module

[0155] This embodiment first employs a Learnable Orthogonal Transform (LoT) to map the input features or gradients to a more energy-concentrated subspace. Within this subspace, feature channels are then clustered based on their entropy characteristics, using an energy threshold-based partitioning method. After grouping, the base layer carries only the center coefficients and low-frequency components of each group to ensure the transmission of the most critical information within a limited bit budget. The enhancement layer then performs layer-by-layer residual encoding on the high-frequency details and deviations within each group, achieving a progressive reconstruction from coarse to fine. This method can prioritize covering regions with high energy content or significant uncertainty at the same bit rate, optimizing the use of limited communication resources.

[0156] In the hierarchical learnable orthogonal transformation module, a learnable orthogonal transformation method is used to map the original data to a subspace with more concentrated energy. Then, an energy threshold-based partitioning method is used to cluster and group the feature channels within the subspace according to their entropy characteristics, as shown in the following expression:

[0157] The formula for learnable orthogonal transformation is: ;

[0158] in, The transformed data; The input is the raw data, where C is the number of channels, H is the height, and W is the width. It is a learnable orthogonal transformation matrix; , It is an identity matrix.

[0159] The formula for classifying based on energy thresholds is: , ;

[0160] in, For the first Energy of each channel; For the first Data after transformation of each channel; Represents the L2 norm; For the first The entropy value of each channel; Let be the normalized probability distribution of the channel coefficients; i represents the index of the element in the c-th channel.

[0161] All channels were grouped into clusters based on their energy thresholds. There are several groups, and the formula is:

[0162] ;

[0163] in, For the first Group, containing energy in All channels within the interval; , The first The energy threshold of group k-1; Total number of groups; Indicates the current channel.

[0164] After clustering, the mean of all channel data in each group is calculated, and the low-frequency component is extracted using a low-pass filter as the base layer transmission content. The base layer transmission content is represented as follows:

[0165] ;

[0166] in, Indicates the first The group's base layer transmits content; This indicates a low-pass filter to preserve key low-frequency information; To calculate the within-group centrality coefficient; For the first All channel data for the group.

[0167] Subtract the previous data from all channels in each layer and group. The residuals from the cumulative reconstruction results of each layer are used as the content transmitted in the enhancement layer, and the residuals are encoded layer by layer. The expression is as follows:

[0168] ;

[0169] ;

[0170] in, For the first Layer The group's enhancement layer transmits content; For the first Layer The group's enhancement layer transmits content; Indicates the 0th layer. The enhanced layer of the group transmits the content, namely the first The group's base layer transmits content.

[0171] Traditional methods often employ fixed orthogonal transforms, such as the discrete cosine transform, whose basis functions cannot adapt to the data distribution characteristics of specific tasks. This scheme achieves data-driven basis function optimization through a learnable orthogonal transform, improving energy concentration efficiency by approximately 30% compared to fixed transforms. Existing technologies often use fixed rules or random division for channel grouping, such as dividing into four groups according to channel order. This approach struggles to adapt to the varying importance of different data features. This scheme uses dynamic clustering based on a joint index of energy and entropy, effectively distinguishing between critical and secondary channels. Experimental results show that this method can reduce the quantization error of important channels by approximately 40%. Traditional residual coding typically uses a fixed number of layers, such as forcibly dividing into three enhancement layers, which can easily lead to bandwidth waste or insufficient accuracy. This scheme achieves flexible layering through dynamic threshold division, reducing the range of reconstruction accuracy fluctuations by approximately 25% under the same bandwidth conditions.

[0172] Through the above technical solutions, this application can dynamically optimize the transform domain and grouping structure based on data characteristics, achieving accurate residual coding of enhancement layer data while ensuring the reliability of base layer transmission. The learnable orthogonal transform improves the concentration of feature energy through adaptive basis functions, enabling base layer data to carry main information at a lower bit rate. The dynamic grouping mechanism based on energy thresholds effectively distinguishes feature channels of different importance, providing a physically clear basis for hierarchical coding. Layer-by-layer residual coding combined with low-pass filtering significantly reduces the transmission requirements of high-frequency components, maintaining stable reconstruction quality even in jittery network environments. Experiments show that this scheme can improve the convergence speed of gradient transmission models by approximately 18% under the same bandwidth conditions, while reducing the number of communication retransmissions by approximately 35%.

[0173] (III) Non-uniform quantization module

[0174] During the intra-layer quantization stage, this embodiment employs an importance-based non-uniform quantization strategy. The basic idea is to allocate different bit widths based on the importance of the coefficients. For coefficients with high importance (determined by indicators such as energy, entropy, and gradient consistency), a higher bit width is allocated to reduce quantization error; while for coefficients with low importance, a lower bit width is used. This is combined with an µ-law adaptive non-uniform quantization method to improve effective resolution under low-bit conditions through logarithmic compression.

[0175] The common configuration is a bit width set of {2, 3, 4, 6, 8}. The difference in bit width between the important group and the ordinary group is generally controlled between 1 and 3 bits, which significantly improves the compression ratio while ensuring accuracy.

[0176] The formula for the importance score is:

[0177] ;

[0178] in, For the first The importance of each channel; For the first Energy of each channel; For the first The entropy value of each channel, The L2 norm of the gradient; For the first The gradient of each channel; This is a weighting coefficient, which is adjusted according to the characteristics of the task.

[0179] Then, for coefficients with high importance scores, a higher bit width is assigned to reduce quantization error; while for coefficients with low importance scores, a lower bit width is used; the bit width allocation formula is as follows:

[0180] ;

[0181] in, For the first The allocated bit width for each channel; , , These represent the high, medium, and low bit width settings, respectively. , These represent the high and low importance rating levels, respectively.

[0182] Simultaneously, by combining the µ-law adaptive non-uniform quantization method, logarithmic compression is used to improve the effective resolution of coefficients with low importance scores under low-bit conditions, as expressed in the following expression:

[0183] ;

[0184] in, Represents the input coefficients to be quantized The result after μ-law quantization; These are compression parameters; This represents the maximum value of the quantization coefficient. For symbolic functions, used to preserve The symbol; It is a logarithmic function, and by using logarithmic compression, the small values ​​of the coefficients to be quantized can obtain higher resolution.

[0185] Traditional methods often use a single metric to allocate bit width, such as relying solely on gradient magnitude or energy distribution, which fails to comprehensively reflect feature importance. Fixed-bit-width quantization is prone to losing small-value information in low-bit scenarios, while uniform quantization cannot adapt to the dynamic range differences of features. This scheme achieves fine-grained bit-width control through multi-factor comprehensive scoring and combines it with non-uniform quantization to enhance resolution under low-bit conditions, overcoming the limitations of single-metric decision-making.

[0186] Through the above technical solution, this application achieves dynamic matching between feature sensitivity and quantization accuracy, effectively reducing the transmission overhead of non-critical data while ensuring high fidelity of key information. μ-law quantization improves effective resolution under low-bit conditions, preventing small-amplitude features from being overwhelmed by noise during compression. This solution can adapt to the differentiated feature sensitivity requirements of different tasks, maintaining model training stability under bandwidth-constrained conditions.

[0187] (iv) RTT Adaptive Scheduling Module

[0188] In terms of transmission scheduling, this embodiment designs an adaptive on-chip bitrate adjustment mechanism based on Round Trip Time (RTT). The system monitors RTT and jitter levels in real time and dynamically adjusts the fragment size P and the fragmentation frequency accordingly. When network jitter is high, the fragment size is reduced and an acknowledgment interval is inserted to reduce the risk of timeout and packet loss; while when bandwidth is stable and sufficient, the fragment size is increased to reduce header overhead, thereby improving overall communication efficiency, as shown in the following expression:

[0189] The formula for real-time monitoring of RTT jitter is: ;

[0190] in, RTT jitter is used to measure the degree of fluctuation in round-trip time. Indicates standard deviation; This represents the round-trip time delay value obtained from the nth measurement.

[0191] ;

[0192] in, ; , They represent , The size of the slice at any given moment; , These represent the maximum and minimum values ​​of the fragment size, respectively. , These represent the high and low thresholds for RTT jitter, respectively. This indicates the adjustment step size for the slice size, used to adjust the slice size in non-extreme jitter conditions.

[0193] The formula for the transmission interval is: ;

[0194] in, This indicates the transmission interval, which is the time interval between two transmissions of a fragment; This represents the average round-trip time delay. , This is the adjustment coefficient.

[0195] Traditional methods typically employ fixed fragment sizes and transmission intervals, which cannot adapt to dynamically changing network environments. Existing bandwidth estimation-based adjustment methods only consider bandwidth capacity while ignoring the impact of latency jitter, easily leading to decreased transmission efficiency in scenarios with sudden packet loss. This solution introduces a quantitative evaluation mechanism for RTT jitter, combined with joint optimization of fragment size and transmission interval, achieving rapid response to network fluctuations. Compared to methods relying solely on bandwidth measurement, this solution more accurately balances transmission reliability and efficiency, particularly effectively reducing retransmission probability in high-jitter network environments.

[0196] Through the above technical solution, this application can intelligently adjust transmission parameters according to real-time network conditions, maximizing bandwidth utilization while ensuring reliable data transmission. When sudden network jitter occurs, the risk of data packet loss is effectively reduced by rapidly decreasing the fragment size and extending the transmission interval; after the network stabilizes, the fragment size is promptly increased to improve the effective payload ratio. This dynamic adjustment mechanism significantly improves the robustness of the distributed learning system in complex network environments, while avoiding resource waste or performance bottlenecks caused by fixed parameter settings.

[0197] (v) Error tracking and compensation module

[0198] To avoid information redundancy and transmission waste, this embodiment specifies that the enhancement layer coding object is the residual on the reconstructed result of the base layer or lower layers. Simultaneously, in the error tracking compensation module, an error tracking table is established to record the cumulative reconstruction error of different channels and frequency bands. This error is used for the next round of dynamic bit width allocation and early stop decision, enabling the system to minimize invalid transmission and improve overall resource utilization while ensuring reconstruction accuracy. Specifically:

[0199] First, calculate the residual coding of the enhancement layer for each layer, using the following formula:

[0200] ;

[0201] in, For the first Layer The residuals of the group; For the first The group's enhancement layer transmits content. For the first Layer Reconstructed data for the group.

[0202] Next, the cumulative reconstruction error is calculated, which consists of the sum of the squares of the errors from the previous round and the current round, as shown in the formula:

[0203] ;

[0204] in, For the first The first channel in the The cumulative reconstruction error of the wheel; For the first The first channel in the The enhanced layer of the wheel transmits content. For the first The first channel in the Reconstructed data for the wheel.

[0205] ;

[0206] in, , They represent the first The first channel in the Wheel width; This is the error threshold; This is for rounding down.

[0207] When the maximum error of all channels is lower than the target error threshold, the enhancement layer coding is stopped to avoid invalid transmission. The stopping condition is:

[0208] ;

[0209] in, This represents the maximum cumulative error among all channels in round t; The target error threshold is set.

[0210] Traditional methods employ a fixed bit-width allocation strategy and lack an error tracking mechanism, making it impossible to adjust quantization precision based on the actual error distribution. This can easily lead to resource waste or insufficient accuracy. Existing residual coding techniques typically only focus on single-round error compensation and do not establish a long-term error tracking mechanism, making it difficult to effectively suppress the cumulative effect of errors. This scheme establishes a multi-round error tracking table to achieve dynamic bit-width adjustment based on historical error data. Simultaneously, it introduces an adaptive stopping condition, significantly improving the accuracy of error compensation and the utilization rate of communication resources.

[0211] Through the above technical solutions, this application can effectively suppress error accumulation in distributed machine learning, achieving optimal matching between communication resources and reconstruction accuracy through a dynamic bit-width allocation mechanism. Continuous updates to the error tracking table provide data support for the quantization strategy, enabling the system to accurately compensate for high-frequency error channels. Adaptive stopping conditions prevent the transmission of redundant data after the error threshold is met, thereby reducing communication overhead and improving system response speed. This solution maintains stable error control capabilities even in complex network environments, providing reliable compression guarantees for distributed training.

[0212] (vi) Keyframe Synchronization Module

[0213] During long-term training, errors may gradually accumulate due to residual propagation, leading to offset. Therefore, this embodiment introduces a keyframe synchronization module, i.e., a synchronization strategy, which clears accumulated reconstruction errors by periodically transmitting keyframes. The triggering conditions are as follows:

[0214] In the first case, an independently encoded keyframe is sent every N training rounds, and reconstruction is performed without relying on historical residuals. The expression is:

[0215] ;

[0216] in, Indicates a keyframe; Indicates the current round; Indicates the modulo operation; For synchronization period.

[0217] In the second scenario, a sudden surge in innovation energy triggers the transmission of keyframes, namely: the total energy of the current round. Compared to the previous round of energy When the ratio exceeds the set energy surge threshold, a keyframe is triggered to re-establish the baseline, as expressed by:

[0218] ;

[0219] in, This is the threshold for a sudden increase in energy.

[0220] The third scenario occurs when the cumulative distortion exceeds a set threshold, i.e., when the most recent distortion is calculated. The cumulative reconstruction error of each round; when the cumulative reconstruction error exceeds a set threshold... The expression that triggers the transmission of a keyframe is:

[0221] ;

[0222] in, For the first The average reconstruction error of the wheel; This is the cumulative distortion threshold.

[0223] In the fourth scenario, when packet loss occurs for an extended period, the error tracking state is cleared to ensure that the encoding and decoding ends are resynchronized, thus avoiding accuracy loss due to error drift.

[0224] The system uses periodic keyframes or anchor layer mechanisms to achieve error correction and state resynchronization, preventing drift and accuracy degradation caused by residual accumulation.

[0225] Traditional methods typically rely solely on sending keyframes at fixed intervals, lacking the ability to proactively respond to sudden changes in data distribution and network anomalies. For example, existing technologies using periodic keyframes cannot handle sudden changes in feature energy during training, while this scheme, through an energy surge detection mechanism, can promptly reconstruct the baseline when significant changes occur in data distribution. Furthermore, existing technologies lack proactive monitoring of accumulated errors, while this scheme calculates error trends using a sliding window and proactively triggers synchronization before the error accumulates to a critical point, avoiding accuracy loss.

[0226] Through the above technical solution, this application effectively solves the encoding / decoding mismatch problem caused by error accumulation in residual coding systems, and can maintain the consistency of the encoder / decoder state under complex scenarios such as network fluctuations and changes in data distribution. By employing a multi-condition triggered synchronization mechanism, it reduces invalid transmissions while ensuring communication efficiency, reducing redundant keyframe transmissions by more than 30% compared to a single-cycle synchronization strategy. This solution is particularly suitable for distributed learning scenarios with limited bandwidth and unstable network conditions, such as environments with frequent device disconnections in mobile edge computing, and can significantly improve system robustness and model convergence stability.

[0227] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0228] This invention proposes a bandwidth-distortion adaptive, multi-layered, scalable feature compression framework by integrating six techniques: joint rate-distortion control, hierarchical learnable orthogonal transform, non-uniform quantization, RTT adaptive scheduling, error tracking compensation, and keyframe synchronization. This framework fully leverages the energy concentration, spatial redundancy, frequency domain sparsity, and temporal consistency of features, exhibiting excellent adaptability and scalability. It is particularly suitable for resource-constrained environments such as weak networks, low bandwidth, high latency, and high jitter. In practical applications, such as in the architecture of personalized federated learning systems, this invention can significantly improve the robustness, convergence stability, and communication efficiency of distributed training, meeting the practical needs of diverse scenarios.

[0229] Example 2

[0230] like Figure 2 As shown, a bandwidth-distortion-adaptive multilayer scalable feature compression method includes the following steps:

[0231] S1, collect real-time network status, training phase and accuracy requirements; wherein, the network status includes available bandwidth, round-trip latency and jitter;

[0232] S2, Obtain raw data; wherein, the raw data includes raw features sent by the client or raw gradients sent by the server;

[0233] S3, the original data is mapped to a more energy-concentrated subspace through a learnable orthogonal transformation, and the transformed feature channels are clustered and grouped so that during data transmission, the base layer only carries the center coefficients and low-frequency components of each group, while the enhancement layer encodes the high-frequency details and deviation parts of each group layer by layer with residuals, and performs progressive data reconstruction from coarse to fine to obtain transmittable layered data.

[0234] S4 calculates importance scores based on the transformed hierarchical data to dynamically allocate bit width;

[0235] S5 adjusts the fragment size and fragmentation interval based on real-time network conditions;

[0236] S6. Calculate the cumulative reconstruction error for each round based on the residual coding of the enhancement layer and dynamically adjust the bit width allocation for the next round to obtain compressed data;

[0237] S7. Based on the real-time network status, training stage, and accuracy requirements, dynamically decide the total number of layers, the bit width allocation of each layer, and the size of the slices to dynamically adjust the input original data and minimize the reconstruction distortion between the original data and the reconstructed data; wherein, the reconstructed data is obtained by the decoder decoding and reconstructing the compressed data.

[0238] Example 3

[0239] The third embodiment of the present invention also provides a distributed machine learning system, which includes several clients and servers, and a bandwidth distortion adaptive multilayer scalable feature compressor as described above, located between the clients and the servers. The clients send raw features to the multilayer scalable feature compressor for compression, and output compressed data. The servers receive and aggregate the compressed data, and generate reconstructed data, i.e., global gradients. The global gradients are compressed by the multilayer scalable feature compressor to generate compressed gradients, which are then sent to each client. Each client updates its local model according to the compressed gradients, and continues to perform the next round of local training based on the updated model and the original features, until the model converges.

[0240] In this context, the client refers to the terminal device deployed at the edge of the distributed network, which can be a smart terminal or IoT device equipped with computing power. It is used to perform local model training and generate the raw features to be transmitted. The server refers to the computing node deployed in the cloud or at a central node, which can be implemented using a high-performance server cluster. It is responsible for aggregating compressed data from multiple clients and generating global model parameters. The multi-layer scalable feature compressor is a communication module embedded between the client and server. It can be implemented using a combination of dynamic encoders and decoders, used to dynamically adjust the compression strategy based on real-time network conditions and the training phase, achieving a balance between bandwidth and distortion.

[0241] Specifically, after generating raw features during local training, the client performs layered encoding using a multi-layer scalable feature compressor. During encoding, the base layer transmits low-frequency components and center coefficients, while the enhancement layers supplement high-frequency details layer by layer through residual encoding. After receiving the compressed data, the server reconstructs the raw features using a decoder and aggregates them to generate global model parameters, which are then distributed to each client for model synchronization. The client updates its local model parameters based on the reconstructed data and continues training using its local dataset. Throughout the training process, the compressor dynamically adjusts the number of layers and bit width allocation based on network bandwidth fluctuations. For example, it increases the number of enhancement layers to improve reconstruction accuracy when bandwidth is sufficient, and reduces the number of layers to prioritize base layer transmission when bandwidth is limited. Simultaneously, an error tracking mechanism accumulates reconstruction errors; when the error exceeds a threshold, keyframe synchronization is triggered to prevent model drift caused by error accumulation.

[0242] Compared to existing technologies, current distributed learning systems typically employ fixed compression strategies, failing to dynamically adjust encoding parameters based on network conditions. This leads to transmission delays or accuracy loss during bandwidth fluctuations. Our system, however, utilizes a multi-layered scalable encoding structure, combined with real-time network awareness and dynamic bit-width allocation, to adaptively adjust compression granularity under varying network conditions. For example, high bit-width encoding is used in the early stages of training to ensure model convergence stability, while the bit-width is reduced in the later stages to improve transmission efficiency. Furthermore, existing methods lack error tracking mechanisms, potentially leading to model mismatch between the client and server during long-term training. Our system, through periodic keyframe synchronization and residual compensation mechanisms, effectively suppresses the impact of accumulated errors on model accuracy.

[0243] Through the above technical solutions, this application addresses the dynamic balance between communication efficiency and model accuracy in distributed machine learning systems. In environments with fluctuating bandwidth, the system employs hierarchical encoding and dynamic bit-width allocation to ensure priority transmission of key feature information, preventing data loss due to network congestion. During model training, an adaptive compression strategy adjusts reconstruction accuracy according to the training stage; for example, it retains more high-frequency details in the early stages of convergence to accelerate convergence, while prioritizing compression efficiency to reduce communication overhead in later stages. Simultaneously, error tracking and keyframe synchronization mechanisms effectively suppress error accumulation during long-term training, improving the accuracy of model updates and system robustness.

[0244] Example 4

[0245] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device in which the computer-readable storage medium resides, implement the bandwidth distortion adaptive multilayer scalable feature compression method described above.

[0246] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A bandwidth-distortion adaptive multilayer scalable feature compressor, applied between the client and server sides of a distributed machine learning system, to compress the raw data acquired from the client or server and output compressed data, characterized in that, include: The system includes a rate-distortion joint controller, a hierarchical learnable orthogonal transform module, a non-uniform quantization module, an RTT adaptive scheduling module, and an error tracking compensation module. The rate-distortion joint controller is used to dynamically decide the total number of layers, the bit width allocation of each layer, and the fragment size based on the real-time network status, training stage, and accuracy requirements, so as to dynamically adjust the input raw data and minimize the reconstruction distortion between the raw data and the reconstructed data; wherein the reconstructed data is obtained by decoding and reconstructing the compressed data by the decoder at the receiving end; the network status includes available bandwidth, round-trip time, and jitter. The hierarchical learnable orthogonal transformation module is used to map the original data to a more energy-concentrated subspace and cluster the feature channels. During data transmission, the base layer only carries the center coefficients and low-frequency components of each group, while the enhancement layer encodes the high-frequency details and deviation parts of each group layer by layer, performing progressive data reconstruction from coarse to fine, to obtain transmittable hierarchical data. The non-uniform quantization module is used to calculate importance scores based on the transformed hierarchical data, so as to dynamically allocate bit width; The RTT adaptive scheduling module is used to dynamically adjust the fragment size and fragmentation interval according to the real-time network status. The error tracking and compensation module is used to calculate the cumulative reconstruction error of each round based on the residual encoding of the enhancement layer and dynamically adjust the bit width allocation of the next round to obtain compressed data.

2. The bandwidth distortion adaptive multilayer scalable feature compressor according to claim 1, characterized in that... The raw data includes: raw features obtained from the client, or global gradients obtained from the server; The rate-distortion joint controller includes a network state awareness submodule, a training phase awareness submodule, and an accuracy requirement submodule. The network state awareness submodule calculates the effective available bandwidth by monitoring available bandwidth, round-trip time, and jitter, using the following formula: ； in, Effective available bandwidth; For real-time monitoring of available bandwidth; RTT stands for Round Trip Time. For RTT jitter; This is a conservative coefficient used to address protocol overhead and network fluctuation redundancy. The training phase perception submodule is used to calculate the distortion weight coefficient based on the current training round and the total number of rounds, using the following formula: ； in, The distortion weight coefficient for the current training round t; Total number of rounds; , These represent the maximum and minimum distortion weight coefficients, respectively; The decay rate parameter is used to control the transition speed from high precision to high compression. The accuracy requirement submodule is used to minimize the weighted reconstruction distortion of the original data and the reconstructed data by jointly optimizing the total number of layers, the bit width of each layer, and the size of each fragment, based on the available bandwidth and distortion weight coefficient, under a preset latency budget constraint. The objective function formula for the joint optimization is: ； in, This represents the total number of layers; For the first Layer width; This refers to the size of the slice; Distortion due to weighted reconstruction; For the first Layer weights; The original data; For the first The reconstructed data corresponding to the layer; It is an L2 norm; This indicates a minimize operation; The constraints are that the total bit rate does not exceed the bandwidth-delay product, and the actual transmission time does not exceed the delay budget. The formula is: ； in, For the first Layer bitrate; Budget for delay; This refers to the actual transmission time. Indicates being bound by; Total number of layers The determination is based on available communication resources and is dynamically determined using the following formula: ； in, Base layer fixed bit rate, The average bit rate for a single enhancement layer; This indicates taking the maximum value; To round down; The optimal bit width is assigned to each layer based on the Lagrange multiplier method, and the formula is as follows: ； in, For the first Optimal bit width for the layer; For the first Layer weighted reconstruction distortion; For the first Layer in position width The predicted distortion is below; B is the candidate bit width set; The fragment size is dynamically adjusted based on the bandwidth-delay product and the number of layers, using the following formula: ； in, To adjust the fragment size; , These are the lower and upper bounds of the fragment size, respectively.

3. A bandwidth-distortion adaptive multilayer scalable feature compressor according to claim 2, characterized in that... It also includes an early stop decision mechanism, which assesses marginal benefits in real time and determines whether the distortion improvement relative to the bitrate overhead at the current layer falls below a set threshold. If it is estimated that completing the next layer of transmission will exceed the latency budget, the data transmission of subsequent enhancement layers should be stopped immediately to avoid wasting communication resources when the marginal benefit is extremely low. The expression is: ； in, , Representing the current layer The next layer Reconstruction distortion after transmission is complete; Indicates the elapsed transmission time; Indicates the estimated transmission time for the next layer; Budget for time delay.

4. A bandwidth-distortion adaptive multilayer scalable feature compressor according to claim 3, characterized in that... In the hierarchical learnable orthogonal transformation module, a learnable orthogonal transformation method is used to map the original data to a subspace with more concentrated energy. Then, an energy threshold-based partitioning method is used to cluster and group the feature channels within the subspace according to their entropy characteristics, as shown in the following expression: The formula for learnable orthogonal transformation is: ; in, The transformed data; The input is the raw data, where C is the number of channels, H is the height, and W is the width. It is a learnable orthogonal transformation matrix; , It is the identity matrix; The formula for classifying based on energy thresholds is: , ; in, For the first Energy of each channel; For the first Data after transformation of each channel; Represents the L2 norm; For the first The entropy value of each channel; Let be the normalized probability distribution of the channel coefficients; i represents the index of the element in the c-th channel. All channels were grouped into clusters based on their energy thresholds. There are several groups, and the formula is: ； in, For the first Group, containing energy in All channels within the interval; , The first The energy threshold of group k-1; Total number of groups; Indicates the current channel; After clustering, the mean of all channel data in each group is calculated, and the low-frequency component is extracted using a low-pass filter as the base layer transmission content. The base layer transmission content is represented as follows: ； in, Indicates the first The group's base layer transmits content; This indicates a low-pass filter to preserve key low-frequency information; To calculate the within-group centrality coefficient; For the first All channel data for the group; Subtract the previous data from all channels in each layer and group. The residuals from the cumulative reconstruction results of each layer are used as the content transmitted in the enhancement layer, and the residuals are encoded layer by layer. The expression is as follows: ； ； in, For the first Layer The group's enhancement layer transmits content; For the first Layer The group's enhancement layer transmits content; Indicates the 0th layer. The enhanced layer of the group transmits the content, namely the first The group's base layer transmits content.

5. A bandwidth-distortion adaptive multilayer scalable feature compressor according to claim 4, characterized in that... The formula for the importance score is: ； in, For the first The importance of each channel; For the first Energy of each channel; For the first The entropy value of each channel, The L2 norm of the gradient; For the first The gradient of each channel; These are weighting coefficients, adjusted according to task characteristics; Then, for coefficients with high importance scores, a higher bit width is assigned to reduce quantization error; while for coefficients with low importance scores, a lower bit width is used; the bit width allocation formula is as follows: ； in, For the first The allocated bit width for each channel; , , These represent the high, medium, and low bit width settings, respectively. , These represent the high and low importance rating levels, respectively. Simultaneously, by combining the µ-law adaptive non-uniform quantization method, logarithmic compression is used to improve the effective resolution of coefficients with low importance scores under low-bit conditions, as expressed in the following expression: ； in, Represents the input coefficients to be quantized The result after μ-law quantization; These are compression parameters; This represents the maximum value of the quantization coefficient. For symbolic functions, used to preserve The symbol; It is a logarithmic function, and by using logarithmic compression, the small values ​​of the coefficients to be quantized can obtain higher resolution.

6. A bandwidth-distortion adaptive multilayer scalable feature compressor according to claim 2, characterized in that, In the RTT adaptive scheduling module, the jitter level of the round-trip time (RTT) is monitored in real time, and the fragment size P and the fragmentation interval are dynamically adjusted. When network jitter is large, the fragment size is reduced and an acknowledgment interval is inserted to reduce the risk of timeout and packet loss. When bandwidth is stable and sufficient, the fragment size is increased to reduce header overhead and improve overall communication efficiency, as shown in the following expression: The formula for real-time monitoring of RTT jitter is: ; in, RTT jitter is used to measure the degree of fluctuation in round-trip time. Indicates standard deviation; This represents the round-trip time delay value obtained from the nth measurement; The formula for adaptively adjusting the fragment size is: ； in, ; , They represent , The size of the slice at any given moment; , These represent the maximum and minimum values ​​of the fragment size, respectively. , These represent the high and low thresholds for RTT jitter, respectively. This indicates the adjustment step size for the slice size, used to adjust the slice size in non-extreme jitter conditions; The formula for the transmission interval is: ； in, This indicates the transmission interval, which is the time interval between two transmissions of a fragment; This represents the average round-trip time delay. , This is the adjustment coefficient.

7. A bandwidth-distortion adaptive multilayer scalable feature compressor according to claim 4, characterized in that... In the error tracking compensation module, an error tracking table is established to record the cumulative reconstruction error of different channels and frequency bands, which will be used for the next round of dynamic bit width allocation and early stop decision, as follows: First, calculate the residual coding of the enhancement layer for each layer, using the following formula: ； in, For the first Layer The residuals of the group; For the first The group's enhancement layer transmits content. For the first Layer Group reconstructed data; Next, the cumulative reconstruction error is calculated, which consists of the sum of the squares of the errors from the previous round and the current round, as shown in the formula: ； in, For the first The first channel in the The cumulative reconstruction error of the wheel; For the first The first channel in the The enhanced layer of the wheel transmits content. For the first The first channel in the Reconstructed data for the wheel; The bit width is dynamically adjusted based on the cumulative reconstruction error, using the following formula: ； in, , They represent the first The first channel in the Wheel width; This is the error threshold; To round down; When the maximum error of all channels is lower than the target error threshold, the enhancement layer coding is stopped to avoid invalid transmission. The stopping condition is: ； in, This represents the maximum cumulative error among all channels in round t; The target error threshold is set.

8. A bandwidth-distortion adaptive multilayer scalable feature compressor according to claim 7, characterized in that... It also includes a keyframe synchronization module, which introduces a synchronization strategy to clear accumulated reconstruction errors by periodically transmitting keyframes. The triggering conditions are as follows: In the first case, an independently encoded keyframe is sent every N training rounds, and reconstruction is performed without relying on historical residuals. The expression is: ； in, Indicates a keyframe; Indicates the current round; Indicates the modulo operation; For synchronization period; In the second scenario, a sudden surge in innovation energy triggers the transmission of keyframes, namely: the total energy of the current round. Compared to the previous round of energy When the ratio exceeds the set energy surge threshold, a keyframe is triggered to re-establish the baseline, as expressed by: ； in, The threshold for a sudden increase in energy; The third scenario occurs when the cumulative distortion exceeds a set threshold, i.e., when the most recent distortion is calculated. The cumulative reconstruction error of each round; when the cumulative reconstruction error exceeds a set threshold... The expression that triggers the transmission of a keyframe is: ； in, For the first The average reconstruction error of the wheel; This is the cumulative distortion threshold; In the fourth scenario, when packet loss occurs for an extended period, the error tracking state is cleared to ensure that the encoding and decoding ends are resynchronized, thus avoiding accuracy loss due to error drift.

9. A bandwidth-distortion adaptive multilayer scalable feature compression method, applied to a bandwidth-distortion adaptive multilayer scalable feature compressor as described in any one of claims 1-8, characterized in that, include: Collect real-time network status, training phase, and accuracy requirements; wherein, the network status includes available bandwidth, round-trip latency, and jitter; Obtain raw data; wherein, the raw data includes raw features sent by the client or raw gradients sent by the server; The original data is mapped to a more energy-concentrated subspace through a learnable orthogonal transformation, and the transformed feature channels are clustered and grouped so that during data transmission, the base layer only carries the center coefficients and low-frequency components of each group, while the enhancement layer encodes the high-frequency details and deviation parts of each group layer by layer with residuals, and performs progressive data reconstruction from coarse to fine to obtain transmittable layered data. Importance scores are calculated based on the transformed hierarchical data, with bit widths dynamically allocated. Adjust the fragment size and fragmentation interval according to the real-time network status; The cumulative reconstruction error for each round is calculated based on the residual coding of the enhancement layer, and the bit width allocation for the next round is dynamically adjusted to obtain compressed data. The total number of layers, the bit width allocation of each layer, and the size of the slices are dynamically determined based on the real-time network status, training stage, and accuracy requirements, so as to dynamically adjust the input original data and minimize the reconstruction distortion between the original data and the reconstructed data; wherein, the reconstructed data is obtained by the decoder decoding and reconstructing the compressed data.

10. A distributed machine learning system comprising several clients and servers, and a bandwidth-distortion adaptive multilayer scalable feature compressor as described in any one of claims 1-8, located between the clients and the servers; the clients send raw features to the multilayer scalable feature compressor for compression, outputting compressed data; the servers receive and aggregate the compressed data, and generate reconstructed data, i.e., global gradients, which are compressed by the multilayer scalable feature compressor to generate compressed gradients and sent to each client; each client updates its local model according to the compressed gradients, and based on the updated model, continues the next round of local training using the raw features until the model converges.

Images