An adaptive channel compression communication split learning system and method thereof, medium

Abstract

The application provides a communication segmentation learning system and method thereof and a medium, and relates to the fields of segmentation learning and communication compression technology.The application comprises a plurality of client ends and server ends, and a channel compression device arranged between the client ends and the server ends.The channel compression device comprises a channel sensitivity modeling module, a rate-distortion adaptive compression module and a cross-client fairness coordination module.In the channel sensitivity modeling module, the channel importance is dynamically evaluated through intermediate layer activation value fusion.In the rate-distortion adaptive compression module, a differentiated quantization and compression strategy is designed based on the sensitivity score, so that the communication overhead is reduced while the key information is retained.Meanwhile, in the cross-client fairness coordination module, a fairness regularization and dual optimization mechanism are used to realize the balanced allocation of communication resources among multiple client ends, so that the global convergence is not affected by excessive compression.The application significantly improves the communication efficiency and training stability of segmentation learning in a complex heterogeneous environment.

Description

Technical Field

[0001] This invention relates to the fields of data communication and distributed deep learning technology, and more specifically, to an adaptive channel compression communication segmentation learning system, method, and medium. Background Technology

[0002] With the rapid development of IoT devices, the amount of data generated at the network edge is exploding. According to predictions by IDC, the total global data volume will reach approximately 221 ZB by 2026. This trend has driven the widespread application of machine learning technology in many fields, such as autonomous driving, smart healthcare, and natural language processing. However, current mainstream centralized learning frameworks require data from terminal devices to be centralized on servers for processing. This approach not only poses serious privacy risks but also results in high communication overhead, making it difficult to meet practical deployment needs. To address these issues, federated learning has been proposed to allow devices to collaboratively train without leaving their local data sources. Nevertheless, with the continuous expansion of model size, especially the emergence of large models containing billions of parameters, resource-constrained edge devices struggle to handle the computational load, becoming a bottleneck for federated learning in edge scenarios.

[0003] To alleviate this problem, segmentation learning has been proposed as an alternative. Segmentation learning significantly reduces the computational burden on devices by layering the model and migrating most of the computational tasks to the server, performing only a small amount of forward propagation on the device side. However, with the increase in the number of participating devices, the amount of "fragmented data," such as intermediate layer activation values ​​and gradients, transmitted between devices and the server increases dramatically, forming a new communication bottleneck and severely impacting training efficiency. To address this challenge, various compression methods have been proposed to reduce the transmission requirements of fragmented data. For example, Eshratifar et al. proposed using an auxiliary neural network to learn a more compact activation representation; Zheng et al. proposed retaining the top k elements with the largest amplitudes in the activations and combining them with a small number of randomly sampled non-top-k elements for compression; and Oh et al. used standard deviation to discard low-variance features and quantize the remaining parts. While these methods reduce the amount of data transmission to some extent, they still have significant shortcomings.

[0004] Existing technologies generally employ a uniform compression strategy, failing to consider the varying information contributions of different channels during model training. In reality, the semantic information carried by different channels plays different roles during training; some channels may be crucial for task performance, while others may be redundant or even noisy. A uniform compression strategy can easily lead to over-compression of important channels or wasted bandwidth on useless channels, thus impacting the final model performance. Furthermore, these methods lack adaptive mechanisms and cannot adjust to the dynamic changes in channel importance during training. Since model training is an evolving process, the contributions of each channel may change at different stages; a fixed compression strategy struggles to adapt to this dynamism, further weakening the balance between communication efficiency and training accuracy.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention aims to provide an adaptive channel compression communication segmentation learning system, method, and medium to solve the problems of high communication overhead, lack of adaptability of compression strategies, and uneven distribution of communication resources in multi-client scenarios in existing segmentation learning frameworks.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] An adaptive channel compression communication segmentation learning system, comprising:

[0009] Several client and server terminals, and a channel compression device located between the client and server terminals;

[0010] The channel compression device includes a channel sensitivity modeling module, a rate-distortion adaptive compression module, and a cross-client fair coordination module.

[0011] The client is used to receive gradient data sent by the server to update the client model, and to perform forward propagation based on local data to generate intermediate layer activation values ​​and upload them to the channel compression device.

[0012] The channel sensitivity modeling module is used to obtain the intermediate layer activation values ​​of each client during the forward propagation process, and to perform correction and fusion through Fisher information, HSIC statistical dependency and consistency to evaluate the importance of each channel and obtain the sensitivity score of each channel.

[0013] The rate-distortion adaptive compression module is used to design differentiated quantization and compression strategies based on the sensitivity score for compression, generate quantization activation values ​​and compression distortion metrics to model the communication bit rate, and send the quantization activation values, compression distortion metrics and communication bit rate to the server.

[0014] The server receives quantized activation values, compression distortion metrics, and communication bitrates, performs forward propagation to generate prediction results and calculate loss, and generates gradients of activation values ​​through backpropagation to send to the client. It also dynamically coordinates the allocation of communication resources among clients in conjunction with a cross-client fair coordination module.

[0015] The cross-client fair coordination module is used to achieve balanced resource allocation in multi-client scenarios by dynamically adjusting the communication budget and communication code rate among each client.

[0016] Preferably, the channel is a feature channel formed after preliminary processing of the raw data input to the client model; in the channel sensitivity modeling module, the sensitivity score of each channel is calculated based on the intermediate layer activation value combined with Fisher information, HSIC dependency, and consistency, specifically as follows:

[0017] First, based on gradient-based Fisher information, the local sensitivity of the capture channel is obtained, resulting in a Fisher sensitivity score, expressed as:

[0018] ;

[0019] in, This represents the Fisher sensitivity score for channel c; Indicates the expected value; Represents the task loss function; This represents the activation value of channel c; This represents the gradient of the loss function with respect to the activation values ​​of channel c; Represents the L2 norm;

[0020] Next, based on the Hilbert–Schmidt Independence Criterion (HSIC), the dependency between channel features and labels is measured using a kernel matrix to obtain the HSIC sensitivity score, expressed as:

[0021] ;

[0022] in, This represents the HSIC sensitivity score of channel c; n represents the total number of clients. The trace of a matrix is ​​calculated by summing the elements on its diagonal. The kernel matrix represents channel c and is used to capture the similarity between channel features; This represents a centered matrix to ensure the zero mean of the kernel matrix; The kernel matrix represents the labels and is used to calculate the dependencies between labels;

[0023] A consistency factor is used to suppress cross-client variance, highlighting globally stable channels in multi-client scenarios and reducing bias caused by data heterogeneity. The resulting consistency sensitivity score is expressed as:

[0024] ;

[0025] ;

[0026] in, The consistency sensitivity score represents channel c. This represents the weighted sensitivity score of channel c on client i, which combines the Fisher sensitivity score and the HSIC sensitivity score. Indicate the variance of client i; This represents a weighted hyperparameter used to balance the contributions of Fisher information and HSIC dependency; , These represent the Fisher sensitivity score and HSIC sensitivity score of channel c on client i, respectively.

[0027] Then, the consistency sensitivity score, Fisher sensitivity score, and HSIC sensitivity score are weighted and fused, and an exponential moving average is used to control the balance between old and new information to reduce instability caused by batch fluctuations. The expression is:

[0028] ;

[0029] in, This represents the smoothness sensitivity score of channel c in round t; This represents the smoothness sensitivity score of channel c in round t-1; The smoothing factor representing the exponential moving average;

[0030] Finally, the smoothness sensitivity score is normalized to maintain scale consistency across different rounds, resulting in the final sensitivity score, expressed as:

[0031] ;

[0032] in, This represents the sensitivity score of channel c after normalization, i.e., the final sensitivity score. Indicates the total number of channels. Represents a variable.

[0033] Preferably, in the rate-distortion adaptive compression module, each channel is assigned an independent quantization step size and zero point for quantization, resulting in differentiated quantization activation values, expressed as:

[0034] ;

[0035] in, This represents the quantization activation value of channel c; This represents the original activation value of channel c; Indicates the zero point of channel c; This indicates the rounding operation; This indicates the quantization step size of channel c;

[0036] Then, based on the sensitivity score and combined with differentiated quantitative activation values, the compression error is dynamically constrained through a sensitivity-weighted distortion function to achieve on-demand allocation of communication resources. The expression is:

[0037] ;

[0038] in, This represents a measure of compression distortion, calculated by the original activation values. With quantized activation value The error between; Indicates the total number of channels; This represents the sensitivity score of channel c after normalization; This represents the original activation value of channel c; This represents the quantization activation value of channel c; This represents the L2 norm.

[0039] Preferably, the method further includes, when modeling the communication code rate, estimating the probability distribution of the quantization symbol based on the quantization activation value using a super-prior learned entropy model, the expression of which is:

[0040] ;

[0041] in, Indicates the quantized activation value The communication code rate; Represents the advanced prior model right Probability estimation; express The expected value.

[0042] Preferably, the server combines task loss, communication bit rate, and compression distortion metrics, and uses a Lagrangian objective function to jointly optimize the channel bit width and step size via gradient descent. The expression is:

[0043] ;

[0044] in, Represents the total loss function; Represents the task loss function; This represents the global model output from the server side. Indicates the true label; , The hyperparameters are represented by weights that balance task loss, communication bitrate, and distortion. Indicates the expected value;

[0045] During training and optimization, the bit width is relaxed to a continuous variable to facilitate gradient optimization and is dynamically updated during iterations, as shown in the formula:

[0046] ;

[0047] ;

[0048] in, , These represent the bit widths of channel c in rounds t and t+1, respectively. , Let represent the quantization step size of channel c in rounds t and t+1, respectively; , Indicates the learning rate; , These represent the gradients of the total loss function L with respect to the bit width and step size, respectively.

[0049] Preferably, in the cross-client fair coordination module, a fairness regularization term is introduced to constrain the difference between the communication bitrate of each client and the global average, as shown in the formula:

[0050] ;

[0051] in, , They represent the client respectively. , The actual communication code rate; Indicates client The actual communication code rate difference; Indicates the total number of clients; This indicates the total number of clients that have communicated in the current round; Represents a variable.

[0052] Preferably, the cross-client fair coordination module further includes setting a dual variable for each client on the server side, and optimizing the objective based on the dual variable using a dual optimization mechanism to measure the difference between the communication budget and the communication code rate, as shown in the formula:

[0053] ;

[0054] ;

[0055] in, , Representing the clients in rounds t and t+1 respectively. The dual variable; This represents the learning rate for dual optimization, used to control the magnitude of dual variable updates; Indicates client Communication budget; Integers are retained, and negative numbers are set to zero; Represents the client in round t. The actual communication code rate;

[0056] Indicates client Optimize the regularized weights during the (t+1)th round of training; This represents the baseline weight, an initial value shared by all clients; This represents the amplification factor, used to control the strength of the influence of the dual variable on the regularization weights.

[0057] Preferably, in the cross-client fair coordination module, the communication budget of each client is dynamically adjusted according to the global situation so that the system gradually converges to a reasonable communication level. The expression is:

[0058] ;

[0059] in, Indicates client The communication budget in round t; This indicates the adjusted communication budget, i.e., the client's... The communication budget in round t+1; Indicates the smoothing coefficient; Indicates the learning rate; Indicates client Communication budget; The gradient representing the fairness regularization; This indicates the difference in actual communication code rate.

[0060] This invention also provides an adaptive channel compression communication segmentation learning method, applied to a channel compression device between a client and a server, comprising:

[0061] The intermediate layer activation values ​​of each client during the forward propagation process are obtained and corrected and fused using Fisher information, HSIC statistical dependency and consistency to evaluate the importance of each channel and obtain the sensitivity score of each channel.

[0062] Based on the sensitivity score, a differentiated quantization and compression strategy is designed for compression, generating quantization activation values ​​and compression distortion metrics to model the communication bit rate, and then sending the quantization activation values, compression distortion metrics, and communication bit rate to the server.

[0063] When the gradient of activation value generation is sent down to the client through backpropagation on the server side, the communication budget and communication bit rate between each client are dynamically adjusted to achieve balanced resource allocation in multi-client scenarios.

[0064] The present invention also provides an adaptive channel compression communication segmentation learning device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement the adaptive channel compression communication segmentation learning method described above.

[0065] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium resides, implement the communication segmentation learning method for adaptive channel compression as described above.

[0066] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0067] This invention introduces three major modules—channel sensitivity modeling, rate-distortion adaptive compression, and cross-client fair coordination—to form a closed-loop optimization structure, thereby reducing communication costs while maintaining model performance.

[0068] Specifically, in the channel sensitivity modeling module, this invention achieves accurate assessment of channel importance by fusing multi-source signals through Fisher information, HSIC statistical dependence and consistency, and combining it with a time smoothing mechanism, effectively mitigating interference caused by data heterogeneity and training fluctuations.

[0069] In the rate-distortion adaptive compression module, differentiated quantization and compression strategies are designed based on sensitivity scores, enabling high-importance channels to obtain more bit resources and low-importance channels to be compressed at a lower cost, thereby reducing the overall communication volume while preserving key information to the greatest extent.

[0070] In the cross-client fair coordination module, the communication budget of each client is dynamically adjusted through fairness regularization and dual optimization mechanism to avoid some clients affecting the convergence of the global model due to excessive compression.

[0071] The three modules work together to form a closed-loop optimization structure, which significantly enhances the convergence speed and robustness of distributed segmentation learning, enabling the method to maintain stable performance in complex heterogeneous environments. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram of the framework of an adaptive channel compression communication segmentation learning system provided in Embodiment 1.

[0074] Figure 2 This is a flowchart of a communication segmentation learning method for adaptive channel compression provided in Example 2.

[0075] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0077] Example 1

[0078] Embodiment 1 of the present invention provides an adaptive channel compression communication segmentation learning method, which can be implemented by an adaptive channel compression communication segmentation learning device (hereinafter referred to as the learning device), and in particular, executed by one or more processors within the learning device.

[0079] In this embodiment, the learning device may be an electronic device equipped with a processor, the processor having a computer program for the adaptive channel compression communication segmentation learning method and the computer program being executable, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0080] In this embodiment, the essence of segmentation learning is to break down a complete machine learning model (such as a neural network) into layers and deploy them on the client and server sides respectively. Together, they constitute a "complete model" capable of performing end-to-end tasks.

[0081] Client-side model: Responsible for the "first half" of the model (usually feature extraction layers, such as convolutional layers and embedding layers), deployed locally on the client, only processing local private data, and not involved in the final prediction / decision logic.

[0082] The server-side global model is responsible for the "second half" of the model (usually the task processing layer, such as the fully connected layer, attention layer, and output layer). It is deployed on the server side, receives intermediate features from the client, completes the final prediction, and calculates the loss.

[0083] In distributed deep learning, the high communication overhead between the client and server is a major bottleneck limiting its application in resource-constrained environments (such as low-bandwidth or high-latency networks). Feature gradients typically have high dimensionality and high precision, resulting in massive data transmission volumes, significantly increasing communication latency and energy consumption, especially when training large-scale models. Existing compression methods, such as simple sparsification or fixed quantization, often struggle to balance compression ratio and model performance, easily leading to information loss or decreased convergence speed. This invention aims to address the problems of high communication costs and lack of flexibility in compression strategies in current segmentation learning. It proposes a fragmented data compression communication segmentation learning system with channel importance recognition capabilities and an adaptive compression mechanism. This system significantly reduces communication burden while maximizing the preservation of information relevant to training performance, thereby achieving efficient, stable, and communication-friendly distributed deep learning.

[0084] like Figure 1 As shown, an adaptive channel compression communication segmentation learning system includes several client and server ends, as well as a channel compression device located between the client and server ends.

[0085] The channel compression device includes a Channel Sensitivity Modeling (CSM) module, a Rate-Distortion Adaptive Compression (RDAC) module, and a Fairness-Aware Coordination (FAC) module. These three parts form a closed loop in each training round: CSM identifies channel importance, RDAC performs adaptive quantization and compression based on this, and FAC coordinates the communication budget among multiple clients and feeds it back to the next round of optimization training, thereby achieving efficient, fair, and stable training.

[0086] On the client side, gradient data sent by the server is received to update the local client model, and forward propagation is performed based on the local data to generate intermediate layer activation values, which are then uploaded to the channel compression device.

[0087] Specifically, the client loads local private data, inputs it into the local model in batches, and performs forward propagation. After passing through the feature extraction layer of the client model, the intermediate layer activation values ​​corresponding to that batch of data are generated.

[0088] In segmentation learning, a channel refers to the feature path formed after the raw data input to the client model has undergone preliminary processing (e.g., applying Gaussian filtering to reduce noise, followed by nonlinear transformation using the ReLU activation function, resulting in a feature vector of dimension d as the feature channel). Essentially, it's the dimension or path in the model used to transmit and represent specific signal features, serving as the core carrier connecting the raw data and model parameters. The importance of channels varies significantly, and different channels contribute unevenly to task performance. Applying the same strategy to all channels during the compression stage can easily lead to the loss of crucial information, thus affecting the model's final convergence accuracy.

[0089] To address this issue, this embodiment proposes a Channel Sensitivity Modeling (CSM) module. Based on intermediate layer activation values, Fisher information, HSIC statistical dependency, and consistency sensitivity scores are calculated, and these scores are weighted and fused to generate a reliable weighted score for each channel.

[0090] Specifically, the channel sensitivity modeling module is used to obtain the intermediate layer activation values ​​of each client during the forward propagation process, and to perform correction and fusion through Fisher information, HSIC statistical dependency (such as Gaussian kernel) and consistency to evaluate the importance of each channel and obtain the sensitivity score of each channel.

[0091] In the channel sensitivity modeling module, input the intermediate layer activation values ​​of all clients and the server-side task labels (for HSIC calculation); output the final sensitivity scores of each channel in this round.

[0092] The intermediate layer activation values ​​are high-dimensional feature matrices generated by the client model after processing local data. They are the only "data interaction carrier" between the client and the server, and are also the core design of segmentation learning for "data privacy protection" and "communication efficiency optimization". The client inputs local raw data (such as medical images and sensor signals) into the local model. After processing by the first half of the network (such as convolutional layers and the ReLU activation function), the output feature map is the "intermediate layer activation value".

[0093] Specifically, in the channel sensitivity modeling module, the sensitivity score for each channel is calculated based on the intermediate layer activation values ​​combined with Fisher information, HSIC dependency, and consistency, as follows:

[0094] First, based on gradient-based Fisher information, the local sensitivity of the capture channel is obtained, resulting in a Fisher sensitivity score, expressed as:

[0095] ;

[0096] in, This represents the Fisher sensitivity score for channel c; Indicates the expected value; Represents the task loss function; This represents the activation value of channel c; This represents the gradient of the loss function with respect to the activation values ​​of channel c; L2 norm represents the Euclidean norm of a vector.

[0097] In this embodiment, Fisher information is a statistical indicator that measures the amount of information the data provides for estimating model parameters. Essentially, it reflects the degree of influence of small changes in parameters on the data distribution. The more Fisher information contained in the data, the more accurate the parameter estimation (the smaller the variance). It is the core theoretical basis for maximum likelihood estimation, Bayesian inference, and model optimization.

[0098] If a channel has a large gradient magnitude during backpropagation, it indicates that it contributes significantly to the optimization of the current task, and therefore needs to be allocated more bit resources during compression.

[0099] However, relying solely on gradient information cannot fully characterize the value of a channel, because gradients primarily reflect instantaneous sensitivity rather than the statistical relationship between channels and labels. To address this deficiency, we further introduce the Hilbert–Schmidt Independence Criterion (HSIC). Based on HSIC, the dependency between channel features and labels is measured using a kernel matrix, yielding the HSIC sensitivity score, expressed as:

[0100] ;

[0101] in, This represents the HSIC sensitivity score of channel c; n represents the total number of clients. The trace of a matrix is ​​calculated by summing the elements on its diagonal. The kernel matrix representing channel c is used to capture the similarity between channel features (e.g., calculated using a Gaussian radial basis function (RBF) kernel). This represents a centered matrix to ensure the zero mean of the kernel matrix; The kernel matrix represents the labels and is used to calculate the dependencies between labels (it can be calculated using a linear kernel based on the one-hot encoded label vectors). If a channel is highly correlated with a label, it means that it contains richer discriminative information.

[0102] In this embodiment, the Hilbert–Schmidt Independence Criterion (HSIC) is a statistical tool based on the Reproducing Kernel Hilbert Space (RKHS). Its core function is to quantify the degree of independence between two random variables. The closer the value is to 0, the stronger the independence of the variables; the larger the value, the stronger the correlation (including linear and nonlinear correlations). It is particularly good at handling correlation analysis of high-dimensional and nonlinear data.

[0103] In multi-client scenarios, the local data distribution across different devices can lead to inconsistent channel sensitivity estimation results. Directly using local estimates can easily result in instability and noise. Therefore, a consistency factor is introduced to suppress cross-client variance, thereby highlighting channels that are more globally stable. Specifically, a consistency factor is used to suppress cross-client variance, highlighting globally stable channels in multi-client scenarios and reducing bias caused by data heterogeneity, resulting in a consistency sensitivity score, expressed as:

[0104] ;

[0105] ;

[0106] in, The consistency sensitivity score represents channel c. This represents the weighted sensitivity score of channel c on client i, which combines the Fisher sensitivity score and the HSIC sensitivity score. Indicate the variance of client i; This represents a weighted hyperparameter used to balance the contributions of Fisher information and HSIC dependency; , These represent the Fisher sensitivity score and HSIC sensitivity score of channel c on client i, respectively.

[0107] This step can significantly reduce bias caused by data heterogeneity and improve overall robustness.

[0108] Then, the consistency sensitivity score, Fisher sensitivity score, and HSIC sensitivity score are weighted and fused, and an exponential moving average is used to control the balance between old and new information to reduce instability caused by batch fluctuations. The expression is:

[0109] ;

[0110] in, This represents the smoothness sensitivity score of channel c in round t; This represents the smoothness sensitivity score of channel c in round t-1; This represents the smoothing factor for the exponential moving average.

[0111] The resulting sensitivity not only reflects the importance of the channel, but also maintains stability and consistency during training.

[0112] Finally, to ensure numerical comparability, the smoothness sensitivity score is normalized to maintain scale consistency across different rounds, resulting in the final sensitivity score, expressed as:

[0113] ;

[0114] in, This represents the sensitivity score of channel c after normalization, i.e., the final sensitivity score. Indicates the total number of channels. Represents a variable.

[0115] After obtaining the sensitivity scores, the next task is to use these scores to design an adaptive compression mechanism. Intuitively, channels with high sensitivity carry more critical information and therefore should be given higher quantization precision; while channels with low sensitivity can be compressed more coarsely without significantly affecting performance. To achieve this goal, Rate-Distortion Adaptive Compression (RDAC) is proposed.

[0116] The rate-distortion adaptive compression module is used to design differentiated quantization and compression strategies based on the sensitivity score, generate quantization activation values ​​and compression distortion metrics to model the communication bit rate, and send the compression distortion metrics and communication bit rate to the server.

[0117] Specifically, in the rate-distortion adaptive compression module, the intermediate layer activation value and channel sensitivity score of each client are input, and the quantization activation value, compression distortion metric and communication bit rate are output.

[0118] Furthermore, each channel is assigned an independent quantization step size and zero point for quantization, resulting in differentiated quantization activation values, expressed as:

[0119] ;

[0120] in, This represents the quantization activation value of channel c; This represents the original activation value of channel c; Indicates the zero point of channel c; This indicates the rounding operation; This indicates the quantization step size of channel c.

[0121] High-sensitivity channels use smaller step sizes to ensure accuracy, while low-sensitivity channels use larger step sizes to reduce bit consumption. This differentiated design avoids a "one-size-fits-all" quantization approach, making the overall compression strategy more flexible.

[0122] Then, based on the sensitivity score and combined with differentiated quantitative activation values, the compression error is dynamically constrained through a sensitivity-weighted distortion function to achieve on-demand allocation of communication resources. The expression is:

[0123] ;

[0124] in, This represents a measure of compression distortion, calculated by the original activation values. With quantized activation value The error between; Indicates the total number of channels; This represents the sensitivity score of channel c after normalization; This represents the original activation value of channel c; This represents the quantization activation value of channel c; This represents the L2 norm.

[0125] While reducing distortion, we also need to accurately model the communication code rate. To this end, based on the quantization activation values, a super-prior learned entropy model is used to estimate the probability distribution of the quantization symbols to model the communication code rate, expressed as:

[0126] ;

[0127] in, Indicates the quantized activation value The communication code rate; Represents the advanced prior model right Probability estimation; express The expected value.

[0128] This model can provide a differentiable bitrate approximation in end-to-end training, thus effectively balancing communication overhead and task accuracy.

[0129] To balance task loss, bit rate, and distortion, we combine them into a unified Lagrangian objective function and jointly optimize the channel width and step size using gradient descent.

[0130] Specifically, the server receives the quantized activation value, compression distortion metric, and communication bit rate, performs forward propagation to generate prediction results and calculate the loss, and generates the gradient of the activation value through backpropagation and sends it to the client. At the same time, it dynamically coordinates the allocation of communication resources among clients in conjunction with the cross-client fair coordination module.

[0131] During the forward propagation process, the server inputs the quantized activation values ​​of each client into the global model to perform forward propagation, generating prediction results, which are then combined with the real labels. Calculate task loss Then, combining task loss, communication bit rate, and compression distortion metrics, a Lagrangian objective function is used to jointly optimize the channel bit width and step size via gradient descent. The expression is:

[0132] ;

[0133] in, Represents the total loss function; Represents the task loss function; This represents the global model output from the server side. Indicates the true label; , The hyperparameters are represented by weights that balance task loss, communication bitrate, and distortion. This represents the expected value.

[0134] During training and optimization, the bit width is relaxed to a continuous variable to facilitate gradient optimization and is dynamically updated during iterations, as shown in the formula:

[0135] ;

[0136] ;

[0137] in, , These represent the bit widths of channel c in rounds t and t+1, respectively. , Let represent the quantization step size of channel c in rounds t and t+1, respectively; , Indicates the learning rate; , These represent the gradients of the total loss function L with respect to the bit width and step size, respectively.

[0138] During backpropagation, the gradient of the total loss function with respect to the quantized activation values ​​is calculated and used for client model updates.

[0139] In multi-client segmentation learning scenarios, differences in bandwidth and data distribution across different devices can lead to imbalances in compression strategies. If some clients are forced to over-compress, the quality of their model updates will degrade, thus affecting the convergence of the global model. To address this issue, we designed a cross-client fair coordination (FAC) module to ensure the overall balance of training.

[0140] The cross-client fair coordination module is used to achieve balanced resource allocation in multi-client scenarios by dynamically adjusting the communication budget and communication bitrate among each client.

[0141] Specifically, in the cross-client fair coordination module, a fairness regularization term is introduced to constrain the difference between each client's communication bitrate and the global average, ensuring that resource allocation is not excessively skewed. The formula is as follows:

[0142] ;

[0143] in, , They represent the client respectively. , The actual communication code rate; Indicates client The actual communication code rate difference; Indicates the total number of clients; This indicates the total number of clients that have communicated in the current round; Represents a variable.

[0144] Relying solely on regularization terms is insufficient for dynamic adaptation; therefore, a dual optimization mechanism is introduced in FAC. The server sets a dual variable for each client and, based on this dual variable, uses the dual optimization mechanism to optimize the objective, measuring the difference between the communication budget and the communication bit rate. The formula is:

[0145] ;

[0146] ;

[0147] in, , Representing the clients in rounds t and t+1 respectively. The dual variable; This represents the learning rate for dual optimization, used to control the magnitude of dual variable updates; Indicates client Communication budget; Integers are retained, and negative numbers are set to zero; Represents the client in round t. The actual communication code rate;

[0148] Indicates client Optimize the regularized weights during the (t+1)th round of training; This represents the baseline weight, an initial value shared by all clients; This represents the amplification factor, used to control the strength of the influence of the dual variable on the regularization weights.

[0149] In this way, clients that exceed the budget will be penalized more severely in subsequent rounds, while clients that stay below the budget will have more freedom.

[0150] The cross-client fair coordination module (FAC) also includes dynamically adjusting the communication budget of each client based on the global situation, so that the system gradually converges to a reasonable communication level. The expression is:

[0151] ;

[0152] in, Indicates client The communication budget in round t; This indicates the adjusted communication budget, i.e., the client's... The communication budget in round t+1; Indicates the smoothing coefficient; Indicates the learning rate; Indicates client Communication budget; The gradient representing the fairness regularization; This indicates the difference in actual communication code rate.

[0153] This mechanism ensures overall efficiency while preventing individual clients from being unfairly restricted for extended periods.

[0154] During overall operation, the three main modules of the channel compression device form a closed-loop optimization structure. The channel sensitivity modeling module generates sensitivity scores, the rate-distortion adaptive compression module designs compression strategies based on the sensitivity scores and generates compressed intermediate layer activation values ​​and gradient data, and the cross-client fair coordination module dynamically adjusts the communication budget and achieves balanced global resource allocation through communication budget and bitrate feedback information. The modules interact with each other via data flow to ensure the system maintains stable performance in complex heterogeneous environments.

[0155] In practical applications, this communication segmentation learning system is suitable for distributed deep learning scenarios, especially in resource-constrained edge computing environments. For example, in smart home networks, multiple IoT devices need to collaborate to complete image classification tasks, but the bandwidth between devices is limited and their computing capabilities vary significantly. In this case, this communication segmentation learning system can significantly reduce communication overhead while ensuring model performance.

[0156] Taking a smart home network as an example, suppose multiple camera devices need to collaboratively complete an object detection task. Due to the varying data quality collected by different devices and limited network bandwidth, traditional segmentation learning methods are prone to communication bottlenecks and model performance degradation. Using the framework of this invention, the channel sensitivity modeling module can accurately identify the importance of key channels, and the rate-distortion adaptive compression module designs differentiated compression strategies based on sensitivity scores, thereby reducing communication overhead while retaining important information. The cross-client fair coordination module ensures that each device can complete the training task within a reasonable range through dynamic budget adjustment and dual optimization mechanisms, avoiding model performance degradation caused by over-compression. Compared to existing technologies, this framework achieves higher model accuracy and faster convergence speed under the same communication budget, while significantly improving the robustness and fairness of the system.

[0157] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0158] This invention effectively solves the problems of high communication costs, lack of flexibility in compression strategies, and uneven resource allocation in multi-client scenarios by introducing three major modules: channel sensitivity modeling, rate-distortion adaptive compression, and cross-client fair coordination. It has broad application prospects and significant technical advantages.

[0159] Example 2

[0160] like Figure 2 As shown, the second embodiment of the present invention also provides an adaptive channel compression communication segmentation learning method, applied to a channel compression device between a client and a server, comprising:

[0161] The intermediate layer activation values ​​of each client during the forward propagation process are obtained and corrected and fused using Fisher information, HSIC statistical dependency and consistency to evaluate the importance of each channel and obtain the sensitivity score of each channel.

[0162] Based on the sensitivity score, a differentiated quantization and compression strategy is designed for compression, generating quantization activation values ​​and compression distortion metrics to model the communication bit rate, and then sending the quantization activation values, compression distortion metrics, and communication bit rate to the server.

[0163] When the gradient of activation value generation is sent down to the client through backpropagation on the server side, the communication budget and communication bit rate between each client are dynamically adjusted to achieve balanced resource allocation in multi-client scenarios.

[0164] Example 3

[0165] The third embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device where the computer-readable storage medium is located, implement the communication segmentation learning method for adaptive channel compression as described above.

[0166] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0167] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0168] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0170] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0171] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0172] The use of "first" and "second" in the embodiments is merely to distinguish similar objects and does not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0173] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A communication segmentation learning system with adaptive channel compression, characterized in that, include: Several client and server terminals, and a channel compression device located between the client and server terminals; The channel compression device includes a channel sensitivity modeling module, a rate-distortion adaptive compression module, and a cross-client fair coordination module. The client is used to receive gradient data sent by the server to update the client model, and to perform forward propagation based on local data to generate intermediate layer activation values ​​and upload them to the channel compression device. The channel sensitivity modeling module is used to obtain the intermediate layer activation values ​​of each client during the forward propagation process, and to refine and fuse them using Fisher information, HSIC statistical dependency, and consistency to evaluate the importance of each channel and obtain a sensitivity score for each channel. Specifically: First, the local sensitivity of the Fisher information capture channel based on gradient is used to obtain the Fisher sensitivity score; Next, based on the Hilbert–Schmidt independence criterion (HSIC), the dependency between channel features and labels is measured by the kernel matrix to obtain the HSIC sensitivity score; A consistency factor is used to suppress cross-client variance, highlighting the globally stable channel in multi-client scenarios and reducing bias caused by data heterogeneity, thus obtaining a consistency sensitivity score. Then, the consistency sensitivity score, Fisher sensitivity score and HSIC sensitivity score are weighted and fused, and the balance between new and old information is controlled by exponential moving average to obtain a smooth sensitivity score, so as to reduce the instability caused by batch fluctuations. Finally, the smoothness sensitivity score is normalized to maintain scale consistency across different rounds, resulting in the final sensitivity score. The rate-distortion adaptive compression module is used to design differentiated quantization and compression strategies based on the sensitivity score for compression, generate quantization activation values ​​and compression distortion metrics to model the communication bit rate, and send the quantization activation values, compression distortion metrics and communication bit rate to the server. The server receives quantized activation values, compression distortion metrics, and communication bitrates, performs forward propagation to generate prediction results and calculate loss, and generates gradients of activation values ​​through backpropagation to send to the client. It also dynamically coordinates the allocation of communication resources among clients in conjunction with a cross-client fair coordination module. The cross-client fair coordination module is used to achieve balanced resource allocation in multi-client scenarios by dynamically adjusting the communication budget and communication code rate among each client.

2. The communication segmentation learning system with adaptive channel compression according to claim 1, characterized in that... The channel is a feature channel formed after preliminary processing of the raw data input to the client model; in the channel sensitivity modeling module, the expression for the Fisher sensitivity score is: ； in, This represents the Fisher sensitivity score for channel c; Indicates the expected value; Represents the task loss function; This represents the activation value of channel c; This represents the gradient of the loss function with respect to the activation values ​​of channel c; Represents the L2 norm; The expression for the HSIC sensitivity score is: ； in, This represents the HSIC sensitivity score of channel c; n represents the total number of clients. The trace of a matrix is ​​calculated by summing the elements on its diagonal. The kernel matrix represents channel c and is used to capture the similarity between channel features; This represents a centered matrix to ensure the zero mean of the kernel matrix; The kernel matrix represents the labels and is used to calculate the dependencies between labels; The expression for the consistency sensitivity score is: ； ； in, The consistency sensitivity score represents channel c. This represents the weighted sensitivity score of channel c on client i, which combines the Fisher sensitivity score and the HSIC sensitivity score. Indicate the variance of client i; This represents a weighted hyperparameter used to balance the contributions of Fisher information and HSIC dependency; , These represent the Fisher sensitivity score and HSIC sensitivity score of channel c on client i, respectively. The expression for the smoothness sensitivity score is: ； in, This represents the smoothness sensitivity score of channel c in round t; This represents the smoothness sensitivity score of channel c in round t-1; The smoothing factor representing the exponential moving average; The final expression for the sensitivity score is: ； in, This represents the sensitivity score of channel c after normalization, i.e., the final sensitivity score. Indicates the total number of channels. Represents a variable.

3. The communication segmentation learning system with adaptive channel compression according to claim 2, characterized in that... In the rate-distortion adaptive compression module, each channel is assigned an independent quantization step size and zero point for quantization, resulting in differentiated quantization activation values, expressed as: ； in, This represents the quantization activation value of channel c; This represents the original activation value of channel c; Indicates the zero point of channel c; This indicates the rounding operation; This indicates the quantization step size of channel c; Then, based on the sensitivity score and combined with differentiated quantitative activation values, the compression error is dynamically constrained through a sensitivity-weighted distortion function to achieve on-demand allocation of communication resources. The expression is: ； in, This represents a measure of compression distortion, calculated by the original activation values. With quantized activation value The error between; Indicates the total number of channels; This represents the sensitivity score of channel c after normalization; This represents the original activation value of channel c; This represents the quantization activation value of channel c; This represents the L2 norm.

4. The communication segmentation learning system with adaptive channel compression according to claim 1, characterized in that... When modeling the communication code rate, based on the quantization activation value, a super-prior learned entropy model is used to estimate the probability distribution of the quantization symbol, the expression of which is: ； in, Indicates the quantized activation value The communication code rate; Represents the advanced prior model right Probability estimation; express The expected value.

5. The communication segmentation learning system with adaptive channel compression according to claim 4, characterized in that... It also includes, on the server side, combining task loss, communication bit rate, and compression distortion metrics, using a Lagrangian objective function and gradient descent to jointly optimize the channel bit width and step size, expressed as: ； in, Represents the total loss function; Represents the task loss function; This represents the global model output from the server side. Indicates the true label; , The hyperparameters are represented by weights that balance task loss, communication bitrate, and distortion. Indicates the expected value; During training and optimization, the bit width is relaxed to a continuous variable to facilitate gradient optimization and is dynamically updated during iterations, as shown in the formula: ； ； in, , These represent the bit widths of channel c in rounds t and t+1, respectively. , Let represent the quantization step size of channel c in rounds t and t+1, respectively; , Indicates the learning rate; , These represent the gradients of the total loss function L with respect to the bit width and step size, respectively.

6. The communication segmentation learning system with adaptive channel compression according to claim 5, characterized in that... In the cross-client fair coordination module, a fairness regularization term is introduced to constrain the difference between each client's communication bitrate and the global average. The formula is as follows: ； in, , They represent the client respectively. , The actual communication code rate; Indicates client The actual communication code rate difference; Indicates the total number of clients; This indicates the total number of clients that have communicated in the current round; Represents a variable.

7. The communication segmentation learning system with adaptive channel compression according to claim 6, characterized in that... The cross-client fair coordination module also includes setting a dual variable for each client on the server side, and using a dual optimization mechanism based on the dual variable to optimize the objective, in order to measure the difference between the communication budget and the communication bit rate. The formula is as follows: ； ； in, , Representing the clients in rounds t and t+1 respectively. The dual variable; This represents the learning rate for dual optimization, used to control the magnitude of dual variable updates; Indicates client Communication budget; Integers are retained, and negative numbers are set to zero; Represents the client in round t. The actual communication code rate; Indicates client Optimize the regularized weights during the (t+1)th round of training; This represents the baseline weight, an initial value shared by all clients; This represents the amplification factor, used to control the strength of the influence of the dual variable on the regularization weights.

8. The communication segmentation learning system with adaptive channel compression according to claim 7, characterized in that... In the cross-client fair coordination module, the communication budget of each client is dynamically adjusted based on the global situation to gradually converge the system to a reasonable communication level. The expression is: ； in, Indicates client The communication budget in round t; This indicates the adjusted communication budget, i.e., the client's... The communication budget in round t+1; Indicates the smoothing coefficient; Indicates the learning rate; Indicates client Communication budget; The gradient representing the fairness regularization; This indicates the difference in actual communication code rate.

9. An adaptive channel compression communication segmentation learning method, applied to a channel compression device between a client and a server, characterized in that, include: The intermediate layer activation values ​​of each client during the forward propagation process are obtained and corrected and fused using Fisher information, HSIC statistical dependency and consistency to evaluate the importance of each channel and obtain the sensitivity score of each channel. Specifically, firstly, the local sensitivity of the channel is captured based on the gradient-based Fisher information to obtain the Fisher sensitivity score. Next, based on the Hilbert–Schmidt independence criterion (HSIC), the dependency between channel features and labels is measured by the kernel matrix to obtain the HSIC sensitivity score; A consistency factor is used to suppress cross-client variance, highlighting the globally stable channel in multi-client scenarios and reducing bias caused by data heterogeneity, thus obtaining a consistency sensitivity score. Then, the consistency sensitivity score, Fisher sensitivity score and HSIC sensitivity score are weighted and fused, and the balance between new and old information is controlled by exponential moving average to obtain a smooth sensitivity score, so as to reduce the instability caused by batch fluctuations. Finally, the smoothness sensitivity score is normalized to maintain scale consistency across different rounds, resulting in the final sensitivity score. Based on the sensitivity score, a differentiated quantization and compression strategy is designed for compression, generating quantization activation values ​​and compression distortion metrics to model the communication bit rate, and then sending the quantization activation values, compression distortion metrics, and communication bit rate to the server. When the gradient of activation value generation is sent down to the client through backpropagation on the server side, the communication budget and communication bit rate between each client are dynamically adjusted to achieve balanced resource allocation in multi-client scenarios.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by the processor of the device on which the computer-readable storage medium resides, implement the communication segmentation learning method for adaptive channel compression as described in claim 9.

Images