High-throughput splitting federated learning method based on activated cache

Abstract

The invention discloses a high-throughput split federated learning method based on an activated cache. The method comprises the following steps: carrying out system initialization and network structure variable definition; performing client parameter synchronization, local feature extraction and independent parameter updating; performing activation feature drift asynchronous calculation and maximum old-age window updating; performing double-branch independent training of the server side based on hybrid retrieval; global federal aggregation based on data volume weighting is carried out; and performing a multi-server integrated reasoning test of the global model. Through an innovative drift sensing activation reuse mechanism and a multi-GPU communication-free integrated training strategy, the throughput performance of the split federated learning system in heterogeneous equipment and a non-independent identically distributed data environment is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of, specifically to a high-throughput split federated learning method based on activation caching. Background Technology

[0002] In recent years, with the widespread adoption of IoT devices and the improvement of edge computing capabilities, massive amounts of data are continuously generated at the network edge. Traditional centralized machine learning paradigms require all this distributed data to be aggregated to a central server for model training. This approach not only consumes enormous network bandwidth but also poses a serious risk of data privacy breaches. To address this challenge, Federated Learning (FL) has emerged. Federated Learning allows a large number of edge devices (clients) to collaboratively train a global model by exchanging model parameters or gradients, while keeping their local data within their domain. However, traditional federated learning requires edge devices to possess complete model training capabilities, which is a huge computational burden for IoT devices, whose computing resources, storage space, and battery power are highly limited.

[0003] To address the challenge of resource-constrained devices struggling to train complete models, split federated learning, as an innovative distributed architecture, has been proposed and gained widespread attention. The core idea of ​​split federated learning is to divide a complete deep neural network into two parts: a device-side model (front-end) and a server-side model (back-end). In a typical training process, each edge device uses only local data to perform the forward propagation of its device-side model, generating intermediate feature representations (activations), which are then sent to the cloud server via the network. Upon receiving this activation data, the server inputs it into its server-side model to continue the forward propagation of the remaining network layers and calculates the loss value.

[0004] Existing technologies have significant limitations in addressing throughput bottlenecks, often only optimizing one aspect in isolation while neglecting the system's overall bottleneck effect.

[0005] Firstly, existing server-side optimization techniques (such as multi-GPU data parallelism) attempt to increase outbound throughput by increasing computing resources. However, these methods ignore the dynamic changes in inbound throughput. In real-world heterogeneous federated learning environments, there are significant differences in the computing power and local data volume of each edge device. At the start of a training epoch, all devices concurrently generate activation data, and inbound throughput reaches its peak. However, as some faster or smaller devices complete their local tasks ahead of schedule, the number of active devices drops sharply. When only a few "stragglers" with extremely poor computing power remain in the system, the rate at which newly generated activation data reaches the server will plummet. At this point, even if the server is equipped with a large multi-GPU cluster, it will be forced into an idle waiting state due to insufficient data in the activation queue, resulting in an extremely high GPU memory access miss rate and severely limited outbound throughput.

[0006] Secondly, existing edge device optimization techniques (such as introducing device-side local auxiliary losses or pipelined parallelism) attempt to improve inbound throughput. However, these methods fail to address another major pain point on the server side: limited by the weak communication bandwidth and limited local data volume of edge devices, the activation data transmitted to the server typically has a very small batch size. When processing these scattered, small batches of activation data, the server cannot fully utilize the powerful parallel matrix operation capabilities of the GPU, resulting in extremely low server-side memory utilization. Furthermore, forcibly aggregating these small batches into large batches on the server side introduces significant latency due to waiting for a sufficient number of activation data.

[0007] Thirdly, traditional split learning architectures typically require the server to perform backpropagation and calculate gradients for the split activation data after calculating the loss. These gradients are then sent back to the edge device via the downlink so that the device can continue to perform backpropagation to update the front-end model parameters. This two-way communication mechanism not only consumes a large amount of downlink network bandwidth but also requires the edge device to be blocked and suspended to wait for the gradients sent back by the server before it can continue to complete backpropagation. This severe synchronous waiting further slows down the overall training pace.

[0008] In summary, existing split-feeding learning systems cannot simultaneously maintain high inbound and outbound throughput in heterogeneous device environments, and the complex gradient backpropagation mechanism severely restricts the efficiency of edge devices. Therefore, there is an urgent need in this field for a novel method that can decouple the strict temporal dependency between device-side data generation and server-side data consumption, eliminate the burden of gradient backpropagation, fully explore and utilize the computational potential of the server, and achieve efficient model training. Summary of the Invention

[0009] The purpose of this invention is to provide a high-throughput split federated learning method based on activation caching to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: A high-throughput split federated learning method based on activation caching includes: S1 performs system initialization and defines network structure variables, including: S1.1, Perform model segmentation and initial model distribution; S1.2, perform activation cache library initialization; S2 performs client parameter synchronization, local feature extraction, and independent parameter updates, including: S2.1, perform global model reception and forward propagation with auxiliary layers; S2.2, perform local joint backpropagation; S3 performs asynchronous calculation of activation feature drift and updates the maximum staleness window, including: S3.1, perform local feature drift calculation; S3.2, perform global drift summaries and window adjustments; S4, perform server-side dual-branch independent training based on hybrid retrieval, including: S4.1, Construct a training branch based on a real-time activation queue; S4.2, Construct a fallback training branch based on activation cache; S5 performs global federated aggregation based on data volume weighting, including: S5.1, upload client parameters and data volume; S5.2, perform weighted model aggregation; S6, conduct multi-server integrated inference tests of the global model, including: S6.1, perform Model Boosting forward feature propagation; S6.2 performs Model Bagging, aggregating logical values ​​from multiple backends.

[0011] Furthermore, S1.1 includes: The complete global deep neural network is segmented into preset layers, and the server initializes the global front-end device network parameters. With global auxiliary classification head network parameters φ 0 And before the start of round 0 communication, and φ 0Distribute to all participating clients; for the i-th participating client, the network of the front-end device it receives and deploys is defined as... θ d,i For the network parameters of the front-end device network, θ d,i This represents the parameters of the model at the edge device with index i; simultaneously, the local auxiliary classification head network accessed after layering is defined as... φ i The network parameters are defined for the local auxiliary classification head network; the server is equipped with K independent GPUs, and the backend server network deployed on the k-th GPU is defined as follows: θ s,k These are the model parameters for the network of the k-th backend server.

[0012] Furthermore, S1.2 includes: The server allocates an activation cache in memory and on disk. The cache records contain device indexes, batch indexes, sample indexes, the cache round number, and an activation data tensor. Represented as: , Where i represents the device index participating in training, b represents the data batch index of the device, s represents the sample index within that batch, and r cache This indicates the communication round in which the activation data is cached. For the rth cache During the round, device i generates an activation data tensor from the b-th batch and the s-th sample; simultaneously, the server maintains a lightweight cache metadata table, CacheTable, in memory. This cache metadata table only stores basic index information and the physical address of the actual activation data on disk; the formula for this cache metadata table is defined as: , Where N, B, and S correspond to the total number of devices participating in training, the number of data batches per device, and the number of samples in each batch, respectively, and addr is the actual storage address of the activation data tensor on the server disk.

[0013] Furthermore, S2.1 includes: At the start of the t-th communication round, the selected i-th active client first receives the latest global front-end network parameters from the server after the aggregation in the previous round. With global auxiliary classification head parameter φ t This is used to override the local model parameters, i.e., to assign values. as well as Subsequently, the client sends the local input sample x into the front-end device network for forward propagation, generating intermediate activation data a. i The client will activate intermediate data ai The activation data (a) is asynchronously sent to the server's receive queue via the uplink network. i Simultaneously, the data is directly input into the local auxiliary classification head network to obtain local prediction outputs. , The forward propagation formula is: , in, Indicates Front-end device network with parameters Indicates An auxiliary classification head network with parameters.

[0014] Furthermore, S2.2 includes: The client utilizes local prediction output Calculate the local loss L with the real label y local,i : , in, This represents the loss function used to measure the difference between the local predicted output and the true label; The client independently executes the backpropagation of the auxiliary classification head and the front-end device network in sequence; the backpropagation of the front-end device network continues to derive through the gradients returned by the auxiliary layer.

[0015] Furthermore, S3.1 includes: In the r-th communication round, when the server receives the new round of activation data uploaded by the client... At that time, before storing it in the queue and overwriting the cache, the server background process asynchronously retrieves historical cache activation data from the same source in the cache metadata table. Canonical correlation analysis was used to calculate the similarity between the two samples, and the average activation drift per round was derived. : , In the formula, This represents the canonical correlation function. Used to quantify the degree of drift between currently active data and historical cached active data, r cache This indicates the communication round when the sample was last cached, rr cache This indicates the time interval between two cached cycles.

[0016] Furthermore, S3.2 includes: By summing the drift amounts of all samples in the current round, the global activation drift for each round can be calculated. , The calculation formula is: , In the formula, M r This represents the total number of samples participating in the evaluation in the current r-th round, calculated using the following formula: , in, Let r be the set of clients participating in the training round. For the batch set transmitted by client i, For the sample set within the corresponding batch, This indicates the number of samples in the sample set; Server update maximum staleness window threshold w r The calculation formula is: , In the formula, △ represents the difference in rounds between the current round and the cache round; δ r This represents the preset drift tolerance for round r. This represents the global activation drift in round t. These are terms in a multiplication process.

[0017] Furthermore, S4.1 includes: When new activation data arrives in the receive queue At that time; the server utilized Update the old record in the cached metadata table; subsequently, the k-th server performs a forward propagation to obtain the output. And calculate server-side losses. , where y s,k This represents the k-th network prediction result on the server side. Indicated by θ s,k For the server network with parameters, θ s,k Let the model parameters of the k-th backend server network be denoted as . The backpropagation and parameter update formulas for the server network are as follows: , In the formula, and Let η represent the server network parameters for round t+1 and round t, respectively. s This represents the training learning rate on the server side. Indicates the server-side loss relative to the parameter The calculated gradient.

[0018] Furthermore, S4.2 includes: If no new activation data is detected in the receiving queue and a timeout is triggered, the data loader will switch to the cache metadata table to select historical cached activation data. The cache management module requires that the round difference △ of the cached data must meet the following requirements: , Where r represents the current communication round, r cache This indicates the communication round when historical activation data is cached; After verification, the k-th server network uses this cached data to perform forward propagation, expressed as: , The obtained output is then used to perform loss calculations on the server side. , And perform the corresponding backpropagation update: , In the formula, This indicates the predicted output result obtained by the server network based on historical cached activation data. Indicated by θ s,k Server network functions with parameters This represents the server-side simulation loss calculated based on the predicted output. This represents the loss function used to measure the difference between the predicted output and the true label. and Let η represent the server network parameters for round t+1 and round t, respectively. s This represents the training learning rate on the server side. This indicates that the parameter θ is based on the simulated loss. s,k The calculated gradient.

[0019] Furthermore, S5.1 includes: At the end of the t-th round of training, all clients participating in this round will update their front-end device network parameters. Auxiliary classification head network parameters φ t+1 and the amount of local training data D processed by each. i Upload them together to the cloud parameter server; S5.2 includes: Parameter server calculates the total data volume D: , Based on the proportion of each client's data volume to the total data volume, adaptive weighted FedAvg aggregation is performed on both the front-end device network and the auxiliary classification head network. The aggregation formula for the front-end device network is as follows: , The aggregation formula for the auxiliary classification head network is: , In the formula, and φ t+1These represent the next round of global front-end device network parameters and global auxiliary classification head network parameters after aggregation, respectively. and Let D represent the front-end device network parameters and auxiliary classification head network parameters updated locally by the i-th client, respectively. N represents the total number of clients participating in the aggregation. i Let represent the amount of local training data owned by the i-th client, and D represent the total amount of local training data of all clients participating in the aggregation.

[0020] Furthermore, S6.1 includes: employing a cascaded forward propagation structure during the test inference phase; inputting test sample x test To the already aggregated global front-end device model In, where θ d For the global front-end device model parameters, perform forward propagation to extract common feature representations, generate and output global intermediate activation data z: , The global intermediate activation data z is then passed as input to a parallel network of K independent backend servers. , where θ s,k The model parameters are for the k-th backend server network; each of the K independent backend server networks performs forward propagation to obtain its own independent prediction output y. k : , S6.2 includes: The server performs a Bagging aggregation operation on the prediction outputs of K parallel back-end sub-models, eliminates the heterogeneous variance of individual models by calculating the arithmetic mean, and generates and outputs the final high-precision prediction result. Its formula is: .

[0021] Compared with the prior art, the beneficial effects of the present invention are:

[0022] This invention significantly improves the throughput performance of a split federated learning system in heterogeneous devices and non-independent, co-distributed data environments through an innovative drift-aware activation reuse mechanism and a multi-GPU communication-free integrated training strategy. By employing unidirectional data flow topology optimization technology, the bidirectional interactive federated learning framework is abstracted into a gradient-free backpropagation unidirectional communication process. By introducing a local auxiliary classification head at the device end to independently complete local backpropagation, the synchronous blocking of edge-cloud training is completely decoupled, reducing downlink communication overhead to zero. Activation drift in the feature space is quantified by calculating the canonical correlation analysis (CCA) similarity between newly generated activation data and cached activation data. A quantitative prediction model of global activation drift and training epochs is established to determine the safety boundary for historical data reuse at different training stages. A hybrid activation retrieval framework of "queue priority, cache fallback" is designed, transforming the management of inbound data flow into an asynchronous supply process decoupled between edge and cloud, with the objective function of maximizing server outbound throughput and hardware utilization, ensuring the system maintains full-load operation.

[0023] To address the issue of a sharp drop in inbound throughput in a decentralized federated learning system operating in a heterogeneous device environment, and considering the long-tail latency inherent in traditional synchronous mechanisms, a decoupled asynchronous data supply method is needed. This method employs an activation cache library to persistently store activation features from historical rounds in real time, and uses a drift-aware control mechanism to dynamically calculate the maximum staleness window based on the global drift. Based on this time window, when the real-time activation queue becomes empty due to active devices exiting during the later stages of a training round, the data loader automatically samples high-quality, unexpired historical data from the cache library, seamlessly filling the gaps in batch data processing. This completely breaks the bottleneck effect in inbound throughput, ensuring a constant supply of inbound data to the server.

[0024] To address the limitations of outbound throughput on multi-GPU servers and the low hardware utilization caused by small batches of data, and considering the significant communication overhead of parameter synchronization in traditional data parallelism, this study avoids frequent blocking gradient swapping and model aggregation. Instead, it employs a multi-GPU communication-free ensemble training strategy to achieve near-linear scaling of computing power. Sufficient scattered activation data is pre-extracted from the cache and freely reassembled into large batches, which are then distributed to individual GPUs for forward and backward propagation updates, fully utilizing GPU memory bandwidth. Simultaneously, by adaptively adjusting the cache window—using a small window during the early stages of training when the model changes rapidly and a large window during the later stages when the model is stable—noisy data injection is fundamentally prevented. During inference, an ensemble strategy based on the average ensemble of the logits output from multiple backend models is employed, effectively reducing the variance of individual models and resulting in a final model with test accuracy exceeding that of traditional baseline systems.

[0025] This invention dynamically decouples edge-cloud synchronization dependencies based on the heterogeneous computing states of the devices. Through unidirectional network adaptation, dynamic cache retrieval, and communication-free integrated training strategies, it avoids the throughput performance bottlenecks of traditional split-fed learning. Simultaneously, it reduces the energy consumption and memory footprint of edge devices, expands the system's compatibility with low-end IoT devices, and significantly improves the overall training efficiency. Whether in the initial training phase where devices concurrently generate large amounts of data or in the long-tail phase where few active devices exist, the system achieves an optimized balance between inbound and outbound throughput, providing strong support for efficient privacy-preserving model training in complex edge network environments. Attached Figure Description

[0026] Figure 1 This is a flowchart of a high-throughput split federated learning method based on activation caching, which is claimed in this invention.

[0027] Figure 2 This is a schematic diagram illustrating the process of aggregating multiple backend logical values ​​in Model Bagging according to the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figure 1 A high-throughput split federated learning method based on activation caching includes:

[0030] S1 performs system initialization and defines network structure variables, including:

[0031] S1.1, Perform model segmentation and initial model distribution, including:

[0032] The complete global deep neural network is segmented into preset layers, and the server initializes the global front-end device network parameters. With global auxiliary classification head network parameters φ 0 And before the start of round 0 communication, and φ 0 Distribute to all participating clients; for the i-th participating client, the network of the front-end device it receives and deploys is defined as... θ d,i For the network parameters of the front-end device network, θ d,iThis represents the parameters of the model at the edge device with index i; simultaneously, the local auxiliary classification head network accessed after layering is defined as... φ i The network parameters are defined for the local auxiliary classification head network; the server is equipped with K independent GPUs, and the backend server network deployed on the k-th GPU is defined as follows: , by parameter θ s,k Characterization.

[0033] S1.2, perform activation cache library initialization, including:

[0034] The server allocates an activation cache in memory and on disk. The cache records contain device indexes, batch indexes, sample indexes, the cache round number, and an activation data tensor. Represented as:

[0035]

[0036] Where i represents the device index participating in training, b represents the data batch index of that device, s represents the sample index within that batch, and r cache This indicates the communication round in which the activation data is cached. For the rth cache During the round, device i generates an activation data tensor from the b-th batch and the s-th sample; simultaneously, the server maintains a lightweight cache metadata table, CacheTable, in memory. This cache metadata table only stores basic index information and the physical address of the actual activation data on disk; the formula for this cache metadata table is defined as:

[0037]

[0038] Where N, B, and S correspond to the total number of devices participating in the training of this federated learning system, the number of data batches for each device, and the number of samples in each batch, respectively, and addr is the actual storage address of the activation data tensor on the server disk.

[0039] S2 performs client parameter synchronization, local feature extraction, and independent parameter updates, including:

[0040] S2.1, Perform global model reception and forward propagation with auxiliary layers, including:

[0041] At the start of the t-th communication round, the selected i-th active client first receives the latest global front-end network parameters from the server after the aggregation in the previous round. With global auxiliary classification head parameter φ t This is used to override the local model parameters, i.e., to assign values. as well as Subsequently, the client sends the local input sample x into the front-end device network for forward propagation, generating intermediate activation data a. i The client will activate intermediate data a i The activation data (a) is asynchronously sent to the server's receive queue via the uplink network. i Simultaneously, the data is directly input into the local auxiliary classification head network to obtain local prediction outputs. , The forward propagation formula is:

[0042]

[0043] in, Indicates Front-end device network with parameters Indicates An auxiliary classification head network with parameters.

[0044] S2.2, Perform local joint backpropagation, including:

[0045] After obtaining the local prediction output, the client then calculates the local auxiliary loss based on this local prediction output; the client uses the local prediction output... Calculate the local loss L with the real label y local,i The expression is:

[0046]

[0047] in, This represents the loss function.

[0048] Subsequently, the client independently executes the backpropagation of the auxiliary classification head and the front-end device network sequentially, completely eliminating the reliance on server gradient backpropagation. The backpropagation and parameter update formulas for the auxiliary classification head network are as follows:

[0049]

[0050] in, and η represents the auxiliary classification header parameters of the i-th client in round t+1 and round t, respectively. local This represents the learning rate during local training. Indicates the local loss with respect to parameter φ i The calculated gradient;

[0051] The backpropagation of the front-end device network, through the gradient returned by the auxiliary layer, is further differentiated, and its update formula is:

[0052]

[0053] in, and Let represent the front-end device network parameters of the i-th client in round t+1 and round t, respectively, and let η represent the learning rate of the front-end network. This represents the local loss with respect to parameter θ. d,i The calculated gradient.

[0054] S3 performs asynchronous calculation of activation feature drift and updates the maximum staleness window, including:

[0055] S3.1, Perform local feature drift calculation, including:

[0056] In round r, when the server receives the new round of activation data uploaded by the client... At that time, before storing it in the queue and overwriting the cache, the server background process asynchronously retrieves historical cache activation data from the same source in the cache metadata table. Canonical correlation analysis was used to calculate the similarity between the two samples, and the average activation drift rate per round was derived. :

[0057]

[0058] In the formula, This indicates canonical correlation analysis. This is used to quantify the degree of drift between currently active data and historical cached active data, where r represents the current communication round. cache This indicates the communication round when the sample was last cached, rr cache The exponent indicates the time interval between two cached cycles. Used to normalize the degree of drift based on the difference in rounds.

[0059] S3.2, perform global drift summaries and window adjustments, including:

[0060] The federated learning system aggregates the drift values ​​of all samples captured in the current round to derive the global activation drift value for each round. Based on this, the maximum allowed obsolescence window threshold w for the current round is dynamically adjusted and output. r To define a safe cache read range, including the global activation drift per round. The specific calculation formula is as follows:

[0061]

[0062] In the above formula, M r This represents the total number of samples participating in the evaluation in the current r-th round. To accurately perform a global average at the sample level, this total number of samples, M, is... r The calculation formula is:

[0063]

[0064] in, Let r be the set of client devices participating in the training round. For client devices The batch set of transmission, For the sample set within the corresponding batch, This indicates the specific number of samples in the sample set.

[0065] Furthermore, the server adaptively updates the maximum staleness window threshold w based on the evolution status of the globally activated data. r Its calculation formula is

[0066]

[0067] In the formula, △ represents the difference in number of rounds between the current round and the cached round (i.e., staleness); δ r This represents the preset drift tolerance for the r-th round, used to limit the maximum allowable deviation of cached data in spatial representation. This represents the global activation drift in round t, and the product term. This formula is used to evaluate the similarity retention probability of activated features from historical cache time to the current time; it dynamically establishes the effective lifespan of cached data at this stage by finding the maximum Δ value that satisfies the drift tolerance limit.

[0068] S4, perform server-side dual-branch independent training based on hybrid retrieval, including:

[0069] S4.1, Constructing a training branch based on a real-time activation queue, including:

[0070] As active clients continue to upload data, the server's data loader retrieves newly arriving activation data from the receive queue. The server uses this activation data Synchronously update and overwrite old records in the cached metadata table; subsequently, the k-th server network performs forward propagation to obtain the output. And calculate server-side losses. , where y s,k This represents the prediction result of the k-th network pin on the server side. Indicated by θ s,k Server network as parameters Let represent the loss function used to measure the difference between the predicted output and the true label. The backpropagation and parameter update formulas for the server network are as follows:

[0071]

[0072] In the formula, and Let η represent the server network parameters for round t+1 and round t, respectively. s This represents the training learning rate on the server side. Indicates the server-side loss relative to the parameter The calculated gradient;

[0073] During this process, since the client has already completed the closed-loop update by the auxiliary layer, the server directly truncates the gradient after backpropagating to the cutting layer, without having to send the gradient back to the client through the downlink.

[0074] S4.2, Construct a fallback training branch based on activation cache, including:

[0075] If no new activation data is detected in the receiving queue and a timeout is triggered, the data loader will switch to the cache metadata table to select historical cached activation data. The cache management module requires that the timestamp round difference △ of the cached data must meet the following requirements:

[0076]

[0077] After verification, the k-th server network uses this cached data to perform forward propagation, expressed as:

[0078]

[0079] The obtained output is then used to perform loss calculations on the server side.

[0080]

[0081] And perform the corresponding backpropagation update:

[0082]

[0083] In the formula, This indicates the predicted output result obtained by the server network based on historical cached activation data. Indicated by θ s,k Server network functions with parameters This represents the server-side simulation loss calculated based on the predicted output. This represents the loss function used to measure the difference between the predicted output and the true label. and These represent the server network parameters for round t+1 and round t, respectively. This indicates that the parameter θ is based on the simulated loss. s,k The calculated gradient.

[0084] S5 performs global federated aggregation based on data volume weighting, including:

[0085] S5.1 performs client parameter and data volume upload, including:

[0086] At the end of the t-th round of training, all clients participating in this round will update their front-end device network parameters. Auxiliary classification head network parameters φ t+1 and the amount of local training data D processed by each. i Upload them together to the cloud parameter server.

[0087] S5.2 performs weighted model aggregation, including: the parameter server calculates the total data volume D:

[0088]

[0089] Based on the proportion of each client's data volume to the total data volume, adaptive weighted FedAvg aggregation is performed on both the front-end device network and the auxiliary classification head network. The aggregation formula for the front-end device network is as follows:

[0090]

[0091] The aggregation formula for the auxiliary classification head network is:

[0092]

[0093] In the formula, and φ t+1 These represent the next round of global front-end device network parameters and global auxiliary classification head network parameters after aggregation, respectively. and Let D represent the front-end device network parameters and auxiliary classification head network parameters updated locally by the i-th client, respectively. N represents the total number of clients participating in the aggregation. i Let represent the amount of local training data owned by the i-th client, and D represent the total amount of local training data of all clients participating in the aggregation.

[0094] S6, conduct multi-server integrated inference tests of the global model, including:

[0095] S6.1, Perform Model Boosting forward feature propagation, including:

[0096] In the test inference phase, a cascaded forward propagation structure is used; the input test sample x test To the already aggregated global front-end device model In, where θ dFor the global front-end device model parameters, perform forward propagation to extract common feature representations, generate and output global intermediate activation data z:

[0097]

[0098] The global intermediate activation data z is then passed as input to a parallel network of K independent backend servers. , where θ s,k The model parameters for the k-th backend server network are obtained by forward propagation from each of the K independent backend server networks, yielding their respective independent prediction outputs y. k :

[0099]

[0100] S6.2 performs Model Bagging, aggregating logical values ​​from multiple backends, such as... Figure 2 As shown, it includes:

[0101] The server performs a Bagging aggregation operation on the prediction outputs of the K parallel back-end sub-models, eliminates the heterogeneous variance of individual models by calculating the arithmetic mean, and generates and outputs the final high-precision prediction result. Its formula is:

[0102]

[0103] Example: To fully demonstrate the advancement of the present invention, this example provides typical operating effects and performance verification in a real heterogeneous edge cloud system.

[0104] A cloud server configured with four high-end NVIDIA Tesla GPUs was deployed to simulate an environment connecting hundreds of heterogeneous edge devices. Widely used image classification tasks were selected, and the VGG-16 network was used with the CIFAR-10 dataset, as well as the MobileNet network with the CIFAR-100 dataset, for testing and validation.

[0105] In actual system operation tests, the traditional baseline split federated learning system can achieve a throughput of 3,000 activation samples per second in the first 30 seconds of each training round. However, due to the sharp decrease in active devices, the throughput plummeted to 250 samples per second. The average utilization of the four GPUs dropped sharply from 60% to 6%, and the memory utilization hovered below 20% for a long time.

[0106] After applying the scheme of this invention, the hybrid activation retrieval mechanism continuously provides cached data during off-peak periods, and the average inbound throughput of the system remains at a high level of 3000 samples per second. Multiple GPUs execute parallel, communication-free ensemble training, and the GPU memory is fully filled with a large amount of activation data. Comprehensive statistics show that the average throughput of the entire communication round of this invention reaches 5.2 times that of the traditional baseline system (and also consistently achieves a throughput improvement of 2.8 to 3.5 times on more complex tasks such as CINIC-10 and CIFAR-100). Due to the exponential growth of data samples consumed per unit time, the time-to-accuracy required for the system to reach the target accuracy is reduced by up to 4.7 times compared to the traditional system. Meanwhile, thanks to the drift-aware quality control and the logical integration mechanism during inference, the model not only does not experience performance degradation due to the use of historical cached data, but its final test accuracy is actually improved by 4.97 percentage points compared to the traditional optimal solution.

[0107] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A high-throughput partitioned federated learning method based on activation caching, characterized in that, include: S1 performs system initialization and defines network structure variables, including: S1.1, Perform model segmentation and initial model distribution; S1.2, perform activation cache library initialization; S2 performs client parameter synchronization, local feature extraction, and independent parameter updates, including: S2.1, perform global model reception and forward propagation with auxiliary layers; S2.2, perform local joint backpropagation; S3 performs asynchronous calculation of activation feature drift and updates the maximum staleness window, including: S3.1, perform local feature drift calculation; S3.2, perform global drift summaries and window adjustments; S4, perform server-side dual-branch independent training based on hybrid retrieval, including: S4.1, Construct a training branch based on a real-time activation queue; S4.2, Construct a fallback training branch based on activation cache; S5 performs global federated aggregation based on data volume weighting, including: S5.1, upload client parameters and data volume; S5.2, perform weighted model aggregation; S6, conduct multi-server integrated inference tests of the global model, including: S6.1, perform Model Boosting forward feature propagation; S6.2 performs Model Bagging, aggregating logical values ​​from multiple backends.

2. The high-throughput split federated learning method based on activation caching according to claim 1, characterized in that, S1.1 includes: The complete global deep neural network is segmented into preset layers, and the server initializes the global front-end device network parameters. With global auxiliary classification head network parameters φ 0 And before the start of round 0 communication, and φ 0 Distribute to all participating clients; for the i-th participating client, the network of the front-end device it receives and deploys is defined as... θ d,i For the network parameters of the front-end device network, θ d,i This represents the parameters of the model at the edge device with index i; simultaneously, the local auxiliary classification head network accessed after layering is defined as... φ i The network parameters are defined for the local auxiliary classification head network; the server is equipped with K independent GPUs, and the backend server network deployed on the k-th GPU is defined as follows: θ s,k These are the model parameters for the network of the k-th backend server.

3. The high-throughput split federated learning method based on activation caching according to claim 1, characterized in that, S1.2 includes: The server allocates an activation cache in memory and on disk. The cache records contain device indexes, batch indexes, sample indexes, the cache round number, and an activation data tensor. Represented as: ， Where i represents the device index participating in training, b represents the data batch index of the device, s represents the sample index within that batch, and r cache This indicates the communication round in which the activation data is cached. For the rth cache During the round, device i generates an activation data tensor from the b-th batch and the s-th sample; simultaneously, the server maintains a lightweight cache metadata table, CacheTable, in memory. This cache metadata table only stores basic index information and the physical address of the actual activation data on disk; the formula for this cache metadata table is defined as: ， Where N, B, and S correspond to the total number of devices participating in training, the number of data batches per device, and the number of samples in each batch, respectively, and addr is the actual storage address of the activation data tensor on the server disk.

4. The high-throughput split federated learning method based on activation caching according to claim 1, characterized in that, S2.1 includes: At the start of the t-th communication round, the selected i-th active client first receives the latest global front-end network parameters from the server after the aggregation in the previous round. With global auxiliary classification head parameter φ t This is used to override the local model parameters, i.e., to assign values. as well as Subsequently, the client sends the local input sample x into the front-end device network for forward propagation, generating intermediate activation data a. i The client will activate intermediate data a i The activation data (a) is asynchronously sent to the server's receive queue via the uplink network. i Simultaneously, the data is directly input into the local auxiliary classification head network to obtain local prediction outputs. , The forward propagation formula is: ， in, Indicates Front-end device network with parameters Indicates An auxiliary classification head network with parameters.

5. The high-throughput split federated learning method based on activation caching according to claim 1, characterized in that, S2.2 includes: The client utilizes local prediction output Calculate the local loss L with the real label y local,i : ， in, This represents the loss function used to measure the difference between the local predicted output and the true label; The client independently executes the backpropagation of the auxiliary classification head and the front-end device network in sequence; the backpropagation of the front-end device network continues to derive through the gradients returned by the auxiliary layer.

6. The high-throughput split federated learning method based on activation caching according to claim 1, characterized in that, S3.1 includes: In the r-th communication round, when the server receives the new round of activation data uploaded by the client... At that time, before storing it in the queue and overwriting the cache, the server background process asynchronously retrieves historical cache activation data from the same source in the cache metadata table. Canonical correlation analysis was used to calculate the similarity between the two samples, and the average activation drift per round was derived. : ， In the formula, This represents the canonical correlation function. Used to quantify the degree of drift between currently active data and historical cached active data, r cache This indicates the communication round when the sample was last cached, rr cache This indicates the time interval between two cached cycles.

7. A high-throughput split federated learning method based on activation caching according to claim 6, characterized in that, S3.2 includes: By summing the drift amounts of all samples in the current round, the global activation drift for each round can be calculated. , The calculation formula is: ， In the formula, M r This represents the total number of samples participating in the evaluation in the current r-th round, calculated using the following formula: ， in, Let r be the set of clients participating in the training round. For the batch set transmitted by client i, For the sample set within the corresponding batch, This indicates the number of samples in the sample set; Server update maximum staleness window threshold w r The calculation formula is: ， In the formula, △ represents the difference in rounds between the current round and the cache round; δ r This represents the preset drift tolerance for round r. This represents the global activation drift in round t. These are terms in a multiplication process.

8. A high-throughput split federated learning method based on activation caching according to claim 7, characterized in that, S4.1 includes: When new activation data arrives in the receive queue At that time; the server utilized Update the old record in the cached metadata table; subsequently, the k-th server performs a forward propagation to obtain the output. And calculate server-side losses. , where y s,k This represents the k-th network prediction result on the server side. Indicated by θ s,k For the server network with parameters, θ s,k Let the model parameters of the k-th backend server network be denoted as . The backpropagation and parameter update formulas for the server network are as follows: ， In the formula, and Let η represent the server network parameters for round t+1 and round t, respectively. s This represents the training learning rate on the server side. Indicates the server-side loss relative to the parameter The calculated gradient.

9. A high-throughput splitting federated learning method based on activation caching according to claim 8, characterized in that, S4.2 includes: If no new activation data is detected in the receiving queue and a timeout is triggered, the data loader will switch to the cache metadata table to select historical cached activation data. The cache management module requires that the round difference △ of the cached data must meet the following requirements: ， Where r represents the current communication round, r cache This indicates the communication round when historical activation data is cached; After verification, the k-th server network uses this cached data to perform forward propagation, expressed as: ， The obtained output is then used to perform loss calculations on the server side. ， And perform the corresponding backpropagation update: ， In the formula, This indicates the predicted output result obtained by the server network based on historical cached activation data. Indicated by θ s,k Server network functions with parameters This represents the server-side simulation loss calculated based on the predicted output. This represents the loss function used to measure the difference between the predicted output and the true label. and Let η represent the server network parameters for round t+1 and round t, respectively. s This represents the training learning rate on the server side. This indicates that the parameter θ is based on the simulated loss. s,k The calculated gradient.

10. A high-throughput split federated learning method based on activation caching according to claim 1, characterized in that, S5.1 includes: At the end of the t-th round of training, all clients participating in this round will update their front-end device network parameters. Auxiliary classification head network parameters φ t+1 and the amount of local training data D processed by each. i Upload them together to the cloud parameter server; S5.2 includes: Parameter server calculates the total data volume D: ， Based on the proportion of each client's data volume to the total data volume, adaptive weighted FedAvg aggregation is performed on both the front-end device network and the auxiliary classification head network. The aggregation formula for the front-end device network is as follows: ， The aggregation formula for the auxiliary classification head network is: ， In the formula, and φ t+1 These represent the next round of global front-end device network parameters and global auxiliary classification head network parameters after aggregation, respectively. and Let D represent the front-end device network parameters and auxiliary classification head network parameters updated locally by the i-th client, respectively. N represents the total number of clients participating in the aggregation. i Let represent the amount of local training data owned by the i-th client, and D represent the total amount of local training data of all clients participating in the aggregation.

11. The high-throughput split federated learning method based on activation caching according to claim 1, characterized in that, S6.1 includes: employing a cascaded forward propagation structure during the test inference phase; inputting test sample x test To the already aggregated global front-end device model In, where θ d For the global front-end device model parameters, perform forward propagation to extract common feature representations, generate and output global intermediate activation data z: ， The global intermediate activation data z is then passed as input to a parallel network of K independent backend servers. , where θ s,k The model parameters are for the k-th backend server network; each of the K independent backend server networks performs forward propagation to obtain its own independent prediction output y. k : ， S6.2 includes: The server performs a Bagging aggregation operation on the prediction outputs of K parallel back-end sub-models, eliminates the heterogeneous variance of individual models by calculating the arithmetic mean, and generates and outputs the final high-precision prediction result. Its formula is: 。

Images