Parallel split learning method, device and system based on adaptive gradient correction

Abstract

This invention provides a parallel split learning method, apparatus, and system based on adaptive gradient correction, relating to the field of distributed machine learning computing technology. The invention receives intermediate layer gradients uploaded by each client, divides them into deep and shallow gradient components, calculates the consistency and comprehensive ranking score of each client's gradient, and generates an ordered gradient queue. Then, gradients are introduced one by one for weighted aggregation, directional stability judgment, and noise truncation to update the aggregation center. The comprehensive ranking score of high-confidence gradients is selected from the ordered gradient queue, and the global confidence score is calculated to dynamically adjust the global model parameters. A global reference direction is output and sent to each client, enabling each client to correct its gradients according to the global reference direction. The residual after gradient correction is used as a regularization constraint for the next round of local training optimization. This invention effectively suppresses gradient conflicts, improves model convergence speed and robustness, and reduces server computational overhead.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning and edge intelligent computing technology, and more specifically, to a parallel split learning method, apparatus and system based on adaptive gradient correction. Background Technology

[0002] With the rapid development of IoT devices and edge computing, massive amounts of data are continuously generated on the terminal side, driving an urgent need for privacy protection and efficient collaborative training mechanisms. Against this backdrop, Federated Learning (FL) and Split Learning (SL) have become mainstream distributed machine learning paradigms, enabling collaborative model training without uploading the original data. Compared to Federated Learning, Split Learning, by logically dividing deep neural networks between the client and server and transmitting only intermediate layer activation values ​​or gradients, significantly reduces the requirements for client computing resources, memory bandwidth, and energy consumption, making it more suitable for resource-constrained edge devices. However, traditional Split Learning uses a serial training method, where the client interacts with the server sequentially to complete forward propagation and backward updates, resulting in a long overall training cycle and low communication efficiency. To improve training throughput, Parallel Split Learning (PSL) was proposed, allowing multiple clients to interact with the server simultaneously, significantly shortening the training time per round.

[0003] Despite the significant efficiency advantages of parallel decomposition learning, it still faces serious challenges in practical deployment. Due to the diverse environments and varying data collection scenarios of edge devices, client-side local data often exhibits highly non-independent and identically distributed (Non-IID) characteristics. As the number of participating devices increases and data distributions differ significantly, intermediate layer gradients uploaded by different clients are prone to severe directional conflicts, seriously interfering with the convergence process of the global model. Further complicating matters, in real-world systems, extreme outlier gradients generated by device failures, network anomalies, or malicious attacks may also be introduced, further exacerbating the risk of model divergence.

[0004] To address such issues, existing technologies mainly rely on two types of strategies: one is the mean-based aggregation method, such as the classic FedAvg, which updates the model by weighting the gradients or parameters of the client; the other is the robust defense mechanism based on distance metrics, such as the Krum algorithm, which selects the candidate gradient with the smallest sum of squared Euclidean distances to other gradients as the update direction, or truncated averaging, which calculates the mean after removing the largest and smallest values ​​in each dimension, in order to suppress the influence of anomalies.

[0005] However, the above methods have significant limitations in complex heterogeneous environments. First, simple weighted averaging is extremely sensitive to outliers and cannot effectively resist poisoning attacks or sudden noise. Second, defense strategies based on hard thresholds (such as Krum or truncated averaging) often require preset parameters that are difficult to adjust adaptively, easily leading to the accidental deletion of long-tail data containing useful information or gradients of lagging nodes, resulting in decreased data utilization. More importantly, existing aggregation schemes are mostly "open-loop" designs, lacking a dynamic feedback mechanism between the server and client, and cannot adaptively adjust based on gradient consistency during training, network latency, or model layer sensitivity. This makes it difficult for the system to balance robustness, convergence speed, and computational efficiency under extreme Non-IID and asynchronous communication conditions.

[0006] In view of the above, this application is hereby submitted. Summary of the Invention

[0007] This invention aims to provide a parallel split learning method, device, and system based on adaptive gradient correction, in order to solve the technical problems of existing parallel split learning in complex edge computing environments such as highly non-independent and identically distributed data, asynchronous communication, and potential malicious attacks, including severe gradient conflicts, slow model convergence, insufficient robustness, and excessive server computing overhead.

[0008] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0009] A parallel split learning method based on adaptive gradient correction, applied to the server side, includes:

[0010] S1 receives intermediate layer gradients uploaded by each client;

[0011] S2, After dividing the intermediate layer gradient into deep and shallow gradient components, calculate the consistency and comprehensive sorting score of each client gradient to generate an ordered gradient queue.

[0012] S3, based on the ordered gradient queue, gradients are introduced one by one for weighted aggregation, directional stability judgment and noise truncation, so as to update the aggregation center;

[0013] S4. Based on the updated aggregation center, select the comprehensive ranking score of high-confidence gradients from the ordered gradient queue, calculate the global confidence to dynamically adjust the global model parameters, output the global reference direction and send it to each client so that each client can correct the gradient according to the global reference direction, and use the residual after gradient correction for the regularization constraint of the next round of local training optimization.

[0014] Preferably, the generation process of the ordered gradient queue is as follows:

[0015] First, based on the neural network hierarchy, the gradient of the intermediate layer is divided into shallow gradient components and deep gradient components, expressed as follows:

[0016] ;

[0017] in, For the first The intermediate layer gradient uploaded by each client; For the first Shallow gradient components of each client; For the first Deep gradient components for each client;

[0018] Next, using the global reference direction from the previous round as the time-series anchor point, we calculate cross-round consistency and same-round consistency for the deep gradient components of each client.

[0019] Wherein, the cross-round consistency is the cosine similarity between the current gradient and the deep component of the previous round's global reference direction, expressed as:

[0020] ;

[0021] in, For the first Consistency across all clients; This is the deep component of the previous round of global reference direction; for and Cosine similarity; To prevent the minimum value where the denominator is 0; This refers to the vector dot product operation. It is an L2 norm;

[0022] The consensus within the same round is the average cosine similarity between the current gradient and the deep gradient components of other clients in the same round, expressed as:

[0023] ;

[0024] in, For the first Consistency across all clients in the same round; This represents the total number of clients that participated in uploading in this round. For the first Deep gradient components for each client; for and Cosine similarity;

[0025] Then set the cross-round consistency threshold. Consistency threshold with the same round The deep gradient components are filtered based on the calculated cosine similarity and the preset cross-cycle consistency threshold and same-cycle consistency threshold:

[0026] like or If the current deep gradient component is determined to be a suspicious gradient, its weight will be reduced in the sorting process.

[0027] Based on the selected deep gradient components, and considering the consistency between shallow and deep gradients, the comprehensive ranking score of the current gradient is calculated. The expression is:

[0028] ;

[0029] in, The comprehensive ranking score of the i-th client gradient; , These are the weight coefficients for shallow general features and deep semantic features, respectively; This refers to the shallow component of the previous round's global reference direction; The weighting coefficient for consistency within the same round;

[0030] Based on the calculated comprehensive ranking score All client gradients are sorted from high to low to generate an ordered gradient queue.

[0031] Preferably, the update process of the aggregation center is as follows:

[0032] First, from the ordered gradient queue Gradients are introduced one by one, and their aggregate weights are calculated by incorporating time delay factors; then the k-th gradient is derived from... Gradients introduced in the queue Its aggregate weight The expression is:

[0033] ;

[0034] in, This is the time delay attenuation coefficient; This is the aggregation center after the (k-1)th gradient is introduced; To avoid extremely small numbers with a denominator of 0; gradient Arrival lag time; It is an exponential function; It is an L2 norm;

[0035] Next, based on the calculated aggregation weights, the improved Weiszfeld algorithm is used to iteratively update the aggregation centers, as shown in the formula:

[0036] ;

[0037] in, The aggregation center is introduced after the k-th gradient; For the rth time from Gradients introduced in the queue Aggregate weights;

[0038] Then, based on the updated aggregation centers, the rate of change of direction between two adjacent aggregation centers is calculated to determine stability. The formula is as follows:

[0039] ;

[0040] in, This represents the rate of change of the polymerization center direction; the smaller the value, the more stable the polymerization direction.

[0041] like Then continuous stable counting ; The directional stability threshold;

[0042] like If the count is reset to 0, then the count is reset to 0. ;

[0043] like If the aggregation direction is stable, the early stopping mechanism is triggered to cut off noise, that is: terminate the subsequent gradient introduction of aggregation and discard the gradients that have not been introduced into the queue.

[0044] in, The threshold for the number of times early stop is determined;

[0045] like Then, the next gradient is introduced, and so on, until all gradients have been traversed. All gradients in the queue that trigger early stopping mechanisms will result in an updated aggregation center.

[0046] Preferably, the calculation process of the global reference direction is as follows:

[0047] First from the ordered gradient queue The combined ranking scores of the top K high-confidence gradients are selected, and the global confidence is calculated by averaging the scores. The formula is as follows:

[0048] ;

[0049] in, Let be the global confidence level in round t; For the ordered gradient queue The overall ranking score of the k-th gradient;

[0050] Next, a mapping relationship between confidence and learning rate is established based on the global confidence level to adaptively adjust the global learning rate on the server side. The formula is as follows:

[0051] ;

[0052] in, The server-side global adaptive learning rate for round t; Use the Sigmoid activation function; , These are the upper and lower bounds of the learning rate, respectively. The sensitivity parameter is adjusted based on the learning rate; The confidence center point;

[0053] Then, the global model parameters on the server are updated based on the global learning rate, using the following formula:

[0054] ;

[0055] in, , These are the global model parameters for rounds t+1 and t, respectively; Let be the aggregation gradient of round t;

[0056] Finally, let's analyze the gradient aggregation in this round. The global reference direction is obtained by performing L2 normalization, as shown in the formula:

[0057] ;

[0058] in, This is the global reference direction for round t; To avoid extremely small numbers with a denominator of 0; It is an L2 norm.

[0059] Preferably, each client corrects the gradient according to the global reference direction, specifically as follows:

[0060] The client receives the global reference direction and global model parameters sent by the server;

[0061] Calculate the inner product of the client's local gradient and the global reference direction to determine whether the local gradient conflicts with the global direction. The formula is as follows:

[0062] ;

[0063] in, Let be the inner product of the local gradient of client i in round t and the global reference direction; Let be the client-local gradient in round t; This is the global reference direction for round t;

[0064] like This indicates that the local gradient update direction conflicts with the global reference direction, and a gradient correction is performed to ensure that the corrected gradient is consistent with the global direction.

[0065] When correcting the gradient, a projection operation is used to remove the inverse component, as shown in the formula:

[0066] ;

[0067] in, The client-side local gradient after correction in round t;

[0068] like This indicates that the directions are consistent and no correction is needed. ;

[0069] The client uploads the corrected gradients to the server for aggregation.

[0070] Preferably, the residual after gradient correction is used as a regularization constraint for the next round of local training optimization, specifically:

[0071] The client calculates the corrected residual vector and residual strength for this round of gradient using the following formula:

[0072] ;

[0073] ;

[0074] in, Let i be the corrected residual vector for client i in round t; Let the residual strength be the value of client i in round t. It is an L2 norm;

[0075] Based on the residual strength Calculate the dynamic regularization coefficient for the next training round, so that the dynamic regularization term is added to the task loss during training. The formula is:

[0076] ;

[0077] ;

[0078] in, Let be the dynamic regularization coefficient for client i; Basic regularization coefficient; This is the residual regularization strength mapping coefficient, used to control the degree of influence of the residual on regularization; Let be the total loss function with regularization constraints for client i in round t+1; Let i be the task loss function for client i in round t+1; These are the local model parameters for client i; These are the global model parameters issued by the server in round t.

[0079] This invention also provides a parallel split learning device based on adaptive gradient correction, applied on a server side, comprising:

[0080] The gradient receiving module is used to receive intermediate layer gradients uploaded by each client.

[0081] The multi-dimensional perception and hierarchical sorting module is used to divide the intermediate layer gradient into deep and shallow gradient components, calculate the consistency and comprehensive sorting score of each client gradient, and generate an ordered gradient queue.

[0082] The time-delay-aware progressive aggregation module is used to introduce gradients one by one into the ordered gradient queue for weighted aggregation, directional stability judgment and noise truncation, so as to update the aggregation center.

[0083] The global adaptive optimization module is used to select the comprehensive ranking score of high-confidence gradients from the ordered gradient queue based on the updated aggregation center, calculate the global confidence to dynamically adjust the global model parameters, output the global reference direction and send it to each client so that each client can correct the gradient according to the global reference direction, and use the residual after gradient correction for the regularization constraint of the next round of local training optimization.

[0084] The present invention also provides a parallel split learning system based on adaptive gradient correction, comprising multiple clients and a central server;

[0085] The client is used to perform forward propagation based on local data, generate intermediate activation values ​​and backpropagate to calculate intermediate layer gradients, and then upload the intermediate layer gradients to the central server.

[0086] The client corrects the gradient based on the global reference direction issued by the central server, and uses the residual after gradient correction as a regularization constraint for the next round of local training optimization.

[0087] The central server includes a processor and a memory. The memory stores a computer program that can be executed by the processor to implement a parallel split learning method based on adaptive gradient correction as described above.

[0088] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium resides, implement the parallel splitting learning method based on adaptive gradient correction as described above.

[0089] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0090] This invention introduces dual constraints of temporal anchor points and spatial consistency to automatically reduce the aggregation ratio of high-latency gradients. At the same time, it combines a directional stability early stopping mechanism to truncate noise gradients at the tail of the queue, effectively identifying and suppressing outlier gradients caused by data heterogeneity, equipment failure, or Byzantine attacks. This avoids the traditional hard truncation method from mistakenly removing lagging node data containing valid information, significantly reducing server computational burden and memory usage, and improving aggregation efficiency in edge asynchronous scenarios.

[0091] After layering the gradients, this invention uses a dual consistency screening method of same-round and cross-round to prioritize retaining gradients that are consistent with the global direction and the direction of most clients in the current round, while filtering out suspicious gradients with serious semantic deviations. This reduces the probability of gradient conflicts from the source and improves the consistency of feature expression of the global model.

[0092] This invention establishes a dynamic mapping relationship between aggregate confidence and learning rate. By sensing gradient quality through global confidence, the learning rate is dynamically adjusted, enabling the model to accelerate convergence when gradient consistency is strong and decelerate and stabilize when conflicts are severe, thereby improving the adaptability of training dynamics. No manual parameter tuning is required, and it is suitable for complex parallel split learning scenarios.

[0093] This invention guides the client to perform half-space projection correction gradient by issuing a normalized global reference direction, and uses residual feedback to dynamically adjust the local regularization strength, so that conflicting clients actively move towards the global direction in subsequent training, prompting deviating clients to continuously return to the globally consistent subspace, forming a virtuous training loop.

[0094] This invention achieves adaptive perception and adjustment from four dimensions: hierarchical, temporal, network, and optimization. Without the need for preset attack ratios or fixed thresholds, it balances robustness, data utilization, and computational efficiency, enabling parallel split learning to achieve convergence performance close to that of centralized training in extreme Non-IID and asynchronous edge scenarios. At the same time, it maintains low communication overhead and strong privacy protection, providing an engineering-deployable technical path for distributed collaborative learning in resource-constrained environments such as the Internet of Things and smart terminals. Attached Figure Description

[0095] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0096] Figure 1This is a flowchart illustrating a parallel split learning method based on adaptive gradient correction, as provided in Example 1.

[0097] Figure 2 This is a schematic diagram of the framework of a parallel split learning method based on adaptive gradient correction provided in Example 1.

[0098] Figure 3 This is a schematic diagram of a parallel split learning device based on adaptive gradient correction provided in Embodiment 2.

[0099] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0101] Example 1

[0102] Embodiment 1 of the present invention provides a parallel split learning method based on adaptive gradient correction, which can be implemented by a parallel split learning device based on adaptive gradient correction (hereinafter referred to as parallel split learning device), specifically, executed by one or more processors within the parallel split learning device.

[0103] In this embodiment, the parallel split learning device can be an electronic device equipped with a processor, which carries a computer program for the parallel split learning method based on adaptive gradient correction and the computer program can be executed, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0104] like Figure 1 As shown, a parallel split learning method based on adaptive gradient correction includes steps S1 to S4.

[0105] S1 receives intermediate layer gradients uploaded by each client.

[0106] like Figure 2As shown, this invention proposes a robust adaptive gradient optimization framework for parallel split learning. This framework consists of four core modules: a multidimensional perceptual hierarchical ranking module (MPS), a latency-aware progressive aggregation module (LPA), a global adaptive optimization module (GAO), and a client-side half-space projection and residual feedback module (HPF). These modules form a closed-loop collaborative mechanism of "server guidance—client correction," achieving high robustness, low computational overhead, and fast convergence in model training under highly non-independent and identically distributed (Non-IID) and asynchronous network conditions.

[0107] Specifically, the server first receives intermediate layer gradients uploaded by multiple clients and records the timestamp of each gradient to calculate the arrival lag delay of each gradient as the basis for subsequent latency-aware aggregation.

[0108] The purpose of this step is to obtain gradient information from the client's local training, providing a data foundation for subsequent aggregation, and to collect gradient latency data to provide a basis for the gradient weight decay of "lagging" gradients.

[0109] S2, after dividing the intermediate layer gradient into deep and shallow gradient components, calculate the consistency and comprehensive sorting score of each client gradient to generate an ordered gradient queue.

[0110] This step distinguishes between the general features of gradients (shallow layer) and the semantic features of the task (deep layer), focuses on core semantic consistency, filters out abnormal gradients with serious directional deviations through dual consistency screening, and prioritizes the retention of high-confidence gradients by robustly sorting gradient quality from "high confidence" to "low confidence", thus building the first barrier against malicious attacks and outlier interference.

[0111] Specifically, firstly, based on the neural network hierarchy, the intermediate layer gradients are divided into shallow gradient components and deep gradient components according to the feature level, as expressed by:

[0112] ;

[0113] in, For the first The intermediate layer gradient uploaded by each client; For the first Shallow gradient components of each client; For the first Deep gradient components for each client;

[0114] Next, using the global reference direction from the previous round as the time-series anchor point, we calculate cross-round consistency and same-round consistency for the deep gradient components of each client.

[0115] Wherein, the cross-cycle consistency (temporal continuity) is the cosine similarity between the current gradient and the deep component of the previous round's global reference direction, expressed as:

[0116] ;

[0117] in, For the first Consistency across all clients; This is the deep component of the previous round of global reference direction; for and Cosine similarity; To prevent the minimum value where the denominator is 0; This refers to the vector dot product operation. It is an L2 norm;

[0118] The in-round consistency (spatial consistency) is the average cosine similarity between the current gradient and the deep gradient components of other clients in the same round, expressed as:

[0119] ;

[0120] in, For the first Consistency across all clients in the same round; This represents the total number of clients that participated in uploading in this round. For the first Deep gradient components for each client; for and The cosine similarity.

[0121] Then set the cross-round consistency threshold. Consistency threshold with the same round The deep gradient components are filtered based on the calculated cosine similarity and the preset cross-cycle consistency threshold and same-cycle consistency threshold:

[0122] like or If the current deep gradient component is determined to be a suspicious gradient, its weight will be reduced in the sorting process.

[0123] Based on the selected deep gradient components, and considering the consistency between shallow and deep gradients, the comprehensive ranking score of the current gradient is calculated, expressed as:

[0124] ;

[0125] in, The comprehensive ranking score of the i-th client gradient; , These are the weight coefficients for shallow general features and deep semantic features, respectively; This refers to the shallow component of the previous round's global reference direction; The weighting coefficient for consistency within the same round;

[0126] Based on the calculated comprehensive ranking score All client gradients are sorted from high to low to generate an ordered gradient queue.

[0127] This step employs shallow gradients to ensure consistency of general features and deep gradients to enhance semantic alignment, thereby improving the targeting of gradient selection. In-round consistency constrains the gradient space distribution, while cross-round consistency maintains training temporal continuity, effectively eliminating outlier gradients. The generated ordered queue ensures that subsequent aggregation prioritizes high-quality gradients, laying the foundation for robust aggregation.

[0128] This invention addresses the problem that different levels have varying sensitivities to gradient conflicts, and that it is difficult to suppress sudden changes in gradient direction (or a sharp drop in consistency) in a single round during PSL training due to factors such as data heterogeneity, parallel asynchrony, and split point coupling.

[0129] S3, based on the ordered gradient queue, gradients are introduced one by one for weighted aggregation, directional stability judgment and noise truncation, so as to update the aggregation center.

[0130] To address the challenges of limited server computing resources and reduced aggregation efficiency due to asynchronous latency introduced by "stragglers" in edge computing scenarios, this step employs an improved Weiszfeld algorithm to introduce gradients one by one based on the sorting results. By reducing the aggregation weight of high-latency gradients, the convergence jitter caused by "stragglers" is mitigated. Progressive aggregation is used instead of full aggregation to improve the robustness of the aggregation center to outlier disturbances. Tail noise gradients are truncated through stability checks to reduce ineffective computational overhead and improve aggregation accuracy.

[0131] First, from the ordered gradient queue Gradients are introduced one by one, and their aggregate weights are calculated by incorporating time delay factors; then the k-th gradient is derived from... Gradients introduced in the queue Its aggregate weight The expression is:

[0132] ;

[0133] in, This is the time delay attenuation coefficient; This is the aggregation center after the (k-1)th gradient is introduced; To avoid extremely small numbers with a denominator of 0; gradient Arrival lag time; It is an exponential function; It is an L2 norm;

[0134] Next, based on the calculated aggregation weights, the improved Weiszfeld algorithm is used to iteratively update the aggregation centers, as shown in the formula:

[0135] ;

[0136] in, The aggregation center is introduced after the k-th gradient; For the rth time from Gradients introduced in the queue Aggregate weights.

[0137] In this embodiment, the Weiszfeld algorithm is an iterative optimization algorithm for solving the weighted geometric median. Its core is to find a point by gradually approximating it, such that the sum of the weighted distances from that point to all sample points is minimized.

[0138] This invention does not use the traditional "full gradient unified iteration" aggregation method. Instead, it introduces gradients one by one according to the sorted order of the output ordered gradient queue and performs improved Weiszfeld progressive aggregation to obtain an aggregation center that is more robust to outlier perturbations.

[0139] Then, based on the updated aggregation centers, the rate of change of direction between two adjacent aggregation centers is calculated to determine stability. The formula is as follows:

[0140] ;

[0141] in, This represents the rate of change of the polymerization center direction; the smaller the value, the more stable the polymerization direction.

[0142] like Then continuous stable counting ; The directional stability threshold;

[0143] like If the count is reset to 0, then the count is reset to 0. ;

[0144] like If the aggregation direction is stable, the early stopping mechanism is triggered to cut off noise, that is: terminate the subsequent gradient introduction of aggregation and discard the gradients that have not been introduced into the queue.

[0145] in, The threshold for the number of times early stop is determined;

[0146] like Then, the next gradient is introduced, and so on, until all gradients have been traversed. All gradients in the queue that trigger early stopping mechanisms will result in an updated aggregation center.

[0147] This early stopping mechanism, while ensuring aggregation robustness, significantly reduces the number of gradients and computational overhead involved in the iteration on the server side, thereby improving the overall training efficiency in edge asynchronous scenarios.

[0148] This step uses a latency factor to automatically "reduce the weight" of high-latency gradients, preventing outdated gradients from dragging the global direction; progressive aggregation gradually approaches the optimal aggregation center, which is more resistant to interference than full aggregation; the early stopping mechanism terminates immediately after the aggregation direction stabilizes, significantly reducing server computation and improving training efficiency in edge scenarios.

[0149] S4. Based on the updated aggregation center, select the comprehensive ranking score of high-confidence gradients from the ordered gradient queue, calculate the global confidence to dynamically adjust the global model parameters, output the global reference direction and send it to each client so that each client can correct the gradient according to the global reference direction, and use the residual after gradient correction for the regularization constraint of the next round of local training optimization.

[0150] To avoid the slow convergence or oscillation issues that traditional methods using a fixed learning rate often encounter in non-IID or asynchronous scenarios, this step establishes a dynamic mapping of "gradient quality - learning rate" to balance convergence speed and stability. An optimized aggregated gradient is used to update the global model, automatically accelerating updates when gradient directions are highly consistent and slowing down to stabilize when conflicts are severe, achieving an adaptive balance between convergence speed and stability and ensuring the correct direction of model iteration. Global consistency information is fed back to the client, guiding it to correct gradients through closed projection, eliminating backward conflict components, and recording the residual magnitude generated by the correction. This residual magnitude is dynamically converted into a regularization term for the next round of local training, forcing the local model to actively converge towards the globally consistent subspace in subsequent training, achieving a leap from "instantaneous gradient correction" to "long-term model alignment."

[0151] Specifically, the calculation process for the global reference direction is as follows:

[0152] First from the ordered gradient queue The combined ranking scores of the top K high-confidence gradients are selected, and the global confidence is calculated by averaging the scores. The formula is as follows:

[0153] ;

[0154] in, Let be the global confidence level in round t; For the ordered gradient queue The overall ranking score of the k-th gradient;

[0155] Next, a mapping relationship between confidence and learning rate is established based on the global confidence level to adaptively adjust the global learning rate on the server side. The formula is as follows:

[0156] ;

[0157] in, The server-side global adaptive learning rate for round t; Use the Sigmoid activation function; , These are the upper and lower bounds of the learning rate, respectively. The sensitivity parameter is adjusted based on the learning rate; The confidence center point is [value].

[0158] In this embodiment, the Sigmoid activation function can be selected as: .

[0159] Then, the global model parameters on the server are updated based on the global learning rate, using the following formula:

[0160] ;

[0161] in, , These are the global model parameters for rounds t+1 and t, respectively; Let be the aggregation gradient of round t;

[0162] Finally, let's analyze the gradient aggregation in this round. The global reference direction is obtained by performing L2 normalization, as shown in the formula:

[0163] ;

[0164] in, This is the global reference direction for round t; To avoid extremely small numbers with a denominator of 0; It is an L2 norm.

[0165] The normalized global reference direction is distributed to each client for subsequent direction constraints / conflict mitigation, thus forming a closed-loop optimization process guided by cross-round consistency.

[0166] In order to effectively feed back the global consistency information from the server to the client and continuously suppress gradient conflicts in subsequent training rounds, each client corrects the gradient and cross-round constraints according to the global reference direction, so as to correct the conflict update in the current round and reduce the probability of the conflict occurring again in subsequent rounds.

[0167] Specifically, the client receives the global reference direction and global model parameters sent by the server;

[0168] Calculate the inner product of the client's local gradient and the global reference direction to determine whether the local gradient conflicts with the global direction. The formula is as follows:

[0169] ;

[0170] in, Let be the inner product of the local gradient of client i in round t and the global reference direction; Let be the client-local gradient in round t; This is the global reference direction for round t;

[0171] like This indicates that the local gradient update direction conflicts with the global reference direction, and a gradient correction is performed to ensure that the corrected gradient is consistent with the global direction.

[0172] When correcting the gradient, a projection operation is used to remove the inverse component, as shown in the formula:

[0173] ;

[0174] in, The client-side local gradient after correction in round t;

[0175] like This indicates that the directions are consistent and no correction is needed. ;

[0176] The client uploads the corrected gradients to the server for aggregation.

[0177] To ensure the client "remembers" the correction direction in subsequent training rounds, the residuals after gradient correction are used as regularization constraints for the next round of local training optimization, specifically:

[0178] The client calculates the corrected residual vector and residual strength for this round of gradient using the following formula:

[0179] ;

[0180] ;

[0181] in, Let i be the corrected residual vector for client i in round t; Let the residual strength be the value of client i in round t. It is an L2 norm;

[0182] Based on the residual strength Calculate the dynamic regularization coefficient for the next training round, so that the dynamic regularization term is added to the task loss during training. The formula is:

[0183] ;

[0184] ;

[0185] in, Let be the dynamic regularization coefficient for client i; Basic regularization coefficient; This is the residual regularization strength mapping coefficient, used to control the degree of influence of the residual on regularization; Let be the total loss function with regularization constraints for client i in round t+1; Let i be the task loss function for client i in round t+1; These are the local model parameters for client i; These are the global model parameters issued by the server in round t.

[0186] The larger the residual, the more severe the conflict in this round, and the stronger the constraint on "approaching the global parameter space" in the next round.

[0187] This invention addresses the poor adaptability of a fixed learning rate by increasing the learning rate when gradients are consistent to accelerate convergence and decreasing the learning rate when gradient conflicts occur to suppress oscillations. Then, it updates the model based on high-confidence aggregated gradients, improving the generalization ability of the global model. The issued reference direction supports the client with "real-time gradient correction + cross-round residual regularization," forming a server-client closed-loop optimization that reduces the probability of gradient conflicts in subsequent rounds from the source.

[0188] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0189] Existing parallel split learning methods often employ full gradient aggregation, which can slow down the overall training pace due to gradients from high-latency clients and easily introduce directional perturbations. This invention uses a latency-aware weight design to automatically reduce the aggregation ratio of high-latency gradients. Simultaneously, it incorporates an early stopping mechanism for directional stability to truncate noisy gradients at the tail of the queue, avoiding invalid computation and significantly improving aggregation efficiency in asynchronous edge scenarios. The progressive aggregation method also makes the aggregation center more robust to outlier gradients, reducing convergence jitter.

[0190] Existing methods typically aggregate raw gradients directly without distinguishing between shallow general features and deep semantic features, which can easily lead to gradient direction conflicts in scenarios with non-independent and identically distributed IID data. This invention, after layering gradients, uses a dual consistency screening method—both within and across rounds—to prioritize retaining gradients consistent with the global direction and the direction of most clients in the current round, while filtering out suspicious gradients with severe semantic deviations. This reduces the probability of gradient conflicts from the source and improves the consistency of feature representation in the global model.

[0191] Existing methods often employ a fixed learning rate, which leads to slow convergence in the early stages of training when gradient consistency is high, and oscillations are prone to occur in the later stages when gradient conflicts intensify. This invention uses global confidence to perceive gradient quality and dynamically adjusts the learning rate: increasing the learning rate accelerates convergence when gradient consistency is high, and decreasing the learning rate stabilizes training when gradient conflicts occur. This eliminates the need for manual parameter tuning and is suitable for complex parallel decomposition learning scenarios.

[0192] Existing methods often involve one-way parameter distribution between the server and client, failing to feed back global consistency information to the client for local training. This invention, by distributing a normalized global reference direction, guides the client to perform half-space projection to correct gradients. It also utilizes residual feedback to dynamically adjust the local regularization strength, causing conflicting clients to actively gravitate towards the global direction during subsequent training. This forms a closed loop of "server-based filtering and aggregation – client-based correction and optimization," continuously suppressing gradient conflicts and improving model generalization ability.

[0193] This invention only adds gradient processing and aggregation modules to the server side, without modifying the client model structure. It can be directly adapted to mainstream split learning networks such as U-Net and Transformer, with low engineering implementation costs and broad application prospects.

[0194] Example 2

[0195] like Figure 3 As shown, the second embodiment of the present invention also provides a parallel split learning device based on adaptive gradient correction, comprising:

[0196] The gradient receiving module is used to receive intermediate layer gradients uploaded by each client.

[0197] The multi-dimensional perception and hierarchical sorting module is used to divide the intermediate layer gradient into deep and shallow gradient components, calculate the consistency and comprehensive sorting score of each client gradient, and generate an ordered gradient queue.

[0198] The time-delay-aware progressive aggregation module is used to introduce gradients one by one into the ordered gradient queue for weighted aggregation, directional stability judgment and noise truncation, so as to update the aggregation center.

[0199] The global adaptive optimization module is used to select the comprehensive ranking score of high-confidence gradients from the ordered gradient queue based on the updated aggregation center, calculate the global confidence to dynamically adjust the global model parameters, output the global reference direction and send it to each client so that each client can correct the gradient according to the global reference direction, and use the residual after gradient correction for the regularization constraint of the next round of local training optimization.

[0200] Example 3

[0201] The third embodiment of the present invention also provides a parallel split learning device based on adaptive gradient correction, which includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the parallel split learning method based on adaptive gradient correction as described above.

[0202] Example 4

[0203] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, they implement the parallel split learning method based on adaptive gradient correction as described above.

[0204] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0205] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0206] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0207] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0208] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0209] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0210] The use of "first" and "second" in the embodiments is merely to distinguish similar objects and does not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0211] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A parallel split learning method based on adaptive gradient correction, applied to the server side, characterized in that, include: Receive intermediate layer gradients uploaded by each client; Based on the hierarchical structure of the neural network, the gradient of the intermediate layer is divided into shallow gradient components and deep gradient components; Using the global reference direction of the previous round as the time-series anchor point, calculate cross-round consistency and same-round consistency for the deep gradient components of each client; wherein, the cross-round consistency is the cosine similarity between the current gradient and the deep components of the global reference direction of the previous round; the same-round consistency is the average cosine similarity between the current gradient and the deep gradient components of other clients in this round; Then, cross-wheel consistency thresholds and same-wheel consistency thresholds are set, and deep gradient components are filtered based on the calculated cosine similarity and the preset cross-wheel consistency thresholds and same-wheel consistency thresholds. Based on the deep gradient components that have passed the screening, and combined with the consistency between shallow and deep layers, the comprehensive ranking score of the current gradient is calculated. Based on the calculated comprehensive ranking score, all client gradients are sorted from high to low to generate an ordered gradient queue. Gradients are introduced one by one from the ordered gradient queue, and their aggregate weights are calculated. The aggregation center is iteratively updated based on the calculated aggregation weights; Based on the updated aggregation centers after weighted aggregation, the rate of change of direction between two adjacent aggregation centers is calculated to determine stability; When it is determined that the aggregation direction has stabilized, the early stopping mechanism is triggered to cut off noise, that is: terminate the introduction of subsequent gradients for aggregation and discard the gradients that have not been introduced into the queue; otherwise, continue to introduce the next gradient until all gradients in the ordered gradient queue have been traversed or the early stopping mechanism is triggered to obtain the stable updated aggregation center. Based on the stable updated aggregation center, the comprehensive ranking scores of the top K gradients are selected from the ordered gradient queue, and the global confidence is calculated by the mean. A mapping relationship between confidence and learning rate is established based on the global confidence level to adaptively adjust the global learning rate on the server side. Update the global model parameters on the server side based on the global learning rate; The aggregated gradient of this round is L2 normalized to obtain the global reference direction, and the global reference direction is output and sent to each client. The client receives the global reference direction and global model parameters sent by the server; Calculate the inner product of the client's local gradient and the global reference direction to determine whether the local gradient conflicts with the global reference direction; When the local gradient update direction conflicts with the global reference direction, each client corrects the gradient according to the global reference direction to ensure that the corrected gradient is consistent with the global reference direction. The client uploads the corrected gradients to the server for aggregation; The client calculates the corrected residual vector and residual strength of the gradient in this round; the residual strength is the L2 norm of the corrected residual vector; Based on the residual strength, the dynamic regularization coefficient for the next round of training is calculated so that the dynamic regularization term is added to the task loss during training.

2. The parallel split learning method based on adaptive gradient correction according to claim 1, characterized in that... The intermediate layer gradient is divided into shallow gradient components and deep gradient components based on the neural network hierarchy, and its expression is as follows: ； in, For the first The intermediate layer gradient uploaded by each client; For the first Shallow gradient components of each client; For the first Deep gradient components for each client; The expression for cross-round consistency is: ； in, For the first Consistency across all clients; This is the deep component of the previous round of global reference direction; for and Cosine similarity; To prevent the minimum value where the denominator is 0; This refers to the vector dot product operation. It is an L2 norm; The expression for the same-round consistency is: ； in, For the first Consistency across all clients in the same round; This represents the total number of clients that participated in uploading in this round. For the first Deep gradient components for each client; for and Cosine similarity; like or If the current deep gradient component is deemed a suspicious gradient, it will be demoted in the ranking; where, The preset cross-wheel consistency threshold; This is the consistency threshold for the same round; The overall ranking score of the current gradient The expression is: ； in, The comprehensive ranking score of the i-th client gradient; , These are the weight coefficients for shallow general features and deep semantic features, respectively; This refers to the shallow component of the previous round's global reference direction; This is the weighting coefficient for consistency within the same round.

3. The parallel split learning method based on adaptive gradient correction according to claim 1, characterized in that... When calculating aggregate weights, from the ordered gradient queue Gradients are introduced one by one, and their aggregate weights are calculated by incorporating time delay factors; then the k-th gradient is derived from... Gradients introduced in the queue Its aggregate weight The expression is: ； in, This is the time delay attenuation coefficient; The aggregation center after the (k-1)th gradient is introduced; To avoid extremely small numbers with a denominator of 0; gradient Arrival lag time; It is an exponential function; It is an L2 norm; When iteratively updating the aggregation center based on the calculated aggregation weights, an improved Weiszfeld algorithm is used, with the following formula: ； in, The aggregation center is introduced after the k-th gradient; For the rth time from Gradients introduced in the queue Aggregate weights; The formula for calculating the rate of change of direction of two adjacent aggregation centers is: ； in, This represents the rate of change of the polymerization center direction; the smaller the value, the more stable the polymerization direction. like Then continuous stable counting ;in, The directional stability threshold; like If the count is reset to 0, then the count is reset to 0. ; like If the polymerization direction is stable, then the polymerization direction is considered to be stable; where, The threshold for the number of times early stops are determined; like If the gradient is not found, the next gradient is introduced until all gradients in the ordered gradient queue have been traversed or the early stopping mechanism is triggered, resulting in a stable updated aggregation center.

4. The parallel split learning method based on adaptive gradient correction according to claim 2, characterized in that... The formula for the global confidence level is: ； in, Let be the global confidence level in round t; For the ordered gradient queue The Middle The combined ranking score of each gradient; The formula for establishing a mapping relationship between confidence and learning rate based on the global confidence level, and adaptively adjusting the global learning rate on the server side, is as follows: ； in, The server-side global adaptive learning rate for round t; Use the Sigmoid activation function; , These are the upper and lower bounds of the learning rate, respectively. The sensitivity parameter is adjusted based on the learning rate; The confidence center point; The formula for updating the global model parameters on the server side based on the global learning rate is: ； in, , These are the global model parameters for rounds t+1 and t, respectively; Let be the aggregation gradient of round t; The formula for the global reference direction is: ； in, This is the global reference direction for round t; To avoid extremely small numbers with a denominator of 0; It is an L2 norm.

5. The parallel split learning method based on adaptive gradient correction according to claim 4, characterized in that... The formula for calculating the inner product of the client's local gradient and the global reference direction to determine whether the local gradient conflicts with the global reference direction is as follows: ； in, Let be the inner product of the local gradient of client i in round t and the global reference direction; Let be the client-local gradient in round t; This is the global reference direction for round t; like This indicates that the local gradient update direction conflicts with the global reference direction, and a gradient correction is performed to ensure that the corrected gradient is consistent with the global reference direction. When correcting the gradient, a projection operation is used to remove the inverse component, as shown in the formula: ； in, The client-side local gradient after correction in round t; like This indicates that the directions are consistent and no correction is needed. .

6. The parallel split learning method based on adaptive gradient correction according to claim 5, characterized in that... The formulas for the corrected residual vector and the residual strength are as follows: ； ； in, Let i be the corrected residual vector for client i in round t; Let the residual strength be the value of client i in round t. It is an L2 norm; Based on the residual strength Calculate the dynamic regularization coefficient for the next training round, so that the dynamic regularization term is added to the task loss during training. The formula is as follows: ； ； in, Let be the dynamic regularization coefficient for client i; Basic regularization coefficient; This is the residual regularization strength mapping coefficient, used to control the degree of influence of the residual on regularization; Let be the total loss function with regularization constraints for client i in round t+1; Let i be the task loss function for client i in round t+1; These are the local model parameters for client i; These are the global model parameters issued by the server in round t.

7. A parallel split learning device based on adaptive gradient correction, applied on a server side, to implement the parallel split learning method based on adaptive gradient correction as described in any one of claims 1-6, characterized in that, include: The gradient receiving module is used to receive intermediate layer gradients uploaded by each client. The multi-dimensional perception and hierarchical sorting module is used to divide the intermediate layer gradient into deep and shallow gradient components, calculate the consistency and comprehensive sorting score of each client gradient, and generate an ordered gradient queue. The time-delay-aware progressive aggregation module is used to introduce gradients one by one into the ordered gradient queue for weighted aggregation, directional stability judgment and noise truncation, so as to update the aggregation center. The global adaptive optimization module is used to select the comprehensive ranking score of the top K gradients from the ordered gradient queue based on the updated aggregation center, calculate the global confidence to dynamically adjust the global model parameters, output the global reference direction and send it to each client so that each client can correct the gradient according to the global reference direction, and use the residual after gradient correction for the regularization constraint of the next round of local training optimization.

8. A parallel split learning system based on adaptive gradient correction, characterized in that, It includes multiple clients and a central server; The client is used to perform forward propagation based on local data, generate intermediate activation values ​​and backpropagate to calculate intermediate layer gradients, and then upload the intermediate layer gradients to the central server. The client corrects the gradient based on the global reference direction issued by the central server, and uses the residual after gradient correction as a regularization constraint for the next round of local training optimization. The central server includes a processor and a memory, the memory storing a computer program that can be executed by the processor to implement a parallel split learning method based on adaptive gradient correction as described in any one of claims 1-6.

Images