FPGA-based video memory decoupling type sparse gradient aggregation and error compensation method and system

Abstract

The application discloses a method and system for sparse gradient aggregation and error compensation based on FPGA, wherein a host end GPU maintains the momentum state of an optimizer in a local video memory, calculates a local weight update amount and unloads it to an intelligent network card; the network card end reads the error residual of the last round, fuses the current update amount after joint compensation to correct the weight update amount; after the transmission cycle gate of conflict perception, the dead zone threshold is adjusted dynamically to quantize the remaining error matrix, the small truncation of the remaining value is taken as a new error residual and stored in the on-board memory, and the rest of the value is packed as a sparse data stream; after the line speed encryption, the transmission between multiple nodes is carried out and the in-network aggregation is carried out, and finally the global ciphertext update amount is returned to the host end, and a new key is dynamically derived according to the global aggregation result. The application releases the GPU video memory and computing power, reduces the communication bandwidth demand, and realizes low-delay safe transmission and aggregation while ensuring the convergence accuracy of the model.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning, neuromorphic computing and network communication technology, specifically involving a method and system for FPGA-based decoupled sparse gradient aggregation and error compensation based on video memory. Background Technology

[0002] Spiking neural networks (SNNs), as the core carrier of neuromorphic computing, can efficiently simulate the spiking and synaptic transmission characteristics of biological neurons, replicating the brain's advantages in parallel processing and sparse coding. As neuromorphic computing develops towards large-scale and high-precision applications, the number of SNN parameters and computational load have surged, making single nodes insufficient for the efficient training requirements of large-scale SNNs. Distributed training has become a key technological path to overcome the performance bottleneck of single nodes and achieve large-scale deployment of SNNs. By splitting models and tasks to achieve parallel collaborative training, it overcomes the limitations of single nodes, significantly improving training efficiency and is crucial for the engineering implementation of neuromorphic computing. However, in the actual deployment of distributed training for large-scale models and spiking neural networks, the network communication bandwidth between nodes becomes a significant bottleneck for system expansion.

[0003] To address this challenge, existing communication bandwidth optimization techniques often employ high compression ratios to quantize gradients. However, to ensure model convergence, an error compensation mechanism must be introduced. Yet, existing error compensation mechanisms require the GPU to maintain a massive residual state matrix in its memory, resulting in significant memory resource consumption. Existing intra-network computation or gradient compression algorithms typically compress the original gradient directly. This leads to the disruption of historical state update logic when applying nonlinear optimizers like Adam, which require maintaining historical momentum states, resulting in abnormal learning rate calculations and model divergence. During the time-wise backpropagation of SNNs, due to the physical characteristics of surrogate gradients, a large number of tiny free-state gradients are generated in the network. Existing standard communication libraries cannot efficiently remove these invalid tiny values ​​at the underlying hardware level, leading to significant communication bandwidth waste. In multi-tenant or cross-datacenter environments, existing intra-network computation solutions typically only process plaintext data. However, plaintext transmission carries the risk of data leakage. If encryption is performed at the host end, intermediate network devices cannot perform arithmetic aggregation on the ciphertext.

[0004] In summary, existing communication bandwidth optimization techniques still have many significant shortcomings in large-scale distributed training scenarios for spiking neural networks, specifically: the error compensation mechanism leads to excessive memory resource consumption, poor compatibility with nonlinear optimizers such as Adam, difficulty in effectively filtering out communication redundancy caused by high sparse gradients during SNN training, and inability to ensure data security while improving computational efficiency. Summary of the Invention

[0005] To address the shortcomings of existing technologies and achieve efficient decoupling of computing and communication resources, this invention adopts the following technical solution:

[0006] A memory-decoupled sparse gradient aggregation and error compensation method based on FPGA includes the following steps:

[0007] Step 1: The host GPU maintains the Adam optimizer momentum state in the node's local video memory and uses a continuously differentiable surrogate gradient function to calculate the local gradient of the spiking neural network (SNN). The obtained local weight update and second-order momentum state parameters are then unloaded to the node's FPGA smart network card.

[0008] Step 2: The network card reads the error residual from the previous round stored in the onboard memory. After joint error compensation through magnitude matching scaling, direction-aware modulation, time-varying compensation, and dual-branch forgetting modulation, it is fused with the current weight update amount to obtain the corrected update amount.

[0009] Step 3: Perform conflict-aware transmission cycle gating on the correction update amount, perform quantization filtering based on the dynamically adjusted dead zone threshold, accumulate small updates below the dead zone threshold into new error residuals and store them in the onboard memory, and package updates above the threshold into an indexed sparse data stream after combining node aggregation weights.

[0010] Step 4: After the sparse data stream is encrypted at line speed using a key, it is transmitted between multiple nodes and globally aggregated within the network. During the aggregation transmission process, the dead zone threshold is adjusted by statistically analyzing the bandwidth utilization. After the aggregation is completed, the global ciphertext update is returned to the host, and a new key is dynamically derived based on the global aggregation result.

[0011] Furthermore, in step 1, the host GPU maintains the first-order momentum state and the second-order momentum state of the Adam optimizer in the node's local video memory.

[0012] The error compensation in step 2 includes magnitude matching scaling. A scaling factor is calculated based on the second-order momentum of the optimizer from the previous and current rounds, aligning the amplitude range of the onboard residuals with the current weight update. The scaling factor calculation formula is as follows:

[0013]

[0014] in, This represents the second-order momentum of the previous optimizer. This represents the second-order momentum of the current round optimizer. , These are the upper and lower limits of the scaling factor. This represents the clamping function, used to suppress extreme values ​​caused by gradient sparsity during training, preventing update explosions or vanishing. The scaling factor is multiplied by the onboard truncation residual error from the previous round to obtain the current onboard truncation residual error after magnitude matching. ,in This refers to the residual error from the previous truncation round stored in the onboard memory.

[0015] Furthermore, the error compensation in step 2 also includes direction-aware modulation, which calculates the direction consistency coefficient between the residual and the current update amount based on the weight update amount and the truncation residual error of the previous round:

[0016]

[0017] in, This indicates the amount of weight updates issued by the GPU in the current round. This indicates the truncation residual error from the previous round. This represents the preset directional smoothing parameter, used to adjust the smoothness of the directional consistency coefficient;

[0018] Then, by applying soft limiting to the directional consistency coefficient using the hyperbolic tangent function, the directional sensing coefficient is obtained:

[0019]

[0020] in, This represents the sensitivity coefficient, used to control the intensity of the response to orientation perception. When the value is large, the function curve is steeper, the coefficients are more sensitive to changes in direction, and the influence of consistency / adversarial residuals can be amplified rapidly. When the value is small, the curve is flatter and the response to changes in direction is more gentle, avoiding drastic changes in residual processing due to single gradient fluctuations.

[0021] The direction-aware coefficient is combined with the direction modulation intensity, which decays exponentially with the number of iterations, to generate a direction enhancement factor. This allows the high modulation intensity in the early stages of training to accelerate gradient direction stabilization, while the low modulation intensity in the later stages of training avoids oscillations during the update process.

[0022]

[0023]

[0024] in, It serves as a directional enhancement factor, enabling adaptive enhancement or suppression of the residual;

[0025] The residuals are modulated using the aforementioned direction enhancement factor to enhance in-direction residuals and suppress in-direction residuals, thereby improving the effectiveness of residual compensation and obtaining the updated truncated residual error:

[0026] .

[0027] Furthermore, the error compensation in step 2 also includes time-varying compensation, which dynamically adjusts the compensation intensity according to the training stage and adopts segmented compensation to adapt to the residual confidence level at different stages.

[0028] Very early training The model is in a state of random initialization, with drastic fluctuations in gradient direction and highly random weight updates. At this point, the truncated residuals generated by fixed-point computation contain almost no effective quantization error information, and the confidence level of the residuals is extremely low. If high-intensity compensation is directly introduced, the random noise will be amplified into a pseudo-update signal, which can easily cause the model parameters to deviate from the effective optimization path in the initial stage, or even diverge directly. Therefore, low-intensity tentative compensation with linear growth is adopted.

[0029]

[0030] in, This indicates the upper limit of the maximum compensation intensity for this stage. Indicates the end round of the very early stage;

[0031] Early training As the model gradient directionality gradually increases, the proportion of effectively quantized errors in the truncated residuals calculated at fixed points begins to rise, and the confidence level of the residuals gradually increases. At this point, an insufficient compensation intensity cannot effectively utilize the residual information, leading to the continuous accumulation of quantization errors and slowing down the convergence speed. Therefore, a linearly accelerated compensation intensity is adopted:

[0032] =

[0033] in, This indicates the increase in compensation intensity during this phase. Indicates the end of the early training phase;

[0034] Mid-training As the model gradient direction stabilizes, the truncated residuals from fixed-point computation can accurately reflect the systematic loss caused by low-precision quantization, and the confidence level of the residuals reaches its peak. At this point, using high-intensity stable compensation can ensure that the residuals in each round are fully utilized, achieving stable and efficient residual compensation. Therefore, a constant peak compensation intensity is adopted:

[0035]

[0036] Late training As the model approaches the optimal solution, the magnitude of weight updates decreases significantly, and the optimization process enters the fine-tuning stage. At this point, the absolute value of the residuals also decreases. If high-intensity compensation is maintained, small residual fluctuations may be amplified into large weight oscillations, causing the model to fluctuate around the optimal solution and even disrupting the converged parameter state. To avoid oscillations caused by overcompensation while retaining basic error correction capabilities, ensuring model stability during the convergence phase, and preventing accuracy regression due to improper compensation, an exponentially decaying compensation intensity is adopted.

[0037]

[0038] in, This indicates the lower limit of the compensation intensity. Represents the decay time constant. This indicates the end of the peak compensation phase.

[0039] Compensation intensity through dynamic adjustment The updated truncation residual error is modulated to obtain the time-varying compensated truncation residual error:

[0040] .

[0041] Furthermore, the error compensation in step 2 also includes joint error compensation for dual-branch forgetting modulation and the basic forgetting factor. The forgetting intensity decreases smoothly with each iteration, gradually transitioning from a low forgetting intensity at the beginning to a high forgetting intensity in the later stages. This solves both the problem of insufficient residual correction in the early stages and the problem of bias drift caused by residual accumulation in the later stages, achieving adaptive changes in forgetting intensity as the training process progresses.

[0042]

[0043] in, A higher value indicates a lower level of forgetting, while a lower value indicates a higher level of forgetting. Represents the maximum forgetting factor. Represents the minimum forgetting factor. This represents the forgetting time constant, used to control the forgetting rate from... Increment to The speed at which the residual forgetting mechanism responds determines its response rhythm, which can be dynamically adjusted according to the model type and training task.

[0044] A two-branch forgetting structure is adopted, and a rapid forgetting factor is set. To adapt to short-term fluctuations, slow forgetting factor To maintain long-term stability, use the fusion coefficient. Achieving a dynamic weighted fusion of the two yields the final forgetting factor. ; , Can be set separately , , Parameters;

[0045]

[0046] Among them, the rapid forgetting factor Its core function is to quickly respond to short-term gradient fluctuations. When the gradient is active and the parameter distribution changes abruptly, it can quickly increase the forgetting rate and avoid historical residuals interfering with new optimization directions; slow forgetting factor. The core function is to ensure the long-term stability of training, avoid drastic changes in the forgetting rate due to short-term gradient fluctuations, and ensure the stable retention of long-term effective residual information. This is the fusion coefficient, used to balance the weights of the two branches;

[0047] Based on the forgetting factor, the truncated residual error after time-varying compensation is corrected to obtain the final residual correction value:

[0048]

[0049] After completing the above four-level compensation, the corrected residual is added to the current weight update amount to obtain the corrected update amount:

[0050] .

[0051] Furthermore, the transmission cycle gating for conflict awareness in step 3 is based on the aggregate weights and weight changes of nodes to construct a global conflict coefficient, which characterizes the consistency of different nodes in updating the same weight direction.

[0052]

[0053] in, Indicates the first Wheel, number The global conflict coefficient of the weights indicates that the update directions of each node for that element are more inconsistent, the conflict is more severe, and the proportion that is canceled out in the aggregation is higher. Indicates the first The aggregate weight of each node, Indicates the first Each node, numbered The change in weight;

[0054] The transmission interval of weight elements is dynamically adjusted based on the conflict coefficient. This ensures that the more severe the conflict, the larger the transmission interval, and all nodes consistently reduce the transmission frequency of the weight elements. Conversely, the less severe the conflict, the smaller the transmission interval and the higher the transmission frequency, guaranteeing timely updates of effective gradients.

[0055]

[0056] in, This represents the transmission interval of the previous round, with an initial value of 0. The interval adjustment factor is used. The more severe the collision, the larger the transmission interval. Nodes consistently reduce the transmission frequency to highly conflicting elements to avoid invalid communication.

[0057] Furthermore, in step 3, the dead zone threshold is dynamically adjusted based on the deviation between the target bandwidth utilization and the actual bandwidth utilization, so that when the actual bandwidth utilization is higher than the target bandwidth utilization, the dead zone threshold is increased to enhance sparsity and release bandwidth margin; conversely, the threshold is decreased to retain more effective information.

[0058]

[0059] in, Indicates the target bandwidth utilization. Indicates actual bandwidth utilization. This represents the bandwidth adjustment coefficient, which controls the intensity of the impact of bandwidth utilization on the threshold.

[0060] Combining the transmission interval and dynamic dead zone threshold, different types of weight update amounts are processed differently, and weight updates are divided into delayed transmission, error storage and sparse transmission, in order to maximize the compression of communication data volume while ensuring the transmission of effective gradients, thereby achieving refined communication optimization.

[0061] For transmission interval The weights are not transmitted in this round. High-conflict updates are temporarily stored locally through delayed transmission to avoid invalid communication. Simultaneously, these updates are accumulated in an error reservoir and sent all at once in subsequent transmissions, reducing the number of communications. Residual Updated to .

[0062] For transmission interval The weights are used to perform dead zone quantization and determination. Based on the relationship between the update magnitude and the dynamic dead zone threshold, two cases are distinguished:

[0063] If the change in weight is less than the dynamic dead zone threshold, then... If the update magnitude is below the dead zone threshold, it has little impact on model optimization. Accumulating this data in the error reservoir and then merging and transmitting it later avoids frequently sending low-effectiveness data. Residual Updated to ;

[0064] If the weight change is greater than or equal to the dynamic dead zone threshold, i.e. The weight change is multiplied by the node aggregation weight, and the resulting data is labeled with a sparse index. The sparse encoding and packaging module packages the data into a sparse data stream using a [address + change] structure and outputs it to the line-rate encryption / decryption subsystem. Simultaneously, the residual is set to 0. When the magnitude of such updates exceeds the dead zone threshold, it has a significant impact on model optimization. Sparse encoding and transmission are directly performed to ensure timely updates of effective gradients, while zeroing the residuals avoids repeated accumulation.

[0065] Further, in step 4, a unified initial symmetric key is obtained from the host during initialization. The hardware encryption module performs line-rate encryption on the sparse data stream and transmits it among multiple nodes according to the ring topology, and performs intra-network aggregation. After each round of aggregation, all nodes generate the next round session key through the hardware key derivation function based on the same global aggregation result. Finally, each node transmits the global weight update amount back to the host. The ring topology aggregation is performed on N nodes, with each node performing N-1 rounds of transmission. In the k-th round of transmission, each node encrypts the (ID+1-k) mod N fragment and sends it to the downstream device. The transmitted data includes the device ID of the node and the sparse data stream. Under the ring aggregation topology, each node locally calculates the weighted sum of the absolute values ​​of the weight changes to calculate the global conflict coefficient for each weight.

[0066] An FPGA-based memory-decoupled sparse gradient aggregation and error compensation system includes a host interaction and global control subsystem, an onboard memory management subsystem, a residual fusion and error compensation subsystem, a dead-zone quantization and sparse coding subsystem, a line-speed encryption / decryption subsystem, and a distributed sparse gradient aggregation operation subsystem. Gradient aggregation and error compensation are performed according to the FPGA-based memory-decoupled sparse gradient aggregation and error compensation method.

[0067] The host interaction and global control subsystem receives the local weight update and Adam optimizer second-order momentum state parameters sent by the host GPU, parses the control commands and schedules other subsystems to work. After the intra-network aggregation is completed, the global weight update is sent back to the host according to the protocol packet.

[0068] The onboard memory management subsystem performs unified storage and read / write management of various types of data during operation;

[0069] The residual fusion and error compensation subsystem reads the previous round of error residuals from the onboard memory and performs multi-level corrections before fusing them with the current weight update amount to obtain the corrected update amount, thereby solving the problem of incompatibility between residuals and GPU-side update amounts under the decoupled memory architecture.

[0070] The dead zone quantization and sparse coding subsystem performs collision-aware gating and dynamic dead zone quantization on the corrected update amount, generates a sparse data stream and updates the error residual matrix to reduce communication bandwidth requirements.

[0071] The line-speed encryption / decryption subsystem uses a symmetric cryptographic algorithm for line-speed encryption / decryption and dynamic key derivation.

[0072] The distributed sparse gradient aggregation subsystem performs intra-network aggregation on the decrypted sparse data stream, and simultaneously completes the distributed statistics and global distribution of gradient conflict coefficients.

[0073] Furthermore, the residual fusion and error compensation subsystem includes a magnitude matching scaling module, a direction-aware modulation module, a time-varying compensation module, a dual-branch forgetting modulation module, and a residual fusion module. After performing joint error compensation on the previous round of error residuals in sequence, the corrected error residuals are added to the current weight update amount to obtain the corrected weight update amount.

[0074] The dead zone quantization and sparse coding subsystem includes a transmission interval gating module, a dynamic dead zone quantization module, and a sparse coding packing module, which are respectively used to perform conflict-aware transmission cycle gating, dynamic dead zone threshold adjustment, weight update amount to perform dead zone quantization and update truncation residual error, and pack update amount higher than the dead zone threshold into sparse format.

[0075] The line-speed encryption / decryption subsystem includes a line-speed encryption / decryption module, a high-speed serial transmission module, and a key derivation module, which are used to perform line-speed encryption on sparse indices and quantized values, transmit and receive data between boards, and automatically generate the next round session key based on a unified global aggregation result in each round.

[0076] The distributed sparse gradient aggregation subsystem includes an aggregation control module and a sparse index alignment module, which are used to configure relevant parameters such as node weighting coefficients, and to perform index alignment and aggregation operations between the local sparse index set and the upstream aggregation results.

[0077] The advantages and beneficial effects of this invention are as follows:

[0078] This invention effectively reduces GPU memory consumption, achieves efficient synergy between high gradient compression ratio and error compensation, eliminates the need for additional GPU memory to maintain redundant state parameters, significantly reduces GPU memory overhead, ensures model convergence performance while improving hardware resource utilization, breaks through the performance bottleneck of the error compensation stage, and facilitates the efficient advancement of large-scale model training.

[0079] This invention has good adaptability to nonlinear optimizers, and can stably support the normal operation of optimizers such as Adam that require the maintenance of historical momentum states. It ensures the integrity of the optimizer's historical momentum state update logic, ensures accurate learning rate calculation and orderly parameter updates, maintains the convergence stability of model training, and improves the adaptability, versatility and reliability of training complex networks.

[0080] This invention can accurately identify and eliminate tiny free-state invalid gradients generated during the backpropagation of SNN time, significantly reduce the amount of invalid data transmission, significantly save communication bandwidth resources in distributed training, improve the transmission efficiency of multi-node collaborative training, and adapt to the communication needs of SNN high-sparseness training scenarios.

[0081] This invention achieves a balance between computational performance and data security, supports intra-network gradient aggregation operations, and can complete intermediate node arithmetic aggregation operations without host-side decryption. It not only ensures the privacy and security of training data in multi-tenant and cross-data center scenarios and avoids the risk of data leakage, but also retains the collaborative acceleration advantages of intra-network computing, ensuring the efficient advancement of distributed training. Attached Figure Description

[0082] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0083] Figure 2 This is a flowchart of multi-level error compensation and residual fusion, conflict-aware gating and dynamic dead zone sparsification in an embodiment of the present invention.

[0084] Figure 3 This is a flowchart illustrating the global aggregation process and key update of ring aggregation in an embodiment of the present invention.

[0085] Figure 4 This is a schematic diagram of the system structure in an embodiment of the present invention.

[0086] Figure 5 This is a flowchart illustrating the execution process of the system in an embodiment of the present invention. Detailed Implementation

[0087] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0088] To address the technical problems in existing technologies, such as excessive GPU memory consumption for error compensation mechanisms, poor compatibility with nonlinear optimizers like Adam, communication redundancy caused by high sparse gradients in SNN training, and conflicts between plaintext transmission and ciphertext aggregation, this invention proposes a memory-decoupled sparse gradient aggregation and error compensation method based on FPGA. This method employs a hardware-software decoupled computing-network collaborative architecture, offloading residual fusion, dead-zone sparsification, linear encryption, and linear aggregation to an FPGA smart network interface card equipped with onboard memory. The host GPU is only responsible for calculating and distributing weight updates, thus achieving efficient decoupling of computing and communication resources. This method is suitable for large-scale spiking neural networks (SNNs) and large-model distributed training scenarios. Figure 1 As shown, the method of the present invention specifically includes the following steps;

[0089] Step 1: Host-side gradient calculation and parameter unloading;

[0090] The host-side GPU independently maintains the first and second momentum states of the Adam optimizer in its local memory. It uses a continuously differentiable surrogate gradient function to solve for the time backpropagation local gradient of the SNN and calculates the local weight update. and the current second-order momentum Along with the weight update, it is offloaded to the FPGA smart network card via the PCIe bus.

[0091] In this step, the GPU is only responsible for local gradient calculation and optimizer momentum maintenance, and does not participate in any error compensation, sparsification or encryption operations, thus achieving complete decoupling of computation and communication.

[0092] Step 2: Multi-level error compensation and residual fusion at the network card end;

[0093] like Figure 2 As shown, the network card reads the truncation residual error generated by the previous round of dead-zone quantization from the onboard memory. The four-level joint compensation operation is performed sequentially, including magnitude matching scaling, direction-aware modulation, time-varying compensation, and dual-branch forgetting modulation. The final result is fused with the current weight update to obtain the corrected update. The steps of the four-level joint compensation operation are as follows:

[0094] Step 2.1: Scale-matching scaling: Receive the current round's second-order momentum from the GPU. Read the previous second-order momentum from the onboard memory. The scaling factor is calculated to match the magnitude of the computational load, aligning the amplitude range of the onboard residuals with the current weight update amount. This ensures consistency in the magnitude of the residuals and the update amount, adapting to the momentum update logic of the Adam optimizer. The scaling factor is calculated using the following formula:

[0095]

[0096] in, The second-order momentum of the previous round of Adam optimizer issued to the GPU. The second-order momentum of the Adam optimizer in the current round, issued by the GPU. , These are the upper and lower limits of the scaling factor. This is a clamping function used to suppress extreme values ​​caused by gradient sparsity during training, preventing update explosions or vanishing. After calculating the scaling factor, it is multiplied by the onboard residual error to obtain the stage residual error after magnitude matching: ,in This refers to the residual error from the previous truncation round stored in the onboard memory.

[0097] Step 2.2: Direction-Aware Modulation: Used to enhance in-direction residuals and suppress out-of-direction residuals, improving the effectiveness of residual compensation. First, calculate the direction consistency coefficient between the residual and the current update:

[0098]

[0099] in, This represents the weight update amount issued by the GPU in the current round. This is the residual error from the previous round of truncation. This is a preset directional smoothing parameter used to adjust the smoothness of the directional consistency coefficient.

[0100] The direction-aware coefficient is then soft-limited using the hyperbolic tangent function, and its value range is restricted to (-1, 1) to eliminate the risk of extreme values.

[0101]

[0102] in, The sensitivity coefficient is used to control the intensity of the response to orientation perception; when When the value is large, the function curve is steeper, the coefficients are more sensitive to changes in direction, and the effect of consistency / adversarial residuals can be amplified rapidly. When the value is smaller, the curve is flatter and the response to changes in direction is more moderate, avoiding drastic changes in residual processing due to single gradient fluctuations.

[0103] Finally, the final enhancement factor is generated by combining the directional modulation intensity, which decays exponentially with the number of iterations. This achieves an adaptive adjustment where high modulation accelerates gradient direction stability in the early stages of training, while low modulation avoids update oscillations in the later stages.

[0104]

[0105]

[0106] in, It serves as a directional enhancement factor, enabling adaptive enhancement or suppression of the residual.

[0107] The core advantage of this exponential decay design is that, in the early stages of training, near With high modulation intensity and active intervention of the direction-aware mechanism, residuals aligned with the update direction are rapidly amplified, adversarial residuals are suppressed, and gradient direction stabilization is accelerated; in the later stages of training... Gradually decrease to The modulation intensity is reduced, and the direction perception mechanism is gradually weakened to avoid oscillations in the update process due to overmodulation and to ensure the stability of model convergence.

[0108] The updated truncation residual error is obtained by modulating the residual using a direction enhancement factor:

[0109] .

[0110] Step 2.3: Time-varying compensation: The compensation intensity is dynamically adjusted according to the training phase, and a segmented compensation method is used to adapt the residual confidence level at different stages.

[0111] Very early training The model is in a stochastic initialization state, with drastic gradient direction fluctuations and highly random weight updates. At this stage, the truncated residuals generated by fixed-point computation contain almost no effective quantization error information, resulting in extremely low confidence levels. Directly introducing high-intensity compensation would amplify the random noise into pseudo-update signals, easily causing the model parameters to deviate from the effective optimization path in the initial stage, or even diverge entirely. Therefore, in this stage, the truncated residuals are introduced tentatively with extremely low intensity and linear growth.

[0112]

[0113] in, This represents the upper limit of the maximum compensation intensity for this stage; This is the final round of the very early stage.

[0114] Early training As the directionality of the model gradient gradually increases, the proportion of effective quantization error in the truncated residuals calculated at fixed points begins to rise, and the confidence level of the residuals gradually improves. At this point, an insufficient compensation intensity cannot effectively utilize the residual information, leading to the continuous accumulation of quantization error and slowing down the convergence speed. This stage employs a linearly accelerated increase in compensation intensity.

[0115]

[0116] in, The increase in compensation intensity at this stage determines the efficiency of the intensity enhancement. This marks the end of the early stage and the point where the compensation intensity reaches its peak.

[0117] Mid-training Once the model gradient direction stabilizes, the truncated residuals from fixed-point computation can accurately reflect the systematic loss caused by low-precision quantization, and the confidence level of the residuals reaches its peak. At this point, employing high-intensity stable compensation ensures that the residuals in each round are fully utilized, achieving stable and efficient residual compensation. A constant peak compensation intensity is used in this stage.

[0118]

[0119] As training enters its later stages The model is close to the optimal solution, and the magnitude of weight updates has significantly decreased, marking the entry into the fine-tuning stage of the optimization process. At this point, the absolute value of the residuals also decreases. If high-intensity compensation is maintained, small residual fluctuations may be amplified into large weight oscillations, causing the model to fluctuate around the optimal solution and even disrupting the converged parameter state. To avoid oscillations caused by overcompensation while preserving basic error correction capabilities, ensuring model stability during the convergence phase, and preventing accuracy regression due to improper compensation, an exponentially decaying compensation intensity is adopted in this stage.

[0120]

[0121] in, The lower limit of the compensation strength is used to prevent the strength from decaying to 0, which would cause the compensation mechanism to fail. The decay time constant controls the rate at which the intensity decreases; This marks the end of the peak compensation phase and the beginning of exponential decay.

[0122] Dynamically adjusted compensation intensity After modulation, the final time-varying compensated truncation residual error is obtained:

[0123] .

[0124] Step 2.4: Dual-branch forgetting modulation: By introducing a forgetting factor that is dynamically adjusted with each iteration, the effective historical error information is retained while the excessive accumulation effect of early residuals is suppressed. This achieves the dual goals of retaining effective information of residual signals and suppressing long-term bias, providing long-term stable error correction capability for low-precision training.

[0125] Basic forgetting factor The forgetting intensity decreases smoothly with each iteration, gradually transitioning from a low initial forgetting intensity to a high later forgetting intensity. This addresses both the problem of insufficient residual correction in the early stages and the bias drift caused by residual accumulation in the later stages. This achieves adaptive variation of forgetting intensity with the training process.

[0126]

[0127] in, A higher value indicates a lower level of forgetting, while a lower value indicates a higher level of forgetting. The maximum forgetting factor, It is the minimum forgetting factor. The forgetting time constant is used to control the forgetting rate from... Increment to The speed at which the residual forgetting mechanism responds determines its response rhythm, which can be dynamically adjusted according to the model type and training task.

[0128] A two-branch forgetting structure is adopted, and a rapid forgetting factor is set. To adapt to short-term fluctuations, slow forgetting factor To maintain long-term stability, use the fusion coefficient. Achieving a dynamic weighted fusion of the two yields the final forgetting factor. . , Can be set separately , , Parameters such as these.

[0129]

[0130] Among them, the rapid forgetting factor Its core function is to quickly respond to short-term gradient fluctuations. When gradients are active and parameter distributions change abruptly, it can rapidly increase the forgetting rate, preventing historical residuals from interfering with new optimization directions. (Slow Forgetting Factor) Its core function is to ensure long-term training stability, avoid drastic changes in the forgetting rate due to short-term gradient fluctuations, and ensure the stable retention of long-term effective residual information. This is the fusion coefficient, used to balance the weights of the two branches.

[0131] Based on the aforementioned forgetting factor, the final residual correction value is obtained. .

[0132]

[0133] After completing the above four-level compensation, the corrected residual is added to the current weight update amount to obtain the corrected update amount:

[0134] .

[0135] Step 3: Conflict-aware gating and dynamic dead zone sparsity reduction;

[0136] like Figure 2As shown, the transmission cycle gating for conflict awareness of the correction update amount is performed. Quantization filtering is performed based on a dynamically adjusted dead zone threshold. Small updates below the threshold are accumulated as new error residuals and stored in onboard memory. Updates above the threshold are packaged into an indexed sparse data stream. The specific steps include the following.

[0137] Step 3.1: Conflict-aware transmission cycle gating: Calculate the global conflict coefficient, perform global conflict-aware collaborative suppression on the weight update amount, quantify the consistency of different nodes on the same weight update direction, aiming to solve the situation where the gradient updates of multiple nodes are inconsistent but converge to 0. This mechanism calculates the global conflict coefficient, quantifies the consistency of different nodes on the same weight update direction, dynamically adjusts the transmission frequency of high-conflict elements, avoids invalid communication, and improves the communication efficiency and convergence stability of distributed training.

[0138] Calculate the global conflict coefficient :

[0139]

[0140] in, Indicates the first Round, the first The global conflict coefficient for each weight; a larger value indicates that the update directions of each node for that element are more inconsistent, the conflict is more severe, and the proportion that is canceled out during aggregation is higher. Indicates the first The aggregate weight of each node, Indicates the first Each node, numbered The change in weight.

[0141] Based on the global collision coefficient, the remaining transmission interval for each weight element is calculated. Make dynamic adjustments:

[0142]

[0143] in, This is the interval between the previous transmission rounds, initially set to 0. This is the interval adjustment factor. When When the remaining transmission interval is decremented by 1, it indicates that more transmission intervals are needed. The element will be transmitted only in the next round; it will not be transmitted in the current round. At that time, the transmission interval for the next round is recalculated based on the collision coefficient; the more severe the collision (…), the higher the probability of a collision. The larger the value, the larger the transmission interval, and all nodes will uniformly reduce the transmission frequency of that element, further compressing invalid communication; the milder the conflict, the smaller the transmission interval and the higher the transmission frequency, ensuring timely updates of effective gradients.

[0144] Step 3.2: Dynamic Dead Zone Threshold Calculation: The dead zone threshold is adaptively adjusted based on the bandwidth utilization from the previous round to achieve a balance between communication bandwidth and model accuracy. The threshold update formula is:

[0145]

[0146] in, Indicates the first dead zone threshold of the wheel This is the bandwidth adjustment coefficient, used to control the intensity of the influence of bandwidth utilization on the threshold, and determines the adjustment range of the threshold with bandwidth deviation. For target bandwidth utilization, The actual bandwidth utilization rate is used to increase the dead zone threshold when the actual bandwidth utilization rate is higher than the target bandwidth utilization rate, thereby enhancing sparsity and releasing bandwidth margin; conversely, the threshold is lowered when the actual bandwidth utilization rate is lower than the target bandwidth utilization rate to retain more effective information.

[0147] Step 3.3: Dead Zone Quantization and Sparse Coding: Differentiated processing of weight update amounts by combining transmission interval and dynamic dead zone threshold: Differentiated processing is applied to different types of weight update amounts by combining transmission interval and dead zone threshold, and weight updates are divided into three categories: delayed transmission, error storage and sparse transmission, to maximize the compression of communication data volume while ensuring the transmission of effective gradients, thereby achieving refined communication optimization.

[0148] for The weights are not transmitted in this round. High-conflict updates are temporarily stored locally through delayed transmission to avoid invalid communication. Simultaneously, these updates are accumulated in an error reservoir and sent all at once in subsequent transmissions, reducing the number of communications. Residual Updated to .

[0149] for The weights are used to perform dead zone quantization and determination. Based on the relationship between the update magnitude and the dynamic dead zone threshold, two cases are distinguished:

[0150] If the change in weight is less than the dynamic dead zone threshold, then... If the update magnitude is below the dead zone threshold, it has little impact on model optimization. It is accumulated in the error reservoir and subsequently merged and transmitted, avoiding frequent transmission of low-effectiveness data. Residual Updated to .

[0151] If the weight change is greater than or equal to the dynamic dead zone threshold, i.e. The weight change is multiplied by the node aggregation weight, and the resulting data is labeled with a sparse index. The sparse encoding and packaging module then packages the data into a sparse data stream using a [address + change] structure and outputs it to the line-rate encryption / decryption subsystem. Simultaneously, the residual is set to 0. When the magnitude of such updates exceeds the dead zone threshold, it has a significant impact on model optimization. Therefore, sparse encoding and transmission are directly performed to ensure timely updates of effective gradients, while zeroing the residuals avoids repeated accumulation.

[0152] Step 4: Line-speed encryption and intranet ciphertext aggregation;

[0153] like Figure 3 As shown, during system initialization, a unified initial symmetric key is obtained from the host; sparse data streams are encrypted at line speed, with the encryption rate perfectly matching the line speed of the high-speed serial interface; data is aggregated and transmitted in each round, and bandwidth utilization is statistically analyzed in real time to adaptively adjust the dead zone threshold; after each round of aggregation, the global ciphertext update is packetized according to the protocol and returned to the host GPU, and the next round session key is generated using the SHA-256 hardware hash function based on the global aggregation result.

[0154] The aggregation method is a circular aggregation topology, totaling Each node performs [the following actions]. Round transmission, the first Round-robin transmission, each node will transmit the first... After being fragmented and encrypted, the data packets are sent to downstream devices. The transmitted data packets include the device number and sparse data stream. In the ring aggregation topology, the absolute value of the change in local statistical weights of each node is weighted and summed to calculate the global conflict coefficient for each weight.

[0155] like Figure 4 As shown, this invention also proposes an FPGA-based memory-decoupled sparse gradient aggregation and error compensation system, which includes a host interaction and global control subsystem, an onboard memory management subsystem, a residual fusion and error compensation subsystem, a dead-zone quantization and sparse coding subsystem, a line-speed encryption / decryption subsystem, and a distributed sparse gradient aggregation operation subsystem.

[0156] Host Interaction and Global Control Subsystem: This subsystem serves as the communication interface and global scheduling core between the FPGA smart network card and the host GPU. It is responsible for data transmission, command interaction, and overall system operation scheduling between the host and the network card. It includes a PCIeDMA module, a command parsing module, and a global control module.

[0157] The PCIe DMA module enables bidirectional high-speed data transmission between the FPGA and the host GPU. It receives and unpacks data packets from the host, automatically sends local weight updates and Adam second-order momentum to the onboard memory management subsystem, and sends control commands and parameter configuration commands to the command parsing and global control module. After aggregation, it packages the global aggregation results from the FPGA and sends them back to the GPU.

[0158] The command parsing and global control module is the global scheduling and command center of the FPGA smart network card. It is responsible for parsing various control commands issued by the host, such as training start / stop, topology configuration, parameter update, and key management, as well as the operation parameter configuration commands of each subsystem. It synchronously distributes the parsed commands and parameters to all functional modules such as error compensation, sparse coding, encryption, and aggregation, and coordinates the working sequence and execution flow of each module. At the same time, it collects the running status and task progress of all subsystems in real time, summarizes and organizes them, and feeds them back to the host after querying the commands. When other subsystems encounter anomalies, they directly send interrupt data packets to the host through the PCIe bus to realize real-time response to anomalies.

[0159] Onboard memory management subsystem: This subsystem is implemented based on the external DDR SDRAM of the FPGA and the internal control module of the FPGA. It is responsible for independently storing various data during system operation, achieving complete decoupling between error states and GPU memory, and avoiding the GPU occupying additional memory to maintain the residual matrix. It includes an address mapping and management module, a read / write arbitration module, and a DDR memory control module.

[0160] The address mapping and management module is responsible for managing the data partitions of DDR physical memory. It adopts a linear addressing method of [base address + local offset]. The base address of each partition can be dynamically configured, and the local offset is automatically calculated by the data length and access location. The DDR4 physical address space is divided into 6 independent storage partitions: the weight update amount partition is used to store the local weight update amount matrix of the current round issued by the GPU; the Adam momentum parameter partition is used to store the second-order momentum parameters of the current round and the previous round issued by the GPU; the error residual matrix partition is used to store the truncation residual error generated by dead zone quantization, which is the core partition for realizing the decoupling of video memory; the sparse index data partition is used to store the sparse data stream of [index + quantization value] generated by sparse coding; the transmission interval partition is used to store the transmission interval of each weight change amount; and the global aggregation result partition is used to store the global weight update amount completed by multi-node network aggregation and the weighted sum of the absolute values ​​of the global weight update amounts.

[0161] The read / write arbitration module uses a polling arbitration mechanism to resolve conflicts when multiple functional modules simultaneously request DDR4 read / write operations, ensuring that all modules' access requests are responded to in a timely manner. It supports burst transmission of the AXI4-Stream protocol, automatically merging multiple small read / write requests from consecutive addresses into a large burst transmission request to increase bandwidth utilization.

[0162] The DDR memory control module is the hardware interface between the FPGA and the external DDR. It is customized based on the MIG IP core and is responsible for the power-on initialization, timing calibration, read / write timing control and data transmission of the DDR chip, realizing high-speed and reliable data interaction between the FPGA logic and physical memory.

[0163] The residual fusion and error compensation subsystem is the core module for error closed-loop correction. Implemented using fully pipelined hardware logic, it is responsible for multi-level correction of the previous round's error residuals in onboard memory, followed by fusion with the current weight update, thus resolving the incompatibility issue between residuals and GPU-side updates in a decoupled memory architecture. The subsystem comprises a magnitude matching scaling module, a direction-aware module, a time-varying compensation module, a dual-branch forgetting module, and a residual fusion module. Intermediate residual results are transmitted between modules via a data bus with an enable signal, and the processing of a single weight element is completed in a fully pipelined manner.

[0164] Within each module, multiplication and addition operations are implemented using DSP resources, division operations can be implemented using shift-subtract pipelined flip-flops, clamping functions are implemented using comparators, and square root operations, exponential operations, and hyperbolic tangent operations can all be implemented using the CORDIC algorithm. Among them, the direction modulation intensity, time-varying compensation intensity, and forgetting factor are coefficients related to the training rounds and training stages. They are calculated only once per training round and stored in a global register for all data to be shared in this round, which greatly saves hardware resources.

[0165] Dead-zone quantization and sparse coding subsystem: This subsystem is the core module of communication compression, implemented using fully pipelined hardware logic. It is responsible for collision-aware gating and dynamic dead-zone quantization of the corrected update quantity, generating a sparse data stream to reduce communication bandwidth requirements. The subsystem consists of a transmission interval gating module, a dynamic dead-zone quantization module, and a sparse coding and packing module.

[0166] The transmission interval gating module, based on the pre-stored remaining transmission interval for each weight, filters out elements allowed to be transmitted in the current round. Elements not participating in this round are directly accumulated using residuals to avoid invalid communication. The remaining transmission interval parameter for each weight is stored in a dedicated transmission interval partition in onboard memory. After each round of aggregation, it is updated uniformly by combining the global weight change and a weighted sum of the weight changes. Pipeline processing... When assigning a weight, synchronously index the current weight's address and read the remaining transmission interval from the previous round for that weight from the onboard memory. .like Mark this weight as non-transferable. Marked as transferable in this round, it is input into the dynamic dead zone quantization module.

[0167] The dynamic dead zone quantization module receives bandwidth utilization statistics from the line-rate encryption / decryption subsystem. In each training round, it dynamically generates a globally unified dead zone quantization threshold, quantizes the transmittable weight updates and marks them with sparse coding, and marks the non-transmittable weight updates and minor redundancies with residual accumulation. The results are then output to the sparse coding and packing module.

[0168] The sparse coding and packaging module multiplies the effective update amount of the sparse coding marker by the node aggregation weight to generate a weighted effective update amount, which is then converted into a standard sparse data stream in the format of [index + quantization value]. It also supports continuous address index compression: when the weights of multiple consecutive addresses are all effective updates, they are automatically merged into the format of [base address + offset + numerical array], further improving the compression ratio, while the residual of the corresponding weight update amount is set to 0. The weight update amounts marked with residual accumulation are removed and directly accumulated in the error reservoir. The generated sparse data stream is sent to the line-rate encryption / decryption subsystem, and the truncated residual error matrix is ​​sent to the onboard memory management subsystem.

[0169] Line-speed encryption / decryption subsystem: This subsystem is the core module for gradient data security protection. It is implemented in fully pipelined hardware using symmetric cryptography algorithms, perfectly matching the line speed of the high-speed serial interface with no additional processing latency. The subsystem includes a line-speed encryption / decryption module, a high-speed serial transmission module, and a key derivation module.

[0170] The line-rate encryption / decryption module, implemented in fully pipelined hardware using symmetric cryptography algorithms, is directly embedded in the data path. It performs real-time encryption of the gradient plaintext in the transmitting direction and real-time decryption of the ciphertext in the receiving direction. The module's throughput perfectly matches the line-rate of the high-speed serial interface. No additional processing delay is introduced as data flows through the encryption / decryption logic, and it can output the corresponding ciphertext or plaintext within a fixed, extremely short period, ensuring that gradient confidentiality protection does not become a communication bottleneck.

[0171] The high-speed serial transmission module is responsible for providing access to high-speed serial physical links and handling underlying transmission functions such as serialization and deserialization, and line encoding. It is tightly coupled with the line-speed encryption / decryption module. Data received from the encryption module is directly serialized into high-speed differential signals for transmission, and serial signals received from the link are directly restored to parallel data and handed over to the decryption module. Throughout the entire process, the data width and rate are perfectly aligned with the encryption processing, requiring no buffering or flow control intervention, achieving true line-speed uninterrupted transmission.

[0172] Before each training round, the key derivation module distributes an initial session key from the host, serving as the starting point for the entire key derivation chain and ensuring that all nodes possess the same symmetric key in the initial state. After each training round, all nodes obtain an identical aggregation result. The key derivation module extracts this aggregation result in real time, using it as the core derivation material to generate new keys using a hardware-implemented key derivation function. Because the initial key of each node and the aggregation results of subsequent rounds are completely consistent, the derived session key for the next round is naturally consistent, requiring no additional key negotiation communication. Key updates are fully tied to the iterative rhythm of distributed training, and each training round uses a one-time session key, greatly enhancing the security of gradient data.

[0173] The distributed sparse gradient aggregation subsystem is responsible for aggregating the decrypted sparse data stream within the network, while simultaneously performing distributed statistics and global distribution of gradient collision coefficients. The subsystem includes an aggregation control module and a sparse index alignment module.

[0174] The aggregation control module is the core of the subsystem's scheduling and control, responsible for managing node roles, synchronizing global timing, and controlling data routing. In a ring topology, it allows configuration of node IDs, left and right neighbor node addresses, and ring transmission direction. Nodes Precise timing control of round-robin transmission; each node locally calculates the weighted sum of the absolute values ​​of all weight changes, generating a sparse data stream with a local collision coefficient.

[0175] The sparse index alignment and aggregation module is responsible for index matching and address mapping of multiple sparse gradient data streams and multiple conflict coefficient sparse data streams. This resolves the issues of discontinuous sparse data and inconsistent indices, ensuring the correctness of aggregation operations. An independent FIFO buffer is allocated to each input data stream to cache sparse data streams from different nodes. During sparse index alignment and aggregation, the index in each FIFO is read, and the index and value with the smallest address are extracted. If there are identical indices, aggregation is performed, and the index and the change in aggregation weights are output. If there are no identical indices, the index and the change in weights are directly output.

[0176] like Figure 5 As shown, a complete workflow example based on the above system is as follows:

[0177] 1. System power-on initialization: The host sends initial configuration parameters to the FPGA smart network cards of multiple nodes via the PCIe bus, including the total number of training rounds, initial dead zone threshold, initial symmetric key, ring aggregation topology mode, data batch size of each node, parameters of each module, etc.; each subsystem completes parameter configuration and enters the ready state.

[0178] 2. Local Training and Parameter Distribution: Each GPU node begins local training, solves the local gradient of the SNN, and calculates the weight update amount through the Adam optimizer. With second momentum It is sent to the local FPGA smart network card via the PCIe bus.

[0179] 3. Residual Fusion and Error Compensation: Each node's FPGA smart network card reads the previous round's error residual from the onboard memory, sequentially performs magnitude matching scaling, direction awareness enhancement, time-varying compensation, and dual-branch forgetting modulation, and fuses it with the current weight update to obtain the corrected update. .

[0180] 4. Dead Zone Quantization and Sparse Coding: Each node performs conflict-aware transmission cycle gating and dead zone quantization on the correction update amount, generating a sparse data stream in the format of [index + quantization value]. Minor updates below the threshold are stored in an error reservoir.

[0181] 5. Encrypted transmission and weighted aggregation: After the sparse data stream is encrypted at line speed, it is transmitted between nodes in a ring topology; each node receives and decrypts the encrypted data packets, performs weighted aggregation on the sparse data stream, and generates a global sparse update quantity.

[0182] 6. Key update and result return: Each node generates the next round of session key based on the global aggregation result, packages the global sparse update according to the protocol, and returns it to the local GPU.

[0183] 7. Model Update and Iteration: After receiving the global update, the GPU updates the local model weights and enters the next round of training; repeat steps 2-6 until all rounds of training are completed.

[0184] 8. Training ends: The host sends a training stop command, and the FPGA smart network cards on each node stop working and enter an idle state; the GPU saves the model parameters after training is completed, completing the entire distributed training task.

[0185] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A memory-decoupled sparse gradient aggregation and error compensation method based on FPGA, characterized in that... Includes the following steps: Step 1: The host GPU maintains the optimizer momentum state in the node's local video memory and calculates the local gradient of the spiking neural network (SNN). The obtained weight update and momentum state parameters are then unloaded to the node's FPGA network card. Step 2: The network card reads the residual error from the previous round stored in the onboard memory, and after error compensation, merges it with the current weight update amount to obtain the corrected update amount. Step 3: Perform conflict-aware transmission cycle gating on the correction update amount, perform quantization filtering based on the dynamically adjusted dead zone threshold, accumulate updates below the dead zone threshold as new error residuals and store them in the onboard memory, and package updates above the threshold into an indexed sparse data stream after combining node aggregation weights. Step 4: After the sparse data stream is encrypted at line speed using a key, it is transmitted between multiple nodes and globally aggregated within the network. During the aggregation transmission process, the dead zone threshold is adjusted by statistically analyzing the bandwidth utilization. After the aggregation is completed, the global ciphertext update is returned to the host, and a new key is dynamically derived based on the global aggregation result.

2. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation method according to claim 1, characterized in that: In step 1, the host GPU maintains the first-order momentum state and the second-order momentum state of the optimizer in the node's local video memory. The error compensation in step 2 includes magnitude matching scaling. The scaling factor is calculated based on the second-order momentum of the optimizer in the previous and current rounds. The scaling factor is multiplied by the residual error of the onboard truncation in the previous round to obtain the current truncation residual error after magnitude matching.

3. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation method according to claim 2, characterized in that: The error compensation in step 2 also includes direction-aware modulation. Based on the weight update amount and the truncation residual error of the previous round, the direction consistency coefficient between the residual and the current update amount is calculated. The direction consistency coefficient is then soft-limited to obtain the direction-aware coefficient. The direction-aware coefficient is combined with the direction modulation intensity, which decays exponentially with the number of iterations, to generate a direction enhancement factor. The residuals are modulated using the aforementioned direction enhancement factor to enhance the in-direction residuals and suppress the reverse-direction residuals, resulting in an updated truncated residual error.

4. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation method according to claim 3, characterized in that: The error compensation in step 2 also includes time-varying compensation, which dynamically adjusts the compensation intensity according to the training stage and adopts segmented compensation to adapt to the residual confidence level at different stages. In the very early stages of training, a low-intensity, tentative compensation with linear growth is used; In the early stages of training, a linearly accelerated compensation intensity was used; in the middle stages of training, a constant peak compensation intensity was used. In the later stages of training, an exponentially decaying compensation intensity is used; the updated truncation residual error is modulated by dynamically adjusted compensation intensity to obtain the time-varying compensated truncation residual error.

5. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation method according to claim 4, characterized in that: The error compensation in step 2 also includes joint error compensation of dual-branch forgetting modulation; a dual-branch forgetting structure is adopted, a fast forgetting factor is set to adapt to short-term fluctuations, and a slow forgetting factor is set to maintain long-term stability. A fusion coefficient is used to realize the dynamic weighted fusion of the two to obtain the final forgetting factor. Based on the forgetting factor, the truncated residual error after time-varying compensation is corrected to obtain the final residual correction value. The corrected residual is then added to the current weight update amount to obtain the corrected update amount.

6. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation method according to claim 1, characterized in that: In step 3, the transmission cycle gating for conflict perception is based on the aggregate weights and weight changes of nodes to construct a global conflict coefficient, which represents the consistency of different nodes in updating the same weight. The transmission interval of the weight is dynamically adjusted according to the conflict coefficient, so that the more severe the conflict, the larger the transmission interval, and all nodes consistently reduce the transmission frequency of the weight. The less severe the conflict, the smaller the transmission interval and the higher the transmission frequency.

7. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation method according to claim 6, characterized in that: In step 3, the dead zone threshold is dynamically adjusted based on the deviation between the target bandwidth utilization and the actual bandwidth utilization, so that the dead zone threshold increases when the actual bandwidth utilization is higher than the target bandwidth utilization. Combining the aforementioned transmission interval and dynamic dead zone threshold, different types of weight update amounts are processed differently, and weight updates are divided into delayed transmission, error storage, and sparse transmission. For transmission interval The weight of the update is not transmitted in this round. The update volume with high collision is temporarily stored locally by delaying the transmission, and these updates are accumulated in the error reservoir and sent all at once in the subsequent transmission. For transmission interval The weights are used to perform dead zone quantization and determination. Based on the relationship between the update magnitude and the dynamic dead zone threshold, two cases are distinguished: If the weight change is less than the dynamic dead zone threshold, sparse coding is not performed and the data is directly stored in the error reservoir. If the change in weight is greater than or equal to the dynamic dead zone threshold, the change in weight is multiplied by the node aggregation weight, and the resulting result is labeled with a sparse index.

8. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation method according to claim 1, characterized in that: In step 4, a unified initial symmetric key is obtained from the host during initialization. The sparse data stream is then encrypted at line speed and transmitted among multiple nodes according to a ring topology and aggregated within the network. After each round of aggregation, all nodes generate the next round session key through a key derivation function based on the same global aggregation result. Finally, each node transmits the global weight update amount back to the host. The aggregation of the ring topology involves N nodes, each of which performs N-1 rounds of transmission. In the k-th round of transmission, each node encrypts the (ID+1-k) mod N fragment and sends it to the downstream device. The transmitted data includes the device ID of the node and the sparse data stream. In the ring aggregation topology, the absolute value of the change in local statistical weights of each node is weighted and used to calculate the global conflict coefficient for each weight.

9. A memory-decoupled sparse gradient aggregation and error compensation system based on FPGA, comprising a host interaction and global control subsystem, an onboard memory management subsystem, a residual fusion and error compensation subsystem, a dead-zone quantization and sparse coding subsystem, a line-speed encryption / decryption subsystem, and a distributed sparse gradient aggregation operation subsystem, characterized in that... Gradient aggregation and error compensation are performed using the FPGA-based memory decoupled sparse gradient aggregation and error compensation method according to any one of claims 1 to 8. The host interaction and global control subsystem receives the local weight update and optimizer momentum state parameters sent by the host GPU, parses the control commands and schedules other subsystems to work. After the intra-network aggregation is completed, the global weight update is sent back to the host according to the protocol packet. The onboard memory management subsystem performs unified storage and read / write management of various types of data during operation; The residual fusion and error compensation subsystem reads the previous round error residual from the onboard memory and performs multi-level corrections before fusing it with the current weight update amount to obtain the corrected update amount. The dead zone quantization and sparse coding subsystem performs conflict-aware gating and dynamic dead zone quantization on the corrected update amount, generates a sparse data stream and updates the error residual. The line-speed encryption / decryption subsystem uses a symmetric cryptographic algorithm for line-speed encryption / decryption and dynamic key derivation. The distributed sparse gradient aggregation subsystem performs intra-network aggregation on the decrypted sparse data stream.

10. The FPGA-based memory-decoupled sparse gradient aggregation and error compensation system according to claim 9, characterized in that: The residual fusion and error compensation subsystem includes a magnitude matching scaling module, a direction-aware modulation module, a time-varying compensation module, a dual-branch forgetting modulation module, and a residual fusion module. After performing joint error compensation on the previous round of error residuals in sequence, the corrected error residuals are added to the current weight update amount to obtain the corrected weight update amount. The dead zone quantization and sparse coding subsystem includes a transmission interval gating module, a dynamic dead zone quantization module, and a sparse coding packing module, which are respectively used to perform conflict-aware transmission cycle gating, dynamic dead zone threshold adjustment, weight update amount to perform dead zone quantization and update truncation residual error, and pack update amount higher than the dead zone threshold into sparse format. The line-speed encryption / decryption subsystem includes a line-speed encryption / decryption module, a high-speed serial transmission module, and a key derivation module, which are used to perform line-speed encryption on sparse indices and quantized values, transmit and receive data between boards, and automatically generate the next round session key based on a unified global aggregation result in each round. The distributed sparse gradient aggregation subsystem includes an aggregation control module and a sparse index alignment module, which are used to configure relevant parameters and perform index alignment and aggregation operations between the local sparse index set and the upstream aggregation results.

Images