Sparse Deep Neural Networks without Multipliers

Abstract

Deep Neural Networks (DNNs) have been used in various applications to provide inference, regression, classification and prediction. Although the high potential of DNNs has been successfully demonstrated in the literature, most DNNs require high computational complexity and high power operations for real-time processing due to a large number of multiply-accumulate (MAC) operations. The present invention provides a method for realizing a hardware-friendly MAC-free DNN framework using round-accumulate (RAC) operations based on power-of-two (PoT) weights. The method and system are based on the recognition that round-aware training for power-of-two extensions can eliminate multiplier components from the system without causing any performance loss. The method and system also provide a method for reducing the number of PoT weights based on knowledge distillation using progressive compression of over-parameterized DNNs. It can achieve high compression, leading to power-efficient inference for resource-limited hardware implementations such as microprocessors and field programmable gate arrays. The compaction rank is optimized with a further DNN model in a reinforcement learning framework. Also, for low-power processing, the rounding granularity is successively decremented to obtain mixed-order PoT weights.,Another learner model is also designed in parallel with the,knowledge distillation to find the Pareto optimal tradeoff,between performance and complexity.

Description

[Technical field]

[0001] The present invention relates to machine learning and inference systems based on artificial neural networks, and more particularly to a multiplier-less method and system for deep neural networks with round-accumulate operations and weight pruning. [Background technology]

[0002] Deep Neural Network (DNN) based machine learning has been applied to many different problems including but not limited to image classification, speech recognition, computational sensing, data analysis, feature extraction, signal processing and artificial intelligence. There are many DNN architectures such as multi-layer perceptrons, recurrent networks, convolutional networks, transformer networks, attention networks, long and short-term memory, generative adversarial networks, autoencoders, U-shaped networks, residual networks, reversible networks, loopy networks, clique networks, and their variants. Although the high performance of DNNs has been demonstrated in many real-world systems, training and deploying DNNs on resource-limited hardware with low memory and processing power for real-time applications has remained a challenge. This is in part because typical DNNs have a large number of trainable parameters such as weights and biases, and many artificial neurons across deep hidden layers also require a large number of arithmetic multiplications and floating-point operations.

[0003] In the prior art, several memory-efficient DNN methods and systems have been proposed to solve part of this problem. One type of such approach involves pruning, sparsifying or compacting weights and nodes. Another type of approach is based on quantization of weights and activations. They are both known as network distillation techniques. Weight pruning forces some DNN weights to be zero and the corresponding edges are removed to achieve a more compact DNN architecture. For example, L1 / L2-regularization is used to achieve sparse weights and compress the network size. Recently, another simple approach for weight pruning has been proposed based on the Lottery Ticket Hypothesis (LTH). LTH pruning creates a pruning mask with the trained DNN and rewinds weights in earlier epochs for edges that have not been pruned. It has been shown that LTH pruning can provide better performance with more compact DNNs.

[0004] Weight quantization compresses DNN weights by converting high-precision values ​​to low-precision values. For example, a DNN is first trained with double-precision floating-point weights, which typically require 64 bits to represent, and then these weights are quantized to 16-bit half-precision or 8-bit integer values. This quantization can reduce the DNN memory requirements for deployment by a factor of four or eight, leading to lower power systems. Binarization or even ternarization of weights is developed to achieve low-memory DNNs. Binarized and ternary DNNs show reasonable performance on some specific datasets, but show performance loss on most real-world datasets. Alternative quantization techniques include rounding, vector quantization, codebook quantization, stochastic quantization, random rounding, adaptive codebook, incremental quantization, sum of powers of two, or power of two quantization. In some systems, in addition to weight quantization, activations and gradients are quantized. In order to reduce the performance degradation caused by those network distillations, there are many training methods based on quantization / pruning-aware training in addition to dynamic quantization.

[0005] However, in general, most distillation techniques suffer from significant performance degradation due to resource constraints. Also, there is no unified method to realize DNNs that are both sparse and quantized without compromising performance. Therefore, there is a need to develop methods and systems for low-power and low-complexity deployment of DNNs applicable for high-speed real-time processing. Summary of the Invention

[0006] Recently, deep neural networks (DNNs) have been studied for various applications including telecommunications, speech recognition, image processing, etc. Although the high potential of DNNs has been successfully demonstrated in many applications, DNNs generally require high computational complexity and high power operations in real-time processing due to a large number of Multiply-Accumulate (MAC) operations. The present invention provides a hardware-friendly DNN framework using Round-Accumulate (RAC) operations to realize multiplier-free operations based on power-of-two quantization or its variants. We demonstrate that Quantization-Aware Training (QAT) for Additive Powers-of-Two (APoT) weights can eliminate multipliers without causing visible performance loss. We also introduce weight pruning based on a progressive version of the Lottery Ticket Hypothesis (LTH) to sparsify over-parameterized DNN weights to further reduce the complexity. We demonstrate that the present invention is able to prune most weights, leading to power-efficient inference for resource-limited hardware implementations such as microprocessors or Field-Programmable Gate Arrays (FPGAs).

[0007] The method and system are based on the realization that rounding-aware training for power-of-two expansion can eliminate multiplier components from the system without incurring significant performance loss. The method and system also provide a method to reduce the number of PoT weights based on knowledge distillation using progressive compaction of over-parameterized DNNs. It can achieve high compaction, leading to power-efficient inference for resource-limited hardware implementations. The compaction rank is optimized with further DNN models in a reinforcement learning framework. Also, for low-power processing, the rounding granularity is successively decremented to obtain mixed-order PoT weights. Another learner model is also designed in parallel with knowledge distillation to find a Pareto-optimal tradeoff between performance and complexity.

[0008] The present invention provides a method to realize low-complexity, low-power DNN design without requiring multipliers. Our method uses progressive updates of LTH pruning and quantization-aware training. Hyperparameters are automatically optimized using Bayesian optimization. Multiple solutions are obtained by multi-seed training to find the Pareto-optimal tradeoff between performance and complexity. The pruning ranking is optimized using further DNN models. And for low-power processing, the quantization granularity is successively decremented to obtain APoT weights of mixture order. In parallel with the knowledge distillation, another learner model derived from the parent model is also designed. The embodiment includes digital predistortion, MIMO equalization, nonlinear turbo equalization, etc.

[0009] Some embodiments of the present invention provide a hardware-friendly DNN framework for non-linear equalization. Our DNN realizes operations without multipliers based on power-of-two quantization. We demonstrate that quantization-aware training (QAT) for power-of-two sum (APoT) weights can eliminate multipliers without causing a visible performance loss. We also introduce lottery ticket hypothesis (LTH)-based weight pruning to sparsify over-parameterized DNN weights to further reduce complexity. We prove that progressive LTH pruning can prune most of the weights, resulting in power-efficient inference in real-time processing.

[0010] To reduce the computational complexity of DNN calculations, we integrate APoT quantization into the DeepShift framework. In the original DeepShift, DNN weights are quantized to signed PoT as w = ±2 u where u is an integer for training. Since PoT weights can be realized by bit-shifting for fixed-point precision or addition operations for floating-point precision, multiplier operations can be eliminated from the DNN equalizer. The present invention further improves it using APoT weights: w = ±(2 u ±2 v ), and we use another trainable integer v < u. It requires additional addition but no multiplication. We used straight-through rounding to obtain u and v after each epoch iteration once the pre-training phase was done in the QAT update. In some embodiments, higher-order additive PoT is used to further reduce quantization error.

[0011] Our DNN equalizer does not require multipliers, but still requires a relatively large number of addition operations. We introduce a progressive version of the LTH pruning method to achieve a low-power sparse DNN implementation. It is known that over-parameterized DNNs can be significantly sparsified without compromising performance, and the sparsified DNNs often outperform dense DNNs. In our progressive LTH pruning, we first train a dense DNN via QAT for APoT quantization, starting from random initial weights. We then prune a small portion of edges based on the trained weights. We rewind the weights for unpruned edges to the weights of the earlier epoch, and then retrain the pruned DNN. The rewinding, QAT update, and pruning are repeated with a gradual increase in the pruning percentage. We use the weights of the first epoch to be rewound later.

[0012] Therefore, zero-multiplier sparse DNNs can be used for many applications, including high-speed optical communications that utilize stochastic shaping. In some embodiments, APoT quantization can achieve floating-point arithmetic performance without multipliers, and progressive LTH pruning can eliminate 99% of the weights for power-efficient implementation.

[0013] According to some embodiments of the present invention, there is provided a computer-implemented method for training a set of artificial neural networks, the method being executable by one or more computing processors in association with a memory storing a computer-executable program. The method includes: (a) initializing a set of trainable parameters of an artificial neural network, the set of trainable parameters including a set of trainable weights and a set of trainable biases, the method further includes: (b) training the set of trainable parameters with a set of training data; (c) generating a pruning mask based on the trained set of trainable parameters; (d) unwinding the set of trainable parameters; (e) pruning a selected set of trainable parameters based on the pruning mask; and (f) repeating the above steps (b) through (e) a predetermined number of times to generate a set of sparse neural networks with incremental sparsity.

[0014] Additionally, some embodiments of the invention may provide a computer-implemented method for testing an artificial neural network, the method being executable by one or more computing processors and including feeding a set of test data to a number of input nodes of the artificial neural network, propagating the set of test data throughout the artificial neural network according to a set of pruning masks, and generating a set of output values ​​from a number of output nodes of the artificial neural network.

[0015] Further, some embodiments provide a system deployed for an artificial neural network, the system comprising at least one computing processor, at least one memory bank, at least one interface link, and at least one trained set of trainable parameters of the artificial neural network, the system having the processor execute a training method and a testing method based on the at least one trained set of trainable parameters, the method comprising: (a) initializing a set of trainable parameters of the artificial neural network, the set of trainable parameters including a set of trainable weights and a set of trainable biases, (b) training the set of trainable parameters with a set of training data, (c) generating a pruning mask based on the trained set of trainable parameters, (d) unwinding the set of trainable parameters, (e) pruning a selected set of trainable parameters based on the pruning mask, and (f) repeating the above steps (b) to (e) a predetermined number of times to generate a set of sparse neural networks with incremental sparsity.

[0016] Therefore, the embodiments can realize multiplier-free sparse DNN computation without performance degradation. Multiplier-free sparse DNN computation can be used for ultra-fast real-time applications that require low power and limited resource deployment.

[0017] The accompanying drawings, which are included to provide a further understanding of the invention, illustrate embodiments of the invention and together with the description explain the principles of the invention. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates an example system that utilizes a sparse deep neural network, according to some embodiments. [Diagram 2] FIG. 2 is an example signal diagram for sparse deep neural network denoising according to some embodiments. [Figure 3A] FIG. 2 illustrates an example deep neural network based on a forward multi-layer perceptron architecture using different loss functions, according to some embodiments. [Figure 3B] FIG. 2 illustrates an example deep neural network based on a forward multi-layer perceptron architecture using different loss functions, according to some embodiments. [Figure 4] FIG. 1 illustrates an example performance of a deep neural network using different loss functions, according to some embodiments. [Diagram 5] FIG. 1 illustrates exemplary performance of deep neural networks using different architectures, according to some embodiments. [Figure 6] FIG. 2 illustrates a multiplier-less affine transformation with sum-of-power-of-two weights for a quantized deep neural network according to some embodiments. [Figure 7A] FIG. 2 illustrates an example operation of a quantized affine transform according to some embodiments. [Figure 7B] FIG. 2 illustrates an example operation of a quantized affine transform according to some embodiments. [Figure 7C] FIG. 2 illustrates an example operation of a quantized affine transform according to some embodiments. [Figure 8] FIG. 2 illustrates an example overview of quantization-aware optimization for multiplier-less affine transformation with sum-of-power-of-two weights of quantized deep neural networks, according to some embodiments. [Figure 9] FIG. 1 illustrates an example performance of a quantized deep neural network with sums of powers of two of different orders, according to some embodiments. [Figure 10] FIG. 2 illustrates an exemplary overview of progressive pruning for incremental sparsification, according to some embodiments. [Figure 11] FIG. 1 illustrates an exemplary performance of a sparse deep neural network according to some embodiments. [Figure 12] FIG. 1 illustrates an example system that utilizes a sparse quantized neural network, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Various embodiments of the present invention will now be described with reference to the drawings. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. It should also be noted that the drawings are intended only to facilitate the description of specific embodiments of the present invention. The drawings are not intended as an exhaustive description of the present invention or as limitations on the scope of the present invention. Also, aspects described in connection with a specific embodiment of the present invention are not necessarily limited to that embodiment, and may be practiced in other embodiments of the present invention.

[0020] Some embodiments of the present disclosure provide a multiplier-less deep neural network (DNN) to mitigate fiber nonlinear distortions of shaped constellations. The novel DNN uses progressive lottery hypothesis (LHT) weight pruning and sum of powers of two (APoT) quantization to achieve a good performance-complexity tradeoff. Some other embodiments include, but are not limited to, digital predistortion, channel equalization, channel estimation, nonlinear turbo equalization, inference and prediction for speech recognition, image processing, biosignal sensing, etc. DNN Equalization for Probabilistic Amplitude Shaping (PAS) QAM Systems

[0021] The optical communication system 100 under consideration is shown in FIG. 1. This figure is an exemplary system utilizing a sparse deep neural network (DNN) according to some embodiments of the present invention. The transmitter side may use a Distribution Matcher (DM) 11 and a Forward Error Correction (FEC) coding 12 such as Low-Density Parity-Check (LDPC) or Turbo Code. Eleven channels of Dual-Polarization Quadrature-Amplitude Modulation (DP-QAM) 13 with 34 GBaud and 35 GHz spacing are multiplexed (MUX) 14 and transmitted over the fiber plant towards a coherent receiver. In some embodiments, we consider N spans of a dispersion-unmanaged link 15 with 80 km of Standard Single-Mode Fiber (SSMF) 16. The SSMF 16 may typically have a dispersion parameter D=17ps / nm / km, a nonlinear factor γ=1.2 / W / km, and an attenuation of 0.2dB / km. The span loss is compensated by an Erbium-Doped Fiber Amplifier (EDFA) 17 with a noise figure of 5dB. We may use digital root raised cosine filters with 2% roll-off at both the transmitter and receiver for Nyquist band-limited pulse shaping. The receiver utilizes standard digital signal processing after demultiplexing (DEMUX) 18 with symbol synchronization, carrier phase recovery, dispersion compensation and polarization recovery using a 61-tap Linear Equalization (LE) 19. Residual distortions are compensated by DNN equalization 20, which produces a Log-Likelihood Ratio (LLR) value for the FEC decoder 22 followed by an inverse DM 23. The LLR values ​​are also used to calculate loss values, such as the binary cross-entropy loss21, for training the DNN parameters.In some embodiments, to improve performance, the FEC decoded output is further fed back into the DNN equalization as turbo iterations.

[0022] Due to fiber nonlinearity, residual distortion after LE will limit the achievable information rate and degrade the bit error rate performance. Figure 2 is an example signal diagram for sparse deep neural network denoising, according to some embodiments. The diagram shows a distorted DP-64QAM constellation after least-squares LE 19 for the transmission of the 1st / 10th / 20th spans. Here, we compare uniform QAM with shaped QAM, which uses a distributed matcher (DM) 11 for probabilistic amplitude shaping (PAS) after Maxwell-Boltzmann distribution, Pr(x i ) ∝ exp(-λ|x i | 2 ) with λ = 2. We can observe that the shaped constellations are more distinguishable (11.51b / s / 4D symbols) since the Euclidean distance increases due to the reduction in entropy.

[0023] 3A and 3B are diagrams illustrating an example deep neural network based on a forward multi-layer perceptron architecture using different loss functions, according to some embodiments. FIG. 3A uses a single-label non-binary cross-entropy loss 310, and FIG. 3B uses a multi-label binary cross-entropy loss 360. To mitigate nonlinear distortions, some embodiments use a DNN architecture composed of multiple layers with artificial neurons. Specifically, an input layer 302, 352, a first hidden layer 303, 353, a second hidden layer 304, 354, and an output layer 305, 355 may be used to configure the DNN. The input layer 302, 352 may obtain several adjacent symbols using a finite-length window of the received sequence via a delay line 301, 351. Depending on the loss function, post-processing of the output values ​​may be required. For example, a DNN 300 using non-binary cross-entropy requires a symbol-to-bit LLR transformer 320, while a DNN 350 using binary cross-entropy can use the output layer as direct LLR values ​​370 without using a transformer.

[0024] A variety of different loss functions can be used to train a DNN, including but not limited to softmax cross entropy, binary cross entropy, distance, mean squared error, mean absolute error, connectionist temporal classification loss, negative log likelihood, Kullback-Leibler divergence, margin loss, ranking loss, embedding loss, hinge loss, Huber loss, etc. Specifically, the trainable parameters of a DNN architecture are trained by several steps of operations, including feeding a set of training data into the input nodes of the DNN, propagating the training data throughout the layers of the DNN, pruning the trainable parameters according to several pruning masks, generating output values ​​from the output nodes of the DNN, calculating loss values ​​for the training data, updating the trainable parameters based on the loss values ​​through backward message passing, and repeating these steps for a predetermined number of iterations.

[0025] 4 illustrates an example performance of a deep neural network using different loss functions according to some embodiments. For post-nonlinearity compensation, binary cross-entropy loss can outperform non-binary cross-entropy loss for training DNNs, especially in the case of high-order modulation.

[0026] To compensate for the residual nonlinear distortion, we use a DNN-based equalizer that directly generates bitwise soft-decision log-likelihood ratios (LLRs) for the decoder. FIG. 5 is a diagram illustrating an example performance of deep neural networks using different architectures, according to some embodiments. The diagram illustrates an example performance of deep neural networks using different architectures, according to some embodiments. The diagram illustrates the achievable speed of DP-256QAM over the SSMF span for various DNN equalizers, namely, residual multi-layer perceptron (6-layer 100-node ResMLP), residual convolutional neural network (4-layer kernel-3ResCNN) and bidirectional long-short-term memory (2-layer 100-memory BiLSTM). The binary cross-entropy loss is 2 14 2 for evaluating individual test symbols 16 The constellation shaping is minimized via Adaptive Momentum (Adam) with a learning rate of 0.001 for 2,000 epochs over training symbols. In some embodiments, in addition to Adam, the optimization algorithm is based on stochastic gradient descent, Nesterov gradient, elastic backpropagation, root mean square propagation, Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, adaptive momentum optimization, adaptive delta, etc. The optimizer has hyperparameters including learning rate and decay controlled by parameter scheduling methods such as exponentially decaying or step functions. We find that in ResMLP, constellation shaping can achieve a range extension of 29% at a target speed of 10 b / s. We find that using more hidden layers in CNN and LSTM architectures further improves training performance, but degrades test performance due to overfitting. It suggests that deeper models require larger training data sizes to provide better performance. Zero-multiplier DNN using Sum of Powers of Two (APoT) quantization

[0027] In order to reduce the computational complexity of DNN in real-time processing, this invention provides a method to integrate APoT quantization into the DeepShift framework. In the original DeepShift, the DNN weights are w=±2 u where u is an integer for training. Note that the PoT weights can be realized with bit-shifting for fixed-point precision or addition operations for floating-point (FP) precision, thus eliminating multiplier operations entirely from the DNN equalizer.

[0028] FIG. 6 illustrates a multiplier-free affine transformation with sum of powers of two (APoT) weights of a quantized deep neural network according to some embodiments. Most DNNs, including linear and convolutional layers, require an affine transformation 610 to propagate values ​​throughout the layers. In general, the affine transformation requires a matrix-vector multiplication that can be realized with a multiply-and-accumulate operation. When the affine weight matrix is ​​represented by APoT (615), the matrix-vector multiplication 620 for the affine transformation can be realized by a balanced ternary DNN 630, which requires bit-plane expansion 650 by bit-shifting and accumulation operations instead of multiplication. More importantly, the balanced ternary DNN 630 has the potential to reduce the number of addition operations 640 due to its high degree of freedom.

[0029] 7A, 7B, and 7C are diagrams illustrating example operations of a quantized affine transform, according to some embodiments. A typical affine transform requires multiply-accumulate operations as shown in FIG. 7A, but the PoT used in the DeepShift framework can eliminate multiplication operations by shift-and-accumulate operations 720. We further consider the APoT weight, i.e., w=±(2 u +2 v) is used to improve DeepShift and another trainable integer v < u is used. It requires additional addition as shown in Figure 7C, but does not require multiplication like PoT. Using the APoT weights, we can significantly reduce the residual quantization error of the conventional PoT as shown in Figure 7C compared to Figure 7B. Note that the original APoT uses a deterministic non-trainable lookup table, while our invention extends it using QAT as an improved DeepShift with trainable APoT weights. In some embodiments, how to quantize the weights is based on powers of two, sums of powers of two, powers of three, sums of powers of three, table lookup, vector quantization, etc. Operations without multiplication can achieve high-speed and low-power real-time DNN operations. In some embodiments, the order of the sum of powers of two, i.e., how many shift operations per weight value, can be adapted according to the accuracy requirements.

[0030] FIG. 8 illustrates an exemplary overview of quantization-aware optimization for multiplier-free affine transformation with sum-of-power-of-two weights of a quantized deep neural network, according to some embodiments. FIG. 8 includes (a) a multiplier-free DNN 800 and (b) a quantization-aware training (QAT) process. In the DNN 800, signals are forward-propagated to compute losses and backward-propagated to update weights with loss gradients. In some embodiments, we use straight-through rounding to find dual-bit-shifted integers u and v after each epoch iteration in the QAT update. Specifically, the DNN output is computed through forward propagation 820, and the loss gradients are back-propagated 830. After updating trainable parameters such as weights and biases through an Adam-like optimization step 840, the updated parameters are quantized in an APoT operation 810. In some embodiments, we may use a pre-training phase before QAT fine-tuning to stabilize DNN learning. In yet another embodiment, the DNN uses hypernetworks and feature-wise linear modulation (FiLM) to control the affinity transform for online adaptation, and these hypernetworks are also quantized similarly to the QAT. In some embodiments, to reduce the computational complexity of the QAT, the gradient values ​​in the backpropagation are further quantized with the APoT.

[0031] FIG. 9 is a diagram illustrating an example performance of quantized deep neural networks with sums of powers of two of different orders, according to some embodiments. The figure shows the achievable rates over launch powers at the 22nd span for a 6-layer 100-node ResMLP. PoT quantization has a small degradation of 0.11 b / s compared to the FP-precision DNN equalizer for shaped DP-64QAM, whereas a significant loss of 0.33 b / s is seen for shaped DP-256QAM. Notably, our multiplier-free DNN with APoT quantization has no degradation from FP precision (in fact, it is slightly improved). This is indeed a great advantage for real-time fiber nonlinearity compensation, since there is no performance loss but no multipliers are required. Lottery Ticket Hypothesis (LTH) Pruning for Sparse DNNs

[0032] Although our DNN equalizer does not require multipliers, it still requires a relatively large number of addition operations due to the over-parameterized DNN architecture with a huge number of trainable weights. We introduce a progressive version of the LTH pruning method to achieve a low-power sparse DNN implementation. It is known that over-parameterized DNNs can be significantly sparsified without compromising performance, and in LTH, sparsified DNNs often outperform dense DNNs. Figure 10 illustrates an exemplary overview of progressive pruning 1000 for incremental sparsification, according to some embodiments. The diagram illustrates progressive LTH pruning. We first train a dense DNN via QAT for APoT quantization, starting from random initial weights 1010, over a finite number of epochs 1020 toward a final update 1030. We then prune 1040 a small portion of edges based on the trained weights to generate a pruning mask. We re-train the pruned DNN after rewinding the weights to those of earlier epochs according to a pruning mask 1050 for unpruned edges. The rewinding 1060, QAT update 1070 over multiple epochs 1080, and pruning 1090 are repeated with a progressive increase in the pruning fraction. In some embodiments, we use later rewinding of the weights of later epochs. In some embodiments, the control of incremental sparsification is regulated by another auxiliary neural network trained by multiple training episodes through a deep reinforcement learning framework.The process is structured as the steps of initializing trainable weights and biases of a DNN, training those trainable parameters with training data, generating a pruning mask based on the trained parameters, unwinding the trainable parameters, pruning selected parameters based on the pruning mask, and repeating those steps a predetermined number of times to generate a set of sparse DNNs with incremental sparsity.

[0033] FIG. 11 is an exemplary performance of a sparse deep neural network, according to some embodiments. The figure shows the tradeoff between achievable speed and the number of non-zero weights. In general, in a dense DNN, more hidden nodes and more hidden layers can improve performance at the expense of computational complexity. As a result, a moderate depth, such as a 4-layer DNN, would be optimal in the Pareto sense of the performance-complexity tradeoff in the low-complexity regime. LTH pruning can significantly improve the tradeoff. That is, the sparse DNN can achieve a target speed of 10 b / s with more than 50% complexity reduction over the dense DNN. We can prune more than 99% of the weights of a 6-layer 100-node ResMLP with progressive pruning to maintain 10 b / s. As a result, the sparse DNN can achieve significantly lower complexity than the optimal dense DNN, by 73% and 87% for shaped 64QAM and 256QAM, respectively.

[0034] As mentioned above, various DNN equalizers are presented for nonlinear compensation in optical fiber communications using stochastic amplitude shaping. We then proposed a zero-multiplier sparse DNN equalizer based on a trainable version of APoT quantization and LTH pruning techniques. We showed that APoT quantization can achieve floating-point arithmetic performance without using multipliers, whereas traditional PoT quantization suffers from significant disadvantages. We also demonstrated that progressive LTH pruning can eliminate 99% of the weights, thus enabling a very power-efficient implementation of DNN equalization in real-time optical fiber systems.

[0035] FIG. 12 illustrates an exemplary system 500 utilizing a sparse quantized neural network, according to some embodiments. The system 500 may be referred to as a sparse quantized neural network system. The system 500 may include at least one computing processor 120, at least one memory bank 130, and at least one interface link 105, which may include an HMI 110, a NIC 150, or a combination of an HMI 110 and a NIC 150. Additionally, the system 500 may include a storage 140 that stores a computer executable program and at least one set of trainable parameters of an artificial neural network, including a reconfigurable DNN 141, hyperparameters 142, a scheduling criterion 143, forward / backward data 144, a temporary cache 145, a pruning algorithm 146, and a quantization algorithm 147.

[0036] When the computer executable program is executed by at least one computing processor 120, the program causes the at least one processor to perform a training method and a testing method based on at least one trained set of trainable parameters, including: (a) initializing a set of trainable parameters of an artificial neural network, the set of trainable parameters including a set of trainable weights and a set of trainable biases; (b) training the set of trainable parameters with a set of training data; (c) generating a pruning mask based on the trained set of trainable parameters; (d) unwinding the set of trainable parameters; (e) pruning a selected set of trainable parameters based on the pruning mask; and (f) repeating the above steps (b) to (e) a predetermined number of times to generate a set of sparse neural networks with incremental sparsity.

[0037] The above-described embodiments of the present invention can be implemented in any of a number of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether these processors are located in one computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit having one or more processors within an integrated circuit component. However, the processor may be implemented using any suitable form of circuitry.

[0038] Also, embodiments of the invention may be embodied as a method, an example of which is provided. The operations performed as part of this method may be ordered in any suitable manner. Thus, embodiments may be constructed in which operations are performed in an order other than that shown, which may include performing some operations simultaneously even though they are shown as sequential operations in the example embodiment.

[0039] The use of ordinal terms such as "first," "second," and the like in the claims to modify claim elements does not, in and of itself, imply a priority, precedence, or order of one claim element over another or the temporal order in which method actions are performed, but is merely used as a marker to distinguish one claim element having a particular name from another element having the same name (but for purposes of the use of ordinal terms) to distinguish those claim elements.

[0040] Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention.

[0041] Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. 1. A computer-implemented method for training a set of artificial neural networks, the method using at least one processor coupled to a memory storing instructions implementing the method, the instructions, when executed by the at least one processor, perform steps of the method, the steps comprising: (a) initializing a set of trainable parameters of an artificial neural network, the set of trainable parameters including a set of trainable weights and a set of trainable biases, the step further comprising: (b) training the set of trainable parameters using a set of training data; (c) generating a pruning mask based on the trained set of trainable parameters; (d) unwinding the set of trainable parameters; (e) pruning a selected set of trainable parameters based on the pruning mask; (f) repeating the above steps (b) through (e) a predetermined number of times to generate a set of sparse neural networks with incremental sparsity; The method further includes using the artificial neural network to perform equalization for a real-time fiber optic system; the equalization is performed by the set of artificial neural networks receiving an input having residual nonlinear distortion and generating an output in which the residual nonlinear distortion is compensated; The method wherein the output is in the form of a log-likelihood ratio (LLR).

2. The method of claim 1, wherein the log-likelihood ratios are input to a forward error correction decoder following a distribution matcher.

3. A method as described in claim 1 or claim 2, wherein the input having residual nonlinear distortion is received from linear equalization.

4. The training step further comprises: feeding said set of training data into a number of input nodes of said artificial neural network; propagating the set of training data throughout the artificial neural network according to the set of pruning masks and the trainable parameters; generating a set of output values ​​from a plurality of output nodes of the artificial neural network; calculating a set of loss values ​​for the set of training data based on the set of output values; updating the set of trainable parameters based on the set of loss values ​​through backward message passing; repeating the steps above a predetermined number of iterations; especially, a) updating the set of trainable parameters is based on stochastic gradient descent, elastic backpropagation, root mean square propagation, Broyden-Fletcher-Goldfarb-Shanno algorithm, adaptive momentum optimization, adaptive subgradient, adaptive delta, or 3. The method of claim 1 or 2, wherein the step of updating the set of trainable parameters further comprises the step of rounding the trainable weights to quantize values ​​based on powers of 2, sums of powers of 2, powers of 3, and sums of powers of 3.

5. 5. The method of claim 4, wherein the set of loss values ​​is based on mean squared error, mean absolute error, cross entropy, connectionist temporal classification loss, negative log-likelihood, Kullback-Leibler divergence, margin loss, ranking loss, embedding loss, hinge loss, and Huber loss.

6. 1. A deployed system for an artificial neural network, comprising: at least one interface link; at least one computing processor; at least one trained set of trainable parameters of the artificial neural network and at least one memory bank configured for a training method and a testing method, the training method and the testing method using the at least one processor coupled to the at least one memory bank storing instructions implementing the training and testing method, the instructions, when executed by the at least one processor, performing steps of the method, the steps comprising: causing the at least one processor to perform the training method and the testing method based on at least one trained set of the trainable parameters; (a) initializing a set of trainable parameters of an artificial neural network, the set of trainable parameters including a set of trainable weights and a set of trainable biases, the step further comprising: (b) training the set of trainable parameters using a set of training data; (c) generating a pruning mask based on the trained set of trainable parameters; (d) unwinding the set of trainable parameters; (e) pruning a selected set of trainable parameters based on the pruning mask; (f) repeating the above steps (b) through (e) a predetermined number of times to generate a set of sparse neural networks with incremental sparsity; The steps further include using the artificial neural network to perform equalization for a real-time fiber optic system; the equalization is performed by the set of artificial neural networks receiving an input having residual nonlinear distortion and generating an output in which the residual nonlinear distortion is compensated; The system, wherein the output is in the form of a log-likelihood ratio (LLR).

7. The system of claim 6, wherein the log-likelihood ratios are input to a forward error correction decoder following a distribution matcher.

8. The system described in claim 6 or claim 7, wherein the input having residual nonlinear distortion is received from linear equalization.

9. 8. The system of claim 6 or claim 7, wherein the artificial neural network is parallelized and serialized.

10. The system of claim 6 or claim 7, wherein the artificial neural network uses a set of pruning masks to reduce arithmetic operations.

11. 8. The system of claim 6 or claim 7, wherein the artificial neural network uses a set of quantized parameters to reduce arithmetic multiplication operations using powers of 2, sums of powers of 2, powers of 3, and sums of powers of 3.

Images