Modulation method, a method for operating a neural network accelerator, the neural network accelerator and a computer processor comprising the neural network accelerator

Abstract

A modulation method for use in a neural network (NN) accelerator is described in an embodiment. The modulation method comprises modulating features or activations of a neural network to form data representations based on a number of transitions associated with the features or activations within a predefined time window. A method for operating a neural network accelerator, the neural network accelerator and a computer processor comprising the neural network accelerator are also described in embodiments.

Description

[0001] Modulation method, a method for operating a neural network accelerator, the neural network accelerator and a computer processor comprising the neural network accelerator

[0002] Technical Field

[0003] The present disclosure relates to a modulation method, a method for operating a neural network accelerator, the neural network accelerator and a computer processor comprising the neural network accelerator.

[0004] State-of-the-art automated standard cell-based designs with digital data representation achieve very few tens of Tera Operations Per Second per Watt (TOPS / W) in low-cost 40 nm / 28 nm semiconductor manufacturing process nodes, and up to 95 TOPS / W in 5 nm semiconductor manufacturing process node.

[0005] At present, improved energy efficiency at reduced level of design automation is achieved under time-domain and phase-domain data representations. The time-domain and phase-domain data representations simplify MAC units into counters via representations, and these are typically achieved in the phase-domain with a fullcustom design, using pulse width with a partially-custom design, and using pulse density with an automated standard cell design. Energy efficiencies of up to ~30 TOPS / W can be achieved using these representations in 28 nm or older semiconductor manufacturing process nodes. Further energy efficiency improvements require activity reductions in the clock network, peripheral timing circuitry driving counters, and MAC computations.

[0006] On the other hand, in-memory Deep Neural Networks (DNNs) generally have energy efficiencies in the region of around 100 TOPS / W. However, in-memory DNNs typically require substantial custom design effort. In-memory DNNs also suffer from a lack of technology portability and less predictable design cycles.

[0007] It is therefore desirable to provide a modulation method, a method for operating a neural network accelerator, the neural network accelerator and a computer processor comprising the neural network accelerator which address the aforementioned problems and / or provide a useful alternative. Further, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

[0008] Summary

[0009] Aspects of the present application relate to a modulation method, a method for operating a neural network accelerator, the neural network accelerator and a computer processor comprising the neural network accelerator.

[0010] In accordance with a first aspect, there is provided a modulation method for use in a neural network (NN) accelerator, the modulation method comprising modulating features or activations of a neural network to form data representations based on a number of transitions associated with the features or activations within a predefined time window.

[0011] By having data representations based on a number of transitions, the resulting encoded data may be directly input into an asynchronous counter. This modulation approach facilitates the execution of MAC operations by counting signal transitions, thereby eliminating the need for a synchronous clock and reducing an overall activity within the circuitry. This results in an improved energy efficiency. Moreover, this technique mitigates the need to distribute and maintain clock signals across a grid of MAC units or processing elements, which would otherwise lead to elevated power consumption due to synchronization and clock distribution overheads.

[0012] Furthermore, when applied to neural network processing, the disclosed approach exploits the statistical properties of activation distributions. Activations of a neural network often exhibit traditional sparsity, where a significant proportion of activations have zero values. In addition, a substantial portion of the non-zero activation values is of low magnitudes, a phenomenon referred to as pseudo-sparsity. In one embodiment, the number of transitions used in the data representation may be proportional to a magnitude of the activation value. This allows the system to leverage the inherent sparsity and pseudo-sparsity of neural network activations to reduce the average number of transitions per data item. As a result, the activity within the asynchronous counter is decreased on average, leading to further improvements in power efficiency and computational performance. In accordance with a second aspect, there is provided a method for operating a neural network (NN) accelerator comprising modulators and a multiply-accumulate (MAC) unit array, the method comprising: (i) receiving, at the modulators, features or activations as inputs of a neural network (NN); (ii) modulating, at the modulators, the features or the activations to form data representations based on a number of transitions associated with the features or activations within a predefined time window; (iii) receiving, at the MAC unit array, weights from a weights controller and the data representations from the modulators; and (iv) processing, using the MAC unit array, the data representations and the weights to generate MAC outputs.

[0013] The method may comprise multiplexing the data representations formed by the modulators to form a plurality of channels received at each MAC unit of the MAC unit array. The plurality of channels may include two channels, four channels, six channels or eight channels.

[0014] The step (iv) may include accumulating the number of transitions using an asynchronous counter in each MAC unit of the MAC unit array.

[0015] The method may comprise determining, using a scheduler, an optimal sequence in which the weights are to be executed at run time by the MAC unit array. Where determining the optimal sequence in which the weights are to be executed may comprise: quantising the weights and the features or the activations for compatibility with hardware operational parameters of the NN accelerator; determining a pattern for storing the features or the activations to improve access to the features or the activations; and arranging the weights in a temporal sequence to maximise utilisation of the MAC unit array.

[0016] The method may comprise executing an activation function based on the generated MAC outputs to form activation outputs.

[0017] The method may comprise storing the activation outputs using a latch-based register in an on-chip memory.

[0018] The method may comprise providing one of the activation outputs from a MAC unit of the MAC unit array to a readout module at any one time using tristate buffers. The method may comprise optimising the weights using a hardware-aware incremental training process for use in approximate computing by the NN accelerator.

[0019] In accordance with a third aspect, there is provided a neural network (NN) accelerator comprising: modulators adapted to receive features or activations as inputs in a neural network (NN) and to modulate the features or the activations to form data representations based on a number of transitions associated with the features or activations within a predefined time window; a weights controller adapted to provide weights for multiply-accumulate (MAC) operations; and a multiply-accumulate (MAC) unit array adapted to receive the weights from the weights controller and the data representations from the modulators and to process the data representations and the weights to generate MAC outputs.

[0020] The neural network accelerator may comprise a multiplexer at each MAC unit of the MAC unit array adapted to multiplex the data representations formed by the modulators to form a plurality of channels received at each MAC unit of the MAC unit array.

[0021] Each row of the MAC unit array may be connected to two of the modulators and may be adapted to receive six channels of the data representations.

[0022] Each MAC unit of the MAC unit array may comprise an asynchronous counter adapted to accumulate the number of transitions.

[0023] Each MAC unit of the MAC unit array may further comprise a combinational logic circuit adapted to execute an activation function based on the generated MAC outputs to form activation outputs.

[0024] Each MAC unit of the MAC unit array may further comprise a latch-based register adapted to store the activation outputs in an on-chip memory.

[0025] The neural network accelerator may further comprise tristate buffers adapted to provide one of the activation outputs from a MAC unit of the MAC unit array to a readout module at any one time.

[0026] The neural network may include a deep neural network (DNN) or a transformer neural network. The NN accelerator may be a standard cell-based NN accelerator.

[0027] The MAC unit array may be in an asynchronous counter domain, and the modulators and the weights controller may be in a synchronous counter domain.

[0028] The NN accelerator may comprise an instruction scheduling logic circuit adapted to determine an optimal sequence in which the weights are to be executed at run time by the MAC unit array.

[0029] In accordance with a fourth aspect, there is provided a computer processor for use with a neural network, the computer processor comprising: an aforementioned neural network (NN) accelerator; and an instruction scheduling logic circuit adapted to determine an optimal sequence in which the weights are to be executed at run time by the MAC unit array.

[0030] The instruction scheduling logic circuit may be further adapted to: quantise the weights and the features or the activations for compatibility with hardware operational parameters of the NN accelerator; determine a pattern for storing the features or the activations to improve access to the features or the activations; and arrange the weights in a temporal sequence to maximise utilisation of the MAC unit array.

[0031] It should be appreciated that features relating to one aspect may be applicable to the other aspects.

[0032] In accordance with embodiments of the present disclosure, modulation of neural network features or activations to form data representations based on a number of transitions enables simplified accumulation of values using asynchronous counters. This approach is more energy-efficient compared to conventional synchronous counters, as it eliminates the power overhead associated with clock signal generation, distribution, and synchronization.

[0033] As previously described, the modulation technique exploits the statistical characteristics of activation patterns in neural networks - specifically, the prevalence of sparsity (where many activations are zero-valued) and pseudo-sparsity (where many non-zero activations are of low magnitudes). Activation values that are zero or nearzero result in a lower number of transitions which provide decreased switching activity within the asynchronous counters and across the entire MAC unit array. To enable maximum utilisation of the entire MAC unit array, some embodiments incorporate a multiplexer at each MAC unit, the multiplexer being configured to receive a plurality of input channels. This configuration enables dynamic selection among the input channels, allowing each MAC unit to flexibly process data from multiple sources with minimal latency or interruption.

[0034] Furthermore, embodiments of the present disclosure incorporate a scheduler configured to optimise the operational efficiency and throughput of the MAC unit array. The scheduler determines an optimal execution sequence for the weights at runtime, ensuring maximum utilisation of the MAC unit array.

[0035] By intelligently coordinating the delivery of weights in alignment with feature availability and computational demand, the scheduler minimizes idle cycles and prevents underutilisation of processing elements. In conjunction with a multiplexer integrated into each MAC unit, these embodiments enable flexible, low-latency routing of data from a plurality of sources. This enhances data flow efficiency and sustains high-performance computation across the MAC unit array.

[0036] The hardware-aware incremental training process used in an embodiment optimises neural network weights prior to deployment on the accelerator for use in approximate computing. This preparatory step adjusts the weights in accordance with the approximation characteristics of the underlying hardware, enabling the neural network accelerator to perform approximate computing with improved peak energy efficiency while incurring only a small loss in accuracy.

[0037] Brief description of the drawings

[0038] Embodiments will now be described, by way of example only, with reference to the following drawings, in which:

[0039] Figure 1 shows examples of digital data representation using pulse width modulation, dyadic digital pulse modulation and the modulation method in accordance with an embodiment;

[0040] Figure 2 shows a graph to illustrate distribution of 8-bit activation magnitudes across all layers of ResNet-50 in accordance with an embodiment; Figure 3 shows a graph of layer-by-layer distribution of activation magnitudes in ResNet-50 in accordance with an embodiment;

[0041] Figures 4A and 4B show plots for comparing synchronous counter activity and asynchronous counter activity in accordance with an embodiment, where Figure 4A shows a plot of counter activity illustrating the reduction in counter activity due to clock suppression and Figure 4B shows a plot of asynchronous counter activity as a percentage of synchronous counter activity across 20 layers of ResNet-20 with CIFAR- 100;

[0042] Figure 5 is a flowchart of a method for operating a neural network (NN) accelerator in accordance with an embodiment;

[0043] Figure 6 is a flowchart of a method for determining an optimal sequence in which the weights are to be executed in accordance with an embodiment;

[0044] Figure 7 shows a schematic of a top-level architecture of a neural network accelerator in accordance with an embodiment;

[0045] Figure 8 shows a schematic of a multiply-accumulate (MAC) unit of the neural network accelerator of Figure 7 in accordance with an embodiment;

[0046] Figure 9 is an illustration of storage pattern of features on a hardware (e.g. a computer processor) in accordance with an embodiment;

[0047] Figure 10 shows an example of a feature map and weights to illustrate weights scheduling in accordance with an embodiment;

[0048] Figure 11 illustrates an application of features in sequence and their corresponding weights to MAC units, with one channel connected to each MAC unit multiplexer (MUX) in accordance with an embodiment;

[0049] Figure 12 illustrates a timing diagram for the scheduling of MAC unit inputs with two channels connected to each MAC unit MUX in accordance with an embodiment;

[0050] Figure 13 illustrates a timing diagram for the scheduling of MAC unit inputs with four channels connected to each MAC unit MUX in accordance with an embodiment; and Figures 14A, 14B and 14C show a comparison between two modes of operation of the neural network accelerator of Figure 7 in accordance with an embodiment, where Figure 14A shows two flow diagrams of the two modes of operation, Figure 14B shows a plot of relative accuracy for the two modes of operation across four neural networks on various datasets, and Figure 14C shows a plot of peak energy efficiency (TOPS / W) for two modes, along with throughput degradation in Mode 1 , in relation to bit precision.

[0051] Detailed description

[0052] Exemplary embodiments relate to a modulation method, a method for operating a neural network accelerator, the neural network accelerator and a computer processor comprising the neural network accelerator.

[0053] It is appreciated that in the present application, the use of the singular includes the plural unless specifically stated otherwise. It should be noted that, as used in the specification and the appended claims, the singular forms “a”, “an” and “the” include plural referents unless the context clearly dictates otherwise. Further, the use of the term “including”, “comprising”, and “having” as well as other forms, such as “include”, “comprise”, “have” are not considered limiting.

[0054] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0055] As used herein, the term “comprising” or “including” is to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more features, integers, steps or components, or groups thereof. However, in context with the present disclosure, the term “comprising” or “including” also includes “consisting of”. The variations of the word “comprising”, such as “comprise” and “comprises”, and “including”, such as “include” and “includes”, have correspondingly varied meanings.

[0056] In embodiments of the present disclosure, a Deep Neural Network (DNN) accelerator architecture is introduced with a new modulation method. The new modulation method, Dyadic Digital Transition Modulation (DDTM), provides a new data representation for high energy efficiency and automated standard cell design. DDTM reduces activity within the processing elements of the accelerator compared to prior time-domain digital representations, keeping common-case transitions in MAC operations to near-zero by exploiting the inherent pseudo-sparsity associated with the data representation.

[0057] In an implementation in the 40 nm processing node in accordance with an embodiment, the proposed architecture achieves 32.3 TOPS / W at 8-bit equivalent precision, and 122.7 TOPS / W at 4 bits considering all energy contributions of memory and control flow (1 17.9 TOPS / W at 8 bits, and 404.6 TOPS / W considering the macro only). Accordingly, the achieved energy efficiency fills the gap between automated standard cell and in-memory computing architectures in terms of energy efficiency and level of design automation in DNN accelerators.

[0058] The DNN architecture of the present disclosure enables edge or remote sensors to perform real-time data analytics with low latency and low cost, or enhances performance in power-limited systems for applications such as autonomous vehicles, smart surveillance, and loT devices. This architecture also enables on-device artificial intelligence, minimizing wireless communication traffic with the cloud and the response time.

[0059] The present disclosure provides a modulation method for use in a neural network (NN) accelerator. The modulation method comprises modulating features or activations of a neural network to form data representations based on a number of transitions associated with the features or activations within a predefined time window. In the present disclosure, the modulation method is referred to as Dyadic Digital Transition Modulation (DDTM).

[0060] Figure 1 shows an illustration 100 of examples and comparisons of digital data representation using three modulation techniques: Pulse Width Modulation (PWM) 1 10, Dyadic Digital Pulse Modulation (DDPM) 120, and Dyadic Digital Transition Modulation (DDTM) 130 of the present disclosure. In PWM 110, data is represented by the width of the pulse. In the example as shown in Figure 1 , a pulse spans five pulse widths with reference to clock cycle 1 1 1 , indicating a value proportional to the pulse width. This approach requires a synchronous counter to track pulse duration. In DDPM 120, data is encoded using pulse density, where the number of pulses within a fixed clock window represents the value. The example shows five pulses over an 8-clock-cycle window. Like PWM 110, DDPM 120 also requires a synchronous counter to count the pulses. In DDTM 130, data is represented by transition density, with the value determined by the number of signal transitions 131 , illustrated as rising or falling edges in Figure 1 , within a clock window. The example illustrates five transitions 131 over an 8 clock-cycle window. DDTM 130 utilises an asynchronous counter, leading to lower circuit activity and enabling clock suppression. In DDTM 130, transitions 131 are treated as asynchronous events and implementation of DDTM 130 enables the simplification of a MAC unit as a counter. As a result, DDTM 130 enables MAC simplification compared to PWM 1 10 and DDPM 120, both of which depend on synchronous counters. Figure 1 further illustrates two modes of operation for an accelerator employing DDTM: a bit- accurate mode 140 and a DDTM approximate mode 150, assuming the weights are quantised such that quantisation error can be ignored.

[0061] As discussed above, the DDTM 130 encodes information based on the number of transitions within a pre-assigned set of clock cycles, referred to as transition density. DDTM 130 treats transitions as asynchronous events, which is a fundamental departure from the synchronous nature of prior modulations such as DDPM 120. An advantage of employing DDTM 130 is that features encoded using this method can be directly input into an asynchronous counter, retaining the ability to simplify MAC operations to mere counting of transitions. DDTM 130 results in reduced activity within the circuitry and eliminates the need for clock signals across the MAC units or Processing Elements (PE) grid, leading to more energy-efficient processing.

[0062] In one embodiment, the DDTM data representation allows the accelerator to function in two distinct modes. Mode 1 140 focuses on bit-accurate computations, ensuring precise results akin to traditional digital signal processing methods. On the other hand, Mode 2 150 allows computations to be executed in an approximate fashion for higher energy efficiency at the cost of some loss in accuracy. To compensate for the accuracy deficit introduced by approximate computing, an incremental re-training process is carried out. The difference among prior time-domain digital data representations including the proposed DDTM is shown through an example in Figure 1. Details of these two modes are discussed further in relation to Figures 14A to 14C.

[0063] Pseudo Sparsity

[0064] In one embodiment related to the proposed DDTM-based architecture, the activations are represented in a sign and magnitude representation. When analysing the activation magnitudes of the ResNet-50 architecture with 8-bit precision across various images, it is noted from Figure 2 that approximately 50% of the activations are zero, indicating the occurrence of traditional sparsity. This phenomenon of traditional sparsity has been leveraged in various research works to enhance energy efficiency. Moreover, the distribution of non-zero activations reveals that a significant portion of them is close to zero, resulting in what is here called pseudo-sparsity.

[0065] Figure 2 shows a distribution 200 of 8-bit activation magnitudes across all layers of ResNet-50. Traditional sparsity is observed with 49.40% of activations being exactly zero, referred to as sparse activations 210. In addition, a large majority of activations have near-zero values, referred to as pseudo-sparse activations. As the magnitude of activation increases, the frequency of occurrence decreases as shown in Figure 2.

[0066] To further illustrate pseudo-sparsity, the distribution 300 of activation magnitudes in ResNet-50 is depicted layer by layer as shown in relation to Figure 3, where activations are categorized into three bins: activations with 0 transitions 302, activations with transitions in the [1 ,32] range 304, and activations with more than 32 transitions 306. The mean or average for the activations across the ResNet-50 layers is also shown by the data line 308. From Figure 3, it is observed that both traditional sparsity and pseudo-sparsity increase as the network depth increases. A near-zero mean for activations implies a near-zero number of transitions to be counted. In general, neural networks exhibit very low average activation magnitudes. In DDTM, activity is proportional to the magnitude of activations, meaning that common-case activity is near-zero. Except for the first layer, all layers exhibit an average activation magnitude close to zero (< 10 in 8-bit precision), indicating near-zero activity is the common case.

[0067] As discussed above, in the context of DDTM, the number of transitions corresponds to the magnitude of the activation, hence its average signal activity follows the average transition density, which is centred around zero, as shown by the data line 308. In other words, the average signal activity is near-zero, enabling energy-efficient computations with very few transitions in the common case.

[0068] Figures 4A and 4B illustrate plots 400, 410 comparing the activity of synchronous and asynchronous counters, according to one embodiment. The asynchronous counter achieved a 98.4% reduction in overall activity compared to the synchronous counterpart. This activity reduction in DDTM processing elements performing MAC operations via counting is attributed to two factors: clock suppression by asynchronous counters, as shown in Figure 4A, and transition suppression due to pseudo-sparsity, which is enabled by the near-zero common-case value of activations, as shown in Figure 4B. Figure 4A shows a reduction in counter activity for asynchronous counter at approximately 4 times compared to synchronous counter due to clock suppression. Figure 4B presents a plot 410 of asynchronous counter activity as a percentage of synchronous counter activity across 20 layers of ResNet-20 on the CIFAR-100 dataset, averaged over 1 ,000 images. The asynchronous counter shows a 2.49 times reduction in activity at the input layer and up to a 57 times reduction at the 14th layer. On average, the asynchronous counter exhibits approximately 24.6x lower activity, primarily due to pseudo-sparsity, without need for zero activation as in conventional sparsity. As discussed above, the use of DDTM data representation significantly decreases the average activity in asynchronous counters by a factor of 98.4 times compared to their synchronous counterparts. This substantial reduction is attributed to two main factors: the effect of clock suppression which contributes to a 4 times reduction as shown in relation to Figure 4A, and the inherent pseudo-sparsity within DDTM encoding as shown in relation to Figure 4B which accounts for an average activity reduction of 24.6 times (and up to 57 times across the layers of the considered neural networks). The underlying reason for this efficiency is that activations in DDTM encoding tend to cluster around zero, as was showed in relation to Figure 2 and Figure 3. Consequently, DDTM-encoded inputs typically exhibit near-zero transitions, leading to minimal activity and energy since the number of transitions directly correlates with the activation magnitude.

[0069] The overall architectural energy efficiency benefits significantly from the simplicity of the processing elements (e.g. MAC units implemented by counters) enabled by DDTM and the pseudo-sparsity property of its encoding. These factors collectively reduce the effective switched capacitance, thus reducing energy in both global data distribution and local computations within the MAC units. This highlights how DDTM's data representation can reduce the energy consumption of deep neural network accelerators by exploiting the natural activation patterns and their statistical properties in real-world neural networks.

[0070] Figure 5 is a flowchart of a method 500 for operating a neural network (NN) accelerator in accordance with an embodiment. The NN accelerator comprises modulators and a multiply-accumulate (MAC) array and the method comprises the following steps. In step 502, the method comprises receiving, at the modulators, features or activations as inputs of a neural network (NN). Features or activations may be spatially encoded, or encoded using position-dependent information. The modulator may be a component or a circuit that transforms input signals, such as features or activations in a neural network, into a different format, often optimised for processing efficiency, power consumption, or bandwidth. A modulator may alternatively be referred to as an encoder, data representation generator or a preprocessing module.

[0071] In step 504, the method comprises modulating, at the modulators, the features or the activations to form data representations based on a number of transitions associated with the features or activations within a predefined time window. In one embodiment, the data representations or modulation method employed by the modulators may be an implementation of the aforementioned DDTM technique, providing energy efficiency by enabling clock suppression and leveraging the pseudo-sparsity characteristics of neural network features or activations. This approach minimises counter activity within the circuitry, thereby reducing overall power consumption.

[0072] In step 506, the method comprises receiving, at the MAC unit array, weights from a weights controller and the data representations from the modulators. The weights controller may be configured to manage the retrieval, buffering, and delivery of weight data to the MAC units in accordance with a neural network's execution schedule. The weights controller may access weight values from on-chip memory, off-chip memory, or a dedicated local latch memory, and may include address generation logic, scheduling logic, and data alignment circuitry to ensure the timely and efficient distribution of weights to each MAC unit. The weights controller may synchronize weights delivery with corresponding data representations received from the modulators, ensuring proper pairing of data representations of features or activations with weights for accurate MAC operations.

[0073] In relation to weight scheduling, the method 500 may further comprise determining, using a scheduler, an optimal sequence in which the weights are to be executed at run time by the MAC unit array.

[0074] Figure 6 is a flowchart of a method 600 for determining an optimal sequence in which the weights are to be executed in accordance with an embodiment. In the embodiment, determining the optimal sequence in which the weights are to be executed may comprise: quantising the weights and the features or the activations for compatibility with hardware operational parameters of the NN accelerator in step 602, determining a pattern for storing the features or the activations to improve access to the features or the activations in step 604, and arranging the weights in a temporal sequence to maximise utilisation of the MAC unit array in step 606.

[0075] The method 500 may further comprise optimising the weights using a hardware-aware incremental training process for use in approximate computing by the NN accelerator. The hardware-aware incremental training process may be a preparatory step ensuring weights are optimally adjusted for the hardware’s approximation capabilities. The training process may take into account hardware-specific constraints, such as switching activity, or energy consumption profiles of the MAC units. This relates to a dual-mode architecture which will be discussed in relation to Figures 14A to 14C.

[0076] In step 508, the method comprises processing, using the MAC unit array, the data representations and the weights to generate MAC outputs.

[0077] The step 508 may further include accumulating the number of transitions using an asynchronous counter in each MAC unit of the MAC unit array. This may correspond to an implementation of the aforementioned DDTM in the MAC units, in relation to step 504, wherein data representations are generated based on the number of transitions.

[0078] In one embodiment, for efficient utilisation of the MAC units, the data representations generated by the modulators may be scheduled and routed in a time-multiplexed manner. The method 500 may further comprise multiplexing the data representations formed by the modulators to form a plurality of channels received at each MAC unit of the MAC unit array. The plurality of channels may include six channels. This allows each MAC unit to sequentially receive and process multiple data streams.

[0079] The method 500 may further comprise executing an activation function based on the generated MAC outputs to form activation outputs. The activation function may be a Rectified Linear Unit (ReLU) and implemented in hardware or software.

[0080] The method 500 may further comprise storing the activation outputs using a latchbased register in an on-chip memory. The method 500 may further comprise providing one of the activation outputs from a MAC unit of the MAC unit array to a readout module at any one time using tristate buffers.

[0081] In various embodiments of the described method, the neural network includes a deep neural network (DNN). Although the present embodiment is described in relation to a DNN, it should be appreciated that other forms of neural network may be applicable.

[0082] Top Architecture

[0083] Figure 7 shows a schematic of a top-level architecture of the proposed DDTM accelerator 700, in accordance with an embodiment. The architecture includes a processing element (PE) array (or MAC unit array 710) that treats feature or weight represented by transitions as asynchronous signals, enabling clock-less operation within the core compute path. As a result, the MAC unit array 710 operates in a separate asynchronous domain, achieving clock suppression for improved energy efficiency. The DDTM accelerator 700 of the present embodiment enables fully automated processing process and is greatly simplified by the use of pre-determined timing at synchronous boundaries.

[0084] In this embodiment, features, which are inputs to the first layer of a neural network, and activations, which are inputs to the second-to-last layers of the neural network, are stored in local latch memory of activations controller 720, which is adapted to prepare the inputs for the DDTM modulators 730. On the other side, weights are stored in local latch memory of weights controller 740. This controller 740 is adapted to process the weights to produce signals that drive the MAC unit array 710. The MAC unit array 710 in the present embodiment includes a grid of 56x48 processing elements (PEs) or MAC units, operating as an asynchronous domain within the accelerator's overall design and hence is clock-less.

[0085] This asynchronous domain interfaces with the synchronous parts of the architecture 700, receiving inputs from and sending outputs to the synchronous boundary. This setup significantly eases the process of fully automated design by ensuring that the timing for the inputs and the outputs to and from the asynchronous domain is predefined, which may be managed through standard automated place and route tools, such as Cadence Innovus and Synopsys ICC2. The local memories within activations and weights controller are fed by dedicated weights SRAM 750 and activations SRAM 760 (288 KB and 392 KB respectively) as highlighted in Figure 7.

[0086] Figure 8 shows a schematic of a multiply-accumulate (MAC) unit 800 of the neural network accelerator 700 of Figure 7 in accordance with an embodiment.

[0087] In the embodiment, each PE or MAC unit in the array in Figure 8 executes MAC operations by accumulating the transitions in the features, which may be counted via an 11 -bit asynchronous up-down counter. To enhance throughput, every row in the PE array may be equipped with two DDTM modulators and is capable of receiving six channels CH1...CH6 of DDTM modulated transitions: two from the current row, and two from each of the adjacent previous row and next row. The six channels may be fed to a multiplexer 810 of each MAC unit 800 and the channels may be multiplexed from three rows for better utilisation. The weight sign sets the up / down control signal in the counters sharing the same weight for correct addition / subtraction in the accumulation process.

[0088] Additionally, each PE may incorporate combinational logic 860 to execute the ReLU activation function 830. Each PE may also feature a memory-mapped 8-bit latch-based register 840 to store the MAC output in an on-chip memory for simultaneous readoutprocessing. The present setup of the MAC unit 800 allows serial readout of the MAC results while the asynchronous counter 820 continues its computation, thereby masking completely the time needed for the readout as it is completely overlapped and simultaneous to the MAC computation and accumulation process.

[0089] To minimise the power consumption associated with the output readout, the PE may include tristate buffers 850 before the output. This ensures that only one PE drives the output line to the readout module at any given time. By reducing the number of parallel wires per row needed for readout, wire capacitance and routing resources are minimized, leading to a reduction in power requirements and congestion avoidance in the routing process. This integration of components within each PE or MAC unit 800 contributes to the overall efficiency and performance of the accelerator architecture.

[0090] To summarise, as shown in relation to Figures 7 and 8 above, a neural network (NN) accelerator is provided in an embodiment. The neural network accelerator comprises modulators adapted to receive features or activations as inputs in a neural network (NN) and to modulate the features or the activations to form data representations based on a number of transitions associated with the features or activations within a predefined time window; a weights controller adapted to provide weights for multiply-accumulate (MAC) operations; and a multiply-accumulate (MAC) unit array adapted to receive the weights from the weights controller and the data representations from the modulators and to process the data representations and the weights to generate MAC outputs.

[0091] The neural network accelerator may further comprise a multiplexer at each MAC unit of the MAC unit array adapted to multiplex the data representations formed by the modulators to form a plurality of channels received at each MAC unit of the MAC unit array. Each row of the MAC unit array may be connected to two of the modulators and is adapted to receive six channels of the data representations. These six channels may include: two channels from the modulators associated with the current row, and two channels each from the modulators of the adjacent previous and next rows. This configuration enables enhanced throughput by increasing the volume of input data processed concurrently within each MAC unit. Moreover, by aggregating modulated transitions across neighboring rows, the architecture facilitates efficient utilisation of computational resources.

[0092] Each MAC unit of the MAC unit array may comprise an asynchronous counter adapted to accumulate the number of transitions. In addition, each MAC unit of the MAC unit array may further comprise a combinational logic circuit adapted to execute an activation function based on the generated MAC outputs to form activation outputs. Each MAC unit of the MAC unit array may further comprise a latch-based register adapted to store the activation outputs in an on-chip memory. This architecture enables simultaneous readout and processing, allowing the system to perform serial readout of the MAC results while the asynchronous counter continues its computation without interruption. As a result, the readout latency is fully masked, achieving overlap between the computation, accumulation, and output stages, which enhances throughput and overall processing efficiency.

[0093] The neural network accelerator may further comprise tristate buffers adapted to provide one of the activation outputs from a MAC unit of the MAC unit array to a readout module at any one time. This minimises the power consumption associated with the output readout. By reducing the number of parallel output wires per row, the design lowers wire capacitance and conserves routing resources, thereby decreasing dynamic power requirement and avoiding routing congestion.

[0094] The NN accelerator may be a standard cell-based NN accelerator. The NN accelerator may be designed in fully-automated standard cell-based fashion for easy and immediate adoption system integration and technology scalability.

[0095] The MAC unit array may be in an asynchronous counter domain, and the modulators and the weights controller may be in a synchronous counter domain. In this configuration, the synchronous domain manages the scheduling and control of weights and activations, ensuring coordinated data flow and timing, whereas the asynchronous domain is optimised for lower power consumption by eliminating the need for a global clock.

[0096] The present disclosure further provides a computer processor for use with a neural network (e.g. a deep neural network). The computer processor comprises an aforementioned neural network (NN) accelerator and an instruction scheduling logic circuit adapted to determine an optimal sequence in which the weights are to be executed at run time by the MAC unit array.

[0097] The instruction scheduling logic circuit may be further adapted to: quantise the weights and the features or the activations for compatibility with hardware operational parameters of the NN accelerator 602, determine a pattern for storing the features or the activations to improve access to the features or the activations 604, and arrange the weights in a temporal sequence to maximise utilisation of the MAC unit array 606.

[0098] Scheduler

[0099] The scheduler in various embodiments plays a crucial role in optimising the utilisation of the MAC unit array over time. It does so by determining the sequence in which weights are to be executed, generating addresses for the corresponding inputs, which are identified by the offsets AX and AY in the two memories placed along the x and the y dimension of the physical array. In one embodiment, the scheduler initially quantises both weights and features to ensure compatibility with the hardware operational parameters. Then, the scheduler is adapted to determine an efficient pattern for feature storage within the system memory banks to facilitate immediate access during computation. The next step in the present embodiment involves arranging the weights in a temporal sequence that guarantees maximum utilisation of the array, given the time-domain and hence irregular representation of each weight. This organisation allows the MAC units to operate at their highest capacity, reducing idle times and enhancing their overall throughput. The scheduler converts these properly sequenced weights into 4-bit instruction codes specifically tailored for the PEs, translating the schedule into low-level control operations for streamlined computation flow and maximum PE array utilisation.

[0100] Feature map storage pattern

[0101] Figure 9 is an illustration of storage pattern of features on a hardware (e.g. a computer processor) in accordance with an embodiment. The storage pattern of input image feature map 910 is designed for efficient processing of features at the MAC unit array. The input image feature is shown with a kernel size (KS) 911 of 5. The input feature data is distributed across four memory banks, also referred to as input feature memory banks. Each memory bank stores a portion of the feature data in a structured and interleaved manner to facilitate parallel access during computation.

[0102] In the embodiment, to ensure a regular flow of features to the array of MAC units, four feature banks are utilised with two banks being active at any given time. The two active banks are shown in Figure 9 as Bank 0 920 and Bank 1 930. The size of these banks is determined by the largest kernel size (KS) that the DNN accelerator can accommodate. For any given kernel size KS, the dimensions of each bank are set to KS columns and 56+KS rows. Figure 9 demonstrates the method of storing features within these banks on the hardware. Specifically, the first (56+KS) rows and KS columns patch (taking KS = 5 as an example) is placed in the first bank, while the subsequent patch of the same size is allocated to the second bank. The following patch is then stored in the third bank, and this sequence is repeated, moving first horizontally and then vertically, to span the entire feature map.

[0103] Weight ordering

[0104] To illustrate how the scheduler performs weight ordering, a kernel size of 3 is assumed in the example embodiment as shown in relation to Figure 10 based on a dummy feature map. Figure 10 shows an example of feature map 1000 and weights to illustrate weights scheduling in accordance with an embodiment. In the first patch 1010, a region spanning twice the kernel size (2xKS) in column width is loaded into the activations local memory for computation of one patch (e.g. if KS = 3, then a first patch having a column width of 6 columns will be loaded). The row height of each patch corresponds to the number of rows in the grid plus the kernel size (KS) (e.g. if a number of rows in the MAC grid is 27 and KS = 3, then (27+3) rows and 6 columns for the first patch will be loaded). The second horizontal patch 1020 is indicated with a dotted outline to show the progression of patch-wise computation. A weight map 1030 with a kernel size (KS) of 3 is depicted, where weights are represented using varying shades to indicate nonzero values. One specific weight, W33, is shown as a zero-weight element 1031 .

[0105] Figure 1 1 shows an illustration 1 100 of an application of features in sequence and their corresponding weights to MAC units, with one channel 1 110 connected to each MAC unit multiplexer (MUX) in accordance with an embodiment.

[0106] As an embodiment for baseline solution as shown in relation to Figure 11 , features in a row are applied in sequence and corresponding weights are applied to the MAC unit when the feature is active. This leads to only one channel 1 110 to MAC unit MUX.

[0107] In this case, a MAC unit is only active for <50% of the cycles. Empty cycles 1 120 lead to partial MAC unit array utilisation and hence degraded throughput compared to the maximum potentially sustainable. This provides scope to interleave weights in empty slots at the expense of an increased number of channels fed to the input multiplexer in each MAC unit.

[0108] Figure 12 illustrates a timing diagram 1200 for the scheduling of MAC unit inputs with two channels 1210 per multiplexer (MUX) in accordance with an embodiment. To pack weights more efficiently, two features in a row (X, X+KS) are applied in this case, leading to two channels 1210 being fed to the MUX of each MAC unit 800. The activity of each MAC unit 800 would depend on the magnitude of the weight which is represented through pulse-width modulation. Representation of the weight in this case is in the form of a digital pulse having a duration in clock cycles that is proportional to magnitude. Since the overall frame length 1220 is determined by the highest magnitude weight, for example W13 in Figure 12, MAC units with smaller magnitude weights 1230 remain idle. Figure 13 illustrates a timing diagram 1300 for the scheduling of MAC unit inputs with four channels 1310 per MUX in accordance with an embodiment. To further boost the array utilisation, weights from the next row 1320 are scheduled to keep MAC units more utilised in free cycles, as highlighted in Figure 13. To facilitate the use of next row weights, four channels 1310 per MAC unit are utilised. Beyond this example embodiment, the number of adjacent rows feeding weights to the considered row can be further increased.

[0109] A similar method can be used to generate 6 channels as shown in the embodiment illustrated by Figure 8 where weights are borrowed from previous and next rows in idle item of current row. It is worth noting that, the frame length is defined by the maximum magnitude weight being computed from the above example illustrated by Figure 12. However, borrowing weights from adjacent rows makes the frame length equal to the average of weights for a kernel rather than the highest, so that the remaining balance in the current row can be transferred to the adjacent row.

[0110] In the embodiment, after organising the weights temporally, the weight sequence may be translated into instructions that use the active feature channel as current reference. This conversion facilitates the execution of accurate MAC operations using DDTM. Concurrently, memory offsets in the two dimensions AX and AY may be calculated for the memory banks storing the features to define the subsequent feature to be computed, enforcing temporal coordination between weight delivery and the feature processing.

[0111] Implementation Details and Measurement Results

[0112] The performance of the proposed architecture is detailed in a Table 1 for two implementations of embodiments in different CMOS technology nodes, the 40 nm node and the 28 nm node.

[0113] Table 1 : Measurements of 40 nm and 28 nm implementation of proposed architecture

[0114] Dual-mode architecture

[0115] In an embodiment, the proposed architecture operates in two distinct modes to balance computational accuracy and energy efficiency. Mode 1 is designated for accurate computation, ensuring that the calculations adhere closely to the expected baseline accuracy as would be achieved in software and floating point representation. On the other hand, Mode 2 is approximate to enhance energy efficiency at the expense of a slight accuracy degradation. In an embodiment, Mode 2 can be implemented in a similar approach taken with DDPMnet. To mitigate the latter while reducing energy, Mode 2 requires a hardware-aware incremental training process. This preparatory step ensures that the weights are optimally adjusted for the hardware's approximation capabilities before deployment on the accelerator, as illustrated Figure 14A.

[0116] Figures 14A, 14B and 14C show a comparison between two modes of operation of the neural network accelerator of Figure 7 in accordance with an embodiment.

[0117] Figure 14A shows two flow diagrams of the two modes 1400, 1410 of operation. In Mode 1 1400, a deep neural network (DNN) with pre-trained weights is directly used for weight scheduling and subsequent hardware deployment. In contrast, Mode 2 1410 includes an additional step: hardware-oriented incremental training. This step is performed before the weight scheduling phase, enabling further optimisation of the network specifically for the target hardware.

[0118] In relation to implementations of the embodiment described herein, Figure 14B shows a plot 1420 of relative accuracy for the two modes of operation across four neural networks (LeNet-5, ResNet-20, SqueezeNet v1.0, ResNet-50) on various datasets (MINIST, CIFAR-100, ImageNet). The relative accuracy is benchmarked against a software baseline 1421 . Figure 14C shows a plot 1430 of peak energy efficiency (TOPS / W) for the two modes, along with throughput degradation in Mode 1 1400, in relation to bit precision. Mode 1 1400 takes more cycles to compute as compared to Mode 2 1410 to maintain multiply-accumulate (MAC) accuracy. At 8-bit precision, Mode 2 1410 demonstrates a 13.75x improvement in peak energy efficiency compared to Mode 1 .

[0119] Figures 14B and 14C demonstrate that Mode 2 1410 significantly improves the peak energy efficiency, while incurring into a small accuracy loss (for example, -1 .27% loss under ResNet-50 on the ImageNet dataset, -0.09% loss under SqueezeNet v1.0 on CIFAR-100 dataset and -0.31% loss under ResNet-20 on CIFAR-100 dataset). This improvement is attributed to the reduced number of cycles required for MAC operations in Mode 2 1410 compared to Mode 1 1400, where maintaining the baseline software accuracy necessitates more computational cycles.

[0120] Benchmarking on different workloads

[0121] Table 2 and Table 3 benchmark the architecture implemented in two different CMOS technology nodes under different datasets (MNIST, CIFAR-100 and ImageNet) and neural networks (LeNet-5, ResNet-20, ResNet-50 and SqueezeNet v1 .0). Table 2 shows average and peak energy efficiency up to 723.13 TOPS / W at 2b for the 28 nm technology node, which is reduced down to 94.35 TOPS / W under ResNet-50 on ImageNet at 0.6 V. For the 40 nm technology node as shown in relation to Table 3, peak and average TOPS / W follows the same trend as in the 28 nm node from 368.9 at 2b and 32.3 at 8b.

[0122] Table 2: Benchmarking for 28 nm and Mode 2

[0123] Table 3: Benchmarking for 40 nm and Mode 2

[0124] Applications

[0125] Distributed vision systems rely on cameras with tightly-constrained power budgets and sufficient flexibility to make sense of the scene regardless of their context and complexity. Such cameras are generally self-powered (e.g., battery-powered), and require energy-efficient Al acceleration to avoid power-costly wireless transmission of raw frames to the cloud, network-level data deluge, and reduce scene analysis latency.

[0126] An embodiment of present disclosure presents a DNN accelerator capable of executing a given Al algorithms and neural network (e.g., convolutional neural network) for applications such as face recognition, crowd monitoring, and vehicle number plate detection at much lower cost and prolonged battery life, while not sacrificing latency.

[0127] Conclusion

[0128] The neural network accelerator of the present embodiments is fully compatible with conventional automated standard cell-based design flows to reduce design cost while achieving the energy efficiency of customized in-memory computing architectures.

[0129] Compared to prior art, embodiments of the present disclosure achieve the state-of-the- art energy efficiency of 122.7 (32.3) TOPS / W in 4 bit (8 bit) using SqueezeNet v1.0 (ResNet-50). In absolute terms, this represents a 4.4X (2.6-31 .1 X) improvement at 4 (8) bit over prior time-domain data representations, and 1 11 ,5X (2.9X) in 4 bit (8 bit) over conventional digital representation in comparable technologies. In another embodiment, energy efficiency is also improved by 1.28X at 4-bit over the 5-nm design, and have comparable efficiency at 8-bit precision. As key finding, macro energy efficiency reaches (or exceeds) the 100 TOPS / W range and is hence equivalent to recent state- of-the-art in-memory DNN accelerator. Table 4 provides a summary of key features in various embodiments of the present disclosure along with their respective advantages.

[0130] Table 4 Features and Corresponding Advantages

[0131] Alternative embodiments may include variations in the number of channels received at each MAC unit of the MAC unit array. While implementations of certain embodiment utilise six channels, other configurations may include one, two, four, six, eight, or more channels received at each MAC unit of the MAC unit array.

[0132] Alternative embodiments of the neural network accelerator may support various types of neural networks. The neural network may comprise a deep neural networks (DNN), a convolutional neural networks (CNN), a recurrent neural networks (RNN), a transformer neural network or any other artificial intelligence model.

[0133] In alternative embodiments, the activation function may include a sigmoid, or hyperbolic tangent (tanh) function depending on an architecture of the neural network used.

[0134] In alternative embodiments of the computer processor for use with a neural network, the instruction scheduling logic circuit may be: (i) integrated within the NN accelerator as an internal component thereof; or (II) implemented as a separate component within the computer processor and operatively coupled to the NN accelerator.

[0135] Although only certain embodiments of the present invention have been described in detail, many variations are possible in accordance with the appended claims. For example, features described in relation to one embodiment may be incorporated into one or more other embodiments and vice versa.

Claims

Claims1 . A modulation method for use in a neural network (NN) accelerator, the modulation method comprising modulating features or activations of a neural network to form data representations based on a number of transitions associated with the features or activations within a predefined time window.

2. A method for operating a neural network (NN) accelerator comprising modulators and a multiply-accumulate (MAC) unit array, the method comprising:(i) receiving, at the modulators, features or activations as inputs of a neural network (NN);(ii) modulating, at the modulators, the features or the activations to form data representations based on a number of transitions associated with the features or activations within a predefined time window;(iii) receiving, at the MAC unit array, weights from a weights controller and the data representations from the modulators; and(iv) processing, using the MAC unit array, the data representations and the weights to generate MAC outputs.

3. The method of claim 2, further comprising multiplexing the data representations formed by the modulators to form a plurality of channels received at each MAC unit of the MAC unit array.

4. The method of claim 3, wherein the plurality of channels includes two channels, four channels, six channels or eight channels.

5. The method of any one of claims 2 to 4, wherein the step (iv) includes accumulating the number of transitions using an asynchronous counter in each MAC unit of the MAC unit array.

6. The method of any one of claims 2 to 5, further comprising: determining, using a scheduler, an optimal sequence in which the weights are to be executed at run time by the MAC unit array.

7. The method of claim 6, wherein determining the optimal sequence in which the weights are to be executed comprises: quantising the weights and the features or the activations for compatibility with hardware operational parameters of the NN accelerator; determining a pattern for storing the features or the activations to improve access to the features or the activations; and arranging the weights in a temporal sequence to maximise utilisation of the MAC unit array.

8. The method of any one of claims 2 to 7 , further comprising executing an activation function based on the generated MAC outputs to form activation outputs.

9. The method of claim 8, further comprising storing the activation outputs using a latch-based register in an on-chip memory.

10. The method of claim 8 or claim 9, further comprising providing one of the activation outputs from a MAC unit of the MAC unit array to a readout module at any one time using tristate buffers.1 1 . The method of any one of claims 2 to 10, further comprising optimising the weights using a hardware-aware incremental training process for use in approximate computing by the NN accelerator.

12. The method of any one of claims 2 to 11 , wherein the neural network includes a deep neural network (DNN) or a transformer neural network.

13. A neural network (NN) accelerator comprising: modulators adapted to receive features or activations as inputs in a neural network (NN) and to modulate the features or the activations to form data representations based on a number of transitions associated with the features or activations within a predefined time window; a weights controller adapted to provide weights for multiply-accumulate (MAC) operations; anda multiply-accumulate (MAC) unit array adapted to receive the weights from the weights controller and the data representations from the modulators and to process the data representations and the weights to generate MAC outputs.

14. The neural network accelerator of claim 13, further comprising a multiplexer at each MAC unit of the MAC unit array adapted to multiplex the data representations formed by the modulators to form a plurality of channels received at each MAC unit of the MAC unit array.

15. The neural network accelerator of claim 14, wherein each row of the MAC unit array is connected to two of the modulators and is adapted to receive six channels of the data representations.

16. The neural network accelerator of any one of claims 13 to 15, wherein each MAC unit of the MAC unit array comprises an asynchronous counter adapted to accumulate the number of transitions.

17. The neural network accelerator of claim 16, wherein each MAC unit of the MAC unit array further comprises a combinational logic circuit adapted to execute an activation function based on the generated MAC outputs to form activation outputs.

18. The neural network accelerator of claim 17, wherein each MAC unit of the MAC unit array further comprises a latch-based register adapted to store the activation outputs in an on-chip memory.

19. The neural network accelerator of claim 17 or claim 18, further comprises tristate buffers adapted to provide one of the activation outputs from a MAC unit of the MAC unit array to a readout module at any one time.

20. The neural network accelerator of any one of claims 13 to 19, wherein the neural network includes a deep neural network (DNN) or a transformer neural network.

21. The neural network accelerator of any one of claims 13 to 20, wherein the NN accelerator is a standard cell-based NN accelerator.

22. The neural network accelerator of any one of claims 13 to 21 , wherein the MAC unit array is in an asynchronous counter domain, and the modulators and the weights controller are in a synchronous counter domain.

23. The neural network accelerator of any one of claims 13 to 21 , further comprising an instruction scheduling logic circuit adapted to determine an optimal sequence in which the weights are to be executed at run time by the MAC unit array.

24. A computer processor for use with a neural network, the computer processor comprising: the neural network (NN) accelerator of any one of claims 13 to 22; and an instruction scheduling logic circuit adapted to determine an optimal sequence in which the weights are to be executed at run time by the MAC unit array.

25. The computer processor of claim 24, wherein the instruction scheduling logic circuit is further adapted to: quantise the weights and the features or the activations for compatibility with hardware operational parameters of the NN accelerator; determine a pattern for storing the features or the activations to improve access to the features or the activations; and arrange the weights in a temporal sequence to maximise utilisation of the MAC unit array.

Images