Crossbar Arrays for Computation in Memory-Augmented Neural Networks

Abstract

In a hardware implementation method for operating a neural network system, a neural network system is provided that includes a controller, a memory, and an interface connecting the controller to the memory, where the controller includes a processing unit configured to execute a neural network, and the memory includes a neuromorphic memory device having a crossbar array structure including input and output lines interconnected at junctions via electronic devices. The electronic devices of the neuromorphic memory device are programmed to incrementally change state by coupling write signals to the input lines based on write instructions received from the controller and write vectors generated by the interface. Data is retrieved from the neuromorphic memory device according to a multiply-and-accumulate operation by coupling read signals to one or more of the input lines of the neuromorphic memory device based on read instructions from the controller and read vectors generated by the interface.

Description

[Technical field]

[0001] The present invention relates generally to the field of neural network systems, and more particularly to a memory-augmented neural network system that uses a neuromorphic memory device having a crossbar array structure as an external memory. [Background technology]

[0002] Machine learning often relies on artificial neural networks (ANNs), which are computational models inspired by biological neural networks in human or animal brains. Such systems learn tasks incrementally and autonomously by example and have been successfully applied, for example, to speech recognition, text processing, and computer vision.

[0003] Neural networks include feedforward neural networks such as multi-layer perceptrons, deep neural networks, and convolutional neural networks. Neural networks are typically implemented in software. However, neural networks can also be implemented in hardware, for example as resistive processing units (crossbar array structures) or optical neuromorphic systems.

[0004] Various types of hardware systems can be used to execute neural networks. Particularly advantageous is the use of memory-augmented neural network systems. The basic idea of ​​memory-augmented neural networks is to augment neural networks with external memory. Memory-augmented neural networks (MANNs) benefit from a powerful architecture that combines the benefits of neural network data processing with those of persistent storage. Summary of the Invention

[0005] An aspect of an embodiment of the present invention discloses a hardware implementation method and a neural network system. A neural network system is provided that includes a controller, a memory, and an interface connecting the controller to the memory, where the controller comprises a processing unit configured to execute a neural network, and the memory comprises a neuromorphic memory device having a crossbar array structure including input and output lines interconnected at junctions via an electronic device. The electronic device of the neuromorphic memory device is programmed to incrementally change a state of the electronic device by coupling a write signal to one or more of the input lines based on (i) a write instruction received from the controller and (ii) a write vector generated by the interface in accordance with the write instruction. Data is retrieved from the neuromorphic memory device in accordance with a multiply-and-accumulate operation by coupling a read signal to one or more of the input lines of the neuromorphic memory device in accordance with (i) a read instruction from the controller and (ii) a read vector generated by the interface in accordance with the read instruction. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is a block diagram that generally illustrates selected components of a neural network system, in accordance with one embodiment of the present invention. [Figure 2A] FIG. 2 is a schematic representation of a memristive crossbar array of neuromorphic memory devices of the system of FIG. 1 in accordance with one embodiment of the present invention. [Figure 2B] FIG. 2 is a schematic diagram of a neuromorphic memory device of the neural network system of FIG. 1 according to one embodiment of the present invention, the memory device comprising a pulse width modulation circuit and a readout circuit. [Diagram 3] 2 is a flowchart illustrating high-level steps of a method of operating the neural network system of FIG. 1 in accordance with one embodiment of the present invention. [Figure 4] 4 is a flow chart illustrating the substeps of a write memory access according to one embodiment of the present invention. [Diagram 5] 1 is a flowchart illustrating how an electronic device of a neuromorphic memory device is programmed to incrementally change the state of the electronic device according to one embodiment of the present invention, where incremental values ​​are first accumulated in an additional memory before writing the accumulated values ​​to the electronic device. [Figure 6] FIG. 2 is a block diagram illustrating components of a computer system that may be included as part of the neural network system of FIG. 1 in accordance with an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Embodiments of the present invention recognize that in a typical neural network system, information can only be stored in the internal state (e.g., if the network is composed of long short-term memory (LSTM) units, information must be stored in the state of the units). Embodiments of the present invention recognize that this has the drawback that the stored information can only be stored for a limited time, potentially being overwritten when new input arrives at the units. Embodiments of the present invention further recognize that if the same network is augmented with an external memory, as in a memory-augmented neural network system, the network can leverage this storage and use it to retain information for a longer period of time.

[0008] Memory-augmented neural networks have several advantages: Embodiments of the present invention use neuromorphic devices (eg, crossbar array structures) as external memory for the neural network system.

[0009] To extend a neural network with an external memory, an interface must be present. This interface may, for example, include a read head and a write head (there may be multiple read heads and multiple write heads). The purpose of these heads is to retrieve data from the external memory and write information to the external memory.

[0010] A controller (also called a controller network) refers to a unit that includes a processing unit that executes a neural network. This controller can be considered as a unit that executes a normal neural network that receives some input and creates some output. In the absence of external memory, this is equivalent to a standard neural network used to solve any particular task. In an embodiment, the controller is coupled to a feature extraction unit (not shown) that is used to extract features from input data (e.g., images), whereby the extracted features are provided to the controller, which executes the neural network based on such extracted features (whether the neural network is for training or inference purposes). Any suitable feature extraction scheme can be contemplated, for example relying on principal component analysis or convolutional neural networks. If necessary, other pre-processing entities can be involved. Now, such pre-processing entities can be integrated into the controller in some cases. Thus, various architectures can be contemplated for the controller network (e.g., the controller can essentially be intended to execute a simple forward propagation neural network or a complex recurrent neural network with multiple layers). For simplicity, for example, this network can be viewed as a single layer of recurrent LSTM units.

[0011] In general, the structure of the read and write heads depends on the chosen type of memory-augmented neural network system. Many variants are available. The read and write heads can be represented by shallow neural networks, for example. The read head receives an input from the controller. The structure of this input also depends on the type of memory-augmented neural network. In a simple scenario, the input to the read head is exactly the same as the input to the controller network itself. Based on this input, the read head calculates a read vector w r Create a vector w r is used to access the external memory M[i,j].

number

[0012] In a very simple case, the input to the read head is passed through a shallow neural network and the result of this operation is the read vector w r There are more complex read head architectures, including for example attention mechanisms. Essentially, the read vector is used to access external memory, regardless of how the read vector was generated. The result r (from accessing memory) is returned to the controller network, which can then use the result r to create an output for the originally given input.

[0013] A write head operates in a similar manner to a read head. It receives input from the controller network, and in the simplest case, this input is simply the input to the controller network itself. However, in contrast to a read head, the task of a write head is not to read information, but to store information in external memory. In essence, a write head receives a write vector w, which is used to write to memory. wWe achieve this by generating vectors w and v. Again, the exact steps of how to generate such vectors are specific to the type of memory-augmented neural network. In the simple case, the input to the write head is passed through two shallow neural networks, and the result is w w and v. Then, M[i,j] = M[i,j](1-w w [i]e T [j])+w w [i]v T [j] The memory contents are modified according to

[0014] An embodiment of the present invention relies on an external memory having a crossbar array structure and using electronic devices such as memristive devices, e.g., phase-change memory (PCM) cells. This type of memory device cannot be used as simply as a conventional memory (e.g., SRAM or DRAM). It can be seen that the memristive cells prefer a particular way of writing to the memory, i.e., a value can be added to the cell incrementally, and the cell can be reset to zero if a value needs to be subtracted. To account for this constraint, an embodiment of the present invention proposes a modification to the write head. An embodiment of the present invention programs an electronic device (cell) of the external memory device to incrementally change the state of the electronic device by coupling a write signal to the input line of the external memory based on a write instruction from the controller and a write vector generated by the interface according to such write instruction. In a particular embodiment, the write head may maintain the previous state of the cell or reset the state of the cell completely to zero. It is noted that this modification does not matter the method steps of obtaining the write vector, and therefore may be applied to various types of memory-augmented neural networks. An embodiment of the present invention will be described in more detail with reference to the figures.

[0015] First, with reference to Figs. 1-3, an aspect of an embodiment of the present invention will be described that relates to a hardware-implemented method for operating a neural network system 1 to generate an output (S30) based on a received input (S10). The method and its variants are collectively referred to as "the method". All references to "Sij(k)" refer to the method steps illustrated in the flow charts of Figs. 3-5. Such a method relies on a neural network system 1 that further relates to another aspect of an embodiment of the present invention. Essentially, the system 1 includes a controller 10, a memory 30, and an interface 20. The interface 20 connects the controller 10 to the memory 30. The controller 10 comprises a processing unit configured to execute a neural network, and thus the controller may be considered a controller network. The memory 30 comprises a neuromorphic memory device having a crossbar array structure. The structure includes input lines 31 and output lines 32, the lines 31, 32 being interconnected at junctions via electronic devices 33 (e.g., memristive devices).

[0016] The electronic devices 33 of the neuromorphic memory device 30 are programmed S21-S23 to incrementally change (S23) the state of the devices 33. This is achieved by coupling a write signal to one or more of the input lines 31 of the crossbar array structure 30. The write signal is generated based on a write vector generated (S22) by the interface 20. The write vector itself is generated in accordance with a write command (S21) from the controller 10.

[0017] If desired, the electronic device 33 can retrieve data from the neuromorphic memory device 30 in accordance with a multiply-and-accumulate operation S24-S26. This is achieved by coupling a read signal to one or more of the input lines 31 of the neuromorphic memory device 30 based on a S25 read vector generated by the interface 20. The read vector itself is generated in accordance with a read instruction S24 from the controller 10.

[0018] The processing unit of the controller 10 is configured to implement and execute a neural network. It should be noted that the processing unit may be, for example, a general-purpose processing unit (e.g., a central processing unit of a standard computer) or may comprise analog circuitry. The processing unit may, among other things, include another neuromorphic device (not shown). However, in the illustrated embodiment, the neuromorphic memory device 30 does not form part of the processing unit. Rather, the neuromorphic device 30 is configured to function as an external memory for the controller 10 and the processing unit of the controller 10. Thus, the neuromorphic device 30 used as an external memory should not be confused with other neuromorphic devices that may form part of the controller 10 (or the write head and read head) in some cases.

[0019] The neural network system 1 can therefore be considered as a memory expansion system in which the external memory 30 is embodied as a crossbar array structure, the latter preferably being a memristive crossbar structure that may potentially use low precision (i.e. a single memristive element may be used to represent a memory element).

[0020] More generally, however, each electronic device 33 of the neuromorphic device 30 may possibly include one or more memristive devices. Also, as is customary in the art, dual output lines (columns) may be included, one for storing positive values ​​and the other for storing negative values. Several types of memristive devices may be contemplated, such as PCM cells, resistive random-access memory (RRAM), static random-access memory (SRAM) cells, or electro-chemical random-access memory (ECRAM) cells. In other variations, flash cells may be used.

[0021] The interface 20 typically includes multiple read / write heads (24 / 22) that connect the controller 10 to the memory 30. Such heads may be implemented in a variety of ways. As will be explained below with reference to another aspect of the invention, these heads may be implemented as shallow neural networks, among others.

[0022] According to this approach, the writing process is simplified to an incremental writing process such that there is no need to read the contents of the external memory 30 before writing (at least for the purpose of programming the cells). As in the embodiment described below, only a complete memory reset of the electronic device may be required, otherwise incremental writing is performed. That is, programming the electronic device 33 results in an incremental change in the state of the device 33 (e.g., a change in the electrical conductance of the device 33). The state of the electronic device 33 corresponds to a constant value that determines the data stored on the neuromorphic device 30.

[0023] It should be noted that the incremental write process contemplated herein may in particular include an incremental programming process, whereby the conductance state of the electronic device is incremented in small, constant steps. In a variant, the incremental write process relies on iterative programming, whereby the electronic device is programmed in a repetitive manner using multiple pulses. In another variant, as in the embodiment described below, the incremental values ​​are cumulatively stored in the additional memory 23 before writing such values.

[0024] Reading from the memory is realized by vector-matrix multiplication that can be performed in a fixed time, i.e. data can possibly be retrieved during any operating cycle. In practice, however, the read operation will only be performed at certain times (according to a general algorithm executed in the controller), and not necessarily at each operating cycle. In all cases, data can be retrieved via a multiply-and-accumulate operation parameterized by values ​​corresponding to the state of the electronic device 33. In this way, data can be stored on the external memory formed by the neuromorphic device, and such data can be called up by the controller 10 as needed to execute S20 a task monitored by the controller 10.

[0025] The read / write memory accesses are determined by respective read / write vectors based on input from the controller 10. Such vectors may be compared to memory addresses. However, whereas memory addresses correspond to specific locations in a normal hardware memory, here the read / write vectors determine the required signals and where to apply such signals to write to or read from the memory 30 according to a given operation, e.g., multiply-accumulate for a read operation. Note that the physical addresses of the electronic devices 33 are required in this case as well, since the read / write operations require accessing each electronic device 33 individually. Inputs to the crossbar array structure 30 may be realized, for example, using digital-to-analog converters (DACs), pulse width modulation (PWM) circuits 35, or alternatively, using a time-to-spike technique, which will be described in more detail later.

[0026] The approach has several advantages. First, the memory-augmented neural network system 1 as disclosed herein can efficiently utilize the crossbar structure, and the in-memory computational power allows read and write operations from the memory to be performed in constant time. Second, the proposed approach is not limited to memristive technology, other technologies such as flash cells can be contemplated. Third, the proposed write access eliminates the need to read the external memory before the write cycle (no partial reset of the memory cells is required). This allows full compatibility with known crossbar structures based on memristive devices, e.g. PCM devices. The proposed solution therefore allows significantly faster memory access while reducing energy consumption in memory-augmented neural networks.

[0027] Regardless of the incremental write approach, differentiability of the system can be restored by utilizing pseudo-derivatives as needed, as in the embodiments described below. Also, as mentioned above, the memory contents can be represented with low precision (e.g., 4 bits is sufficient). This approach allows for area-efficient implementation of the memory, since a single electronic device per junction is sufficient. However, the precision of the memory can be improved by using multiple memristive elements per junction, if needed.

[0028] All these points will now be described in detail with reference to a particular embodiment of the invention. Now referring to Fig. 3-4, programming S21-S23 of the electronic devices 33 may further include performing an erase operation (e.g., reset) before incrementally changing the state of one or more of the electronic devices 33 (S23). The erase operation is determined by a write vector. In an embodiment of the invention, such an erase operation may consist only of completely erasing the electronic devices 33 so as to reset their states. Depending on the data to be written to the memory 30, as determined by the write vector, some electronic devices 33 may need to undergo such an erase operation. In operation, other electronic devices 33 (usually the majority of such devices) may only be incremented at each write cycle without the need to reset the respective electronic devices 33. In other words, a two-value type of erase operation is contemplated, which may be either a complete erase or no erase of the values ​​stored on the electronic devices 33. The values ​​stored on the electronic devices correspond to the states of these devices. To erase the cells, a reset pulse is applied, for example an amorphization pulse for PCM cells. For example, if a value needs to be subtracted, the electronic device 33 may need to be reset to zero. Then, an addition term is written to the cell, which is calculated as the final value, i.e. corresponds to the difference between the initial value and the subtrahend.

[0029] Indeed, the technique may be designed and implemented to prevent partial erasure of any of the steps S21-S23 of the electronic device 33, at least for the purposes of programming the electronic device 33 (i.e., for writing to the electronic device 33). That is, the technique may be designed to ensure that no memory read access is required for the purposes of writing to the cells. Note, however, that the overall algorithm executed by the controller 10 may require data to be read from the memory before writing data to the memory, due to variations in this algorithm. However, the basic memory write process may be designed such that it is not necessary to read the memory contents before writing to the cells (to update the memory contents). In particular, the write vector may be generated by the interface 20 S22 without requiring a memory read access, i.e., without the need to first retrieve the data from the neuromorphic memory device 30. That is, step S22 may itself be independent of steps S24-S26.

[0030] As seen in Figures 1-2, the crossbar array structure 30 of the neural network system 1 may, for example, include N input lines 31 and W output lines 32, where N>1 and W>1. The memory capacity of the neuromorphic memory device 30 is then determined by a memory matrix M of dimension N x W. It should be noted that in a variant, the technique may also be applied to a single input line or a single output line or both. In addition, as mentioned above, dual output lines may be required.

[0031] In an embodiment, the write signal is generated according to two operations based on two types of write vectors. As envisaged in Fig. 4, the first operation determines the scope of the erase operation (i.e., reset completely or not at all), and the second operation is performed to incrementally change the state of the electronic device 33. That is, if the first type of write vector means that a reset is necessary for a given device 33 S231, this device is reset S232 before incrementally changing its state S233 according to the second type of write vector. Otherwise, if a reset is not necessary, the state of this device 33 is directly incremented S233. In other words, the write access is divided into an erase part and an add part.

[0032] For example, the first operation may include a memory matrix multiplied by a binary vector, which determines the extent of the binary erase operation (i.e., no erase or full erase), and the second operation includes an additive term to incrementally change the state of the electronic device 33. That is, two types of vectors are used, i.e., they are denoted by vector e T and v T and the two types of vectors allow for two actions (sums of actions).

[0033] In addition, the program can also use vector e T This may further include binarizing the values ​​of the components of M[i,j]=M[i,j](1-Ee' T [j])+Ev T [j], where E∈1 Nx1 and e'[j]∈(0,1). Wx1 Note that e' represents a binarized vector. The binary vector component e'[j] can be obtained by binarizing the normal vector component e[j]. Ee' T and Ev TThe term can be viewed as a modified write key, which can be generated in different ways depending on the architecture at hand (e.g., Differentiable Neural Computer [DNC], Neural Turing Machine [NTM], etc.).

[0034] In an embodiment, the binarization is performed by dividing the vector e T This is achieved by calculating the values ​​of the components of e'[j] as Heaviside functions of the shifted values ​​of the vector components. For example, one may use e'[j]=Θ(e[j]-0.5). In a variation, one may achieve a similar result using two Heaviside functions or even a comparator circuit, as would be understood by one skilled in the art.

[0035] Note that a direct implementation of the method may result in a loss of differentiability, which may harm training (e.g., in the case of applications to supervised learning based on backpropagation algorithms). Nevertheless, we can resort to pseudoderivatives to restore differentiability and thus backpropagation algorithms. For example, we can use an analytical approximation of the Heaviside function, e.g., Θ(x)≒(1+tanh(kx)) / 2 with a sufficiently large value k, to find the derivative

number

number

[0036] Thus, simplified write techniques can be included where either no erase or a full erase is performed before incrementing the state of device 33, whereas it is not necessary to read the memory contents before writing. Additionally, thresholding techniques can be used to restore differentiability (using pseudo-derivatives).

[0037] As mentioned above, the general algorithm executed by the controller 10 may nevertheless require reading content from the device 30 (besides writing content to the device 30). As shown in FIG. 3, the controller 10 retrieves data from the neuromorphic memory device 30 according to a multiply-and-accumulate operation (S24). For example, the interface 20 may receive a read vector w r (S25) and generate the read vector w r by the memory matrix M. Again, the vector w r is a read key that can be generated in different ways depending on the architecture at hand. Each read result r is a vector w r is obtained by multiplying by the memory matrix M. This operation is

number

[0038] As mentioned above, the incremental write approach is not compatible with cumulative programming. For example, referring to Fig. 5, the summation term included in the second operation may be performed by accumulating (S227) the summation value v of said summation term in an additional memory 23 (see Fig. 1) separate from the external memory 30. Again, such summation value is determined by a second type of write vector. In operation, after checking (S224) the current value v of a given summation value, the accumulated summation value v is compared to a given threshold value v. T(S225), the algorithm commands the state of the given electronic device 33 to be changed incrementally according to this accumulated sum v (S226). Thus, if the value of v is small, there is a possibility to accumulate v locally (in the additional memory 23) and apply v only in subsequent steps. As envisaged in Fig. 5, the accumulation process may be performed as part of the write vector generation (S22). The algorithm then proceeds as previously described with reference to Fig. 4.

[0039] Note that in a variant, updates can also be applied with some granularity during the write process. Such updates can be useful for hardware architectures that only support fixed-size updates, e.g. PCM devices. As a result, a certain value of v that is slightly different from the ideal value of v can be applied.

number

number

[0040] In a typical application, the processing unit of the controller 10 executes an algorithm S20 that causes the system 1 to interleave steps S21-S23 of programming the electronic device 33 and steps S24-S26 of retrieving data from the neuromorphic memory device 30. This algorithm may be executed in particular for training a neural network, thereby causing the processing unit to execute the neural network multiple times during operation. For example, inputs stored on the neuromorphic memory device 30 may be fed to input nodes of the neural network before executing said neural network. Upon execution of the neural network, outputs are stored in the neuromorphic memory device 30, etc.

[0041] Referring again to FIGS. 1-2, another aspect of the present invention will now be described, which relates to the neural network system 1 itself. As previously described, the system 1 includes a controller 10, which itself includes a processing unit. The processing unit is configured to execute the neural network. The system 1 further includes a memory 30 (i.e., an external memory) including a neuromorphic memory device. As previously described, the neuromorphic memory device of the memory 30 has a crossbar array structure including input lines 31 and output lines 32. The lines are interconnected at junctions via electronic devices 33. The system 1 further includes an interface 20 connecting the controller 10 to the memory 30.

[0042] The system 1 is generally configured to program the electronic device 33 to incrementally change the state of the electronic device 33. As described with reference to the method, this is achieved by coupling a write signal to one or more of the input lines 31. In operation, such a signal is generated based on a write instruction from the controller 10 and a write vector generated by the interface 20 in accordance with such a write instruction. In addition, data is retrieved from the neuromorphic memory device 30 in accordance with a multiply-accumulate operation by coupling a read signal to one or more of the input lines 31 of the neuromorphic memory device 30. The read signal is generated based on a read instruction from the controller 10 and a read vector generated by the interface 20 in accordance with such a read instruction.

[0043] In an embodiment, the system 1 is further configured to perform an erase operation before incrementally changing the state of the electronic devices 33. As already explained, said erase operation is determined by a write vector, whereby each erase operation may consist only of (i) completely erasing one of the electronic devices 33 or (ii) not erasing this electronic device 33 at all. Completely erasing an electronic device resets the state of the electronic device during operation. Furthermore, the neural network system 1 may be configured to prevent partial erasure of any of the electronic devices 33 (at least for the purpose of programming, i.e. writing to such electronic device 33) during operation of the system.

[0044] As can be seen in Figures 1-2, the crossbar array structure typically includes N input lines 31 and W output lines 32, where N>1 and W>1, such that the memory capacity of the neuromorphic memory device 30 is determined by a memory matrix M of dimension NxW. The crossbar array structure preferably comprises a single electronic device per junction, each of which may be, for example, a memristive device, i.e., a memristive memory element. In practice, the memory contents may be represented with limited precision (e.g., 4 bits), which allows to fully exploit the advantages of a memristive crossbar implementation.

[0045] As previously mentioned, interface 20 may include a read head 24 connected to both input line 31 and output line 32, and a write head 22 connected to input line 31. As previously explained, various configurations of read head 24 and write head 22 may be contemplated.

[0046] In an embodiment, the crossbar array structure further comprises a PWM circuit 35, as shown in FIG. 2B. The PWM circuit is used to controllably generate the signals necessary to write to and read from the memory device 30. Both the read head 24 and the write head 22 are connected to the input line 31 through respective PWM circuits 35. In a variant, a time-to-spike (TTS) approach can be used; that is, a low latency TTS integrator may be used that utilizes the timing of the signals to transmit information more efficiently.

[0047] Otherwise, a read head 24 is connected to the output of the crossbar structure to read the results from the multiply-accumulate operations and return such results to the controller 10. Standard read circuitry 36 can be used to read the output current from the output lines 32.

[0048] Interestingly, in some cases, the read head 24 may be embodied as a module implementing a shallow neural network, for example a two-layer network, i.e., an input layer whose input nodes are mapped to the outputs from the controller 10 and an output layer whose output nodes are mapped to the input lines 31 of the crossbar array structure. Similarly, the write head 22 may be embodied as a module implementing two shallow neural networks, each of which includes an input layer whose input nodes are mapped to the outputs from the controller 10 and an output layer whose output nodes are mapped to the input lines 31 of the crossbar array structure 30. This allows two types of write vectors to be output, based on which the system 1 can perform the two aforementioned operations, i.e., a first type of operation determines an erase operation and a second type of operation aims to incrementally change the state of the electronic device 33. Note that the read / write heads (24 / 22) may also be embodied as neuromorphic devices, each with a crossbar array structure. Such neuromorphic devices may be used appropriately to implement shallow networks, as needed to generate read and write signals.

[0049] The above embodiment has been briefly described with reference to the attached drawings and may be adapted to several variants. Several combinations of the above features may be contemplated. For example, in an embodiment, a memristive crossbar structure 30 (with PCM cells) is used together with optimized read / write heads (24 / 22) to realize an external memory for the controller 10 and its processing units. The controller aims to execute a neural network, whether for training the neural network or for performing inference based on a trained network. Such a neural network may therefore be augmented with a memory built on the memristive device 33. The memory contents may be represented with low precision or even with binary values ​​stored in the memristive device. A simplified write approach to perform memory write access without reading the current contents of the memory array 30 and without partial reset of the memristive device 33 is reliable.

[0050] The method can be applied, for example, to the so-called copy-repeat task (adopted from A. Graves and G. Wayne et al. Nature 2016, cited in the background section). The architecture adopted can be, for example, the following: A recurrent controller network 10 (containing 10 LSTM units instead of 64 LSTM units) can be used with an external memory device with memory size 16x16, as well as four read heads 24 and one write head 22. The network structure and the operation of the read and write heads 22 can otherwise be similar to those in the above description. As the inventors have observed, the aforementioned simplified memory access does not affect the performance of the network, nor does the limited precision used to represent the memory contents cause a loss of accuracy.

[0051] As another example application, a simple PWM circuit 35 can be used to calculate the similarity measure. To generate the read / write weights, the dot product and norm are calculated, i.e.

number

[0052] k represents the input vector and M represents the memory. Such a calculation could potentially be performed using one generation of the PWM input signal. A fixed portion representing 1 is added to the PWM signal corresponding to the value of k to calculate the norm ||M||1. Two read accesses from the device 30 are required, the first corresponding to the norm ||M||1 and the second corresponding to the vector-matrix multiplication kM. One way to obtain the two results separately is to reset the integrator after ||M||1 is calculated. In a variant, the integrator runs continuously and after the fixed portion has been processed at the input, the current value is stored in the auxiliary memory. After the complete input has been processed, the previously stored value must be subtracted from the sum result to obtain ||M||1 and kM.

[0053] Note that instead of using DAC or PWM circuits to present the input vectors in the rows / columns of the crossbar array 30, a time-to-spike approach can be used, which reduces the energy required to transmit the inputs. In this case too, the procedure to calculate the similarity measure can potentially be performed in a single crossbar operation. The pulse transmitted as a reference point in the TTS approach can be utilized for an L1-norm parallel read (input vector includes everything). A dot-product parallel read can be implemented using a second pulse and the TTS integrator approach. Instead of having a reference signal that activates the modulatory term of the TTS integrator to its initial value, we read the current during the reference signal to obtain ||M||1. As a result, one TTS read returns both the norm M and the value kM.

[0054] Although the present invention has been described with reference to a limited number of embodiments, variants, and accompanying drawings, those skilled in the art will recognize that various modifications may be made and equivalents may be substituted without departing from the scope of the invention. In particular, features (device-like or method-like) described in a given embodiment, variant, or illustrated in a drawing may be combined with or substituted for other features in another embodiment, variant, or drawing without departing from the scope of the invention. Thus, various combinations of the features described with respect to any of the above embodiments or variants may be contemplated and remain within the scope of the appended claims. In addition, many minor modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the scope of the invention. Thus, the invention is not limited to the particular embodiments disclosed, but it is intended to include all embodiments falling within the scope of the appended claims. In addition, many variations other than those expressly mentioned above may be contemplated.

[0055] 6 illustrates a block diagram of components of a computer system 500 that may be included as part of a neural network system 1 in accordance with an exemplary embodiment of the present invention. It should be understood that FIG. 6 is only intended to provide an illustration of one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

[0056] Computer system 500 includes a communications fabric 502 that provides communication between cache 516, memory 506, persistent storage 508, communications units 510, and input / output (I / O) interfaces 512. Communications fabric 502 may be implemented using any architecture designed to pass data and / or control information between a processor (such as a microprocessor, communications and network processor, etc.), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 502 may be implemented using one or more buses, or a crossbar switch.

[0057] Memory 506 and persistent storage 508 are computer readable storage media. In this embodiment, memory 506 includes random access memory (RAM). In general, memory 506 may include any suitable volatile or non-volatile computer readable storage medium. Cache 516 is a high-speed memory that improves the performance of computer processor 504 by holding recently accessed data and data near the accessed data from memory 506.

[0058] Programs may be stored in persistent storage 508 and memory 506 for execution by one or more of the respective computer processors 504 via cache 516. In one embodiment, persistent storage 508 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 508 may include a solid state hard drive, a semiconductor storage device, a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, or any other computer readable storage medium capable of storing program instructions or digital information.

[0059] The media used by persistent storage 508 may also be removable. For example, a removable hard drive may be used for persistent storage 508. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 508.

[0060] In these examples, the communications unit 510 provides for communication with other data processing systems or devices. In these examples, the communications unit 510 includes one or more network interface cards. The communications unit 510 may provide for communication using either or both physical and wireless communications links. Programs may be downloaded to the persistent storage 508 via the communications unit 510.

[0061] The I / O interface 512 allows for input and output of data to and from other devices that may be connected to the computer system 500. For example, the I / O interface 512 may provide a connection to an external device 518, such as a keyboard, a keypad, a touch screen, or some other suitable input device, or a combination thereof. The external device 518 may also include a portable computer-readable storage medium, such as, for example, a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data used to implement embodiments of the present invention may be stored within such a portable computer-readable storage medium and loaded onto the persistent storage 508 via the I / O interface 512. The I / O interface 512 also connects to a display 520.

[0062] Display 520 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0063] The programs described herein are identified based on the application for which they are implemented in a particular embodiment of the invention, however, it should be understood that the specific program nomenclature herein is used for convenience only, and thus the invention should not be limited to use in the particular application identified and / or implied by such nomenclature.

[0064] The present invention may be a system, method, or computer program product, or a combination thereof, at any possible level of integration of technical details. The computer program product may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0065] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0066] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0067] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to perform aspects of the invention by utilizing state information of the computer readable program instructions to personalize the electronic circuitry.

[0068] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0069] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable medium such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0070] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to create a computer-implemented process that causes a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0071] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in an order different from that noted in the figures. For example, two blocks shown in succession may in fact be performed simultaneously, as one step performed substantially simultaneously, in a partially or fully overlapping manner in time, depending on the functionality involved, or the blocks may sometimes be performed in the reverse order. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or a combination of dedicated hardware and computer instructions.

[0072] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A hardware implemented method for operating a neural network system, comprising: providing a neural network system including a controller, a memory, and an interface connecting the controller to the memory, the controller comprising a processing unit configured to execute a neural network, and the memory comprising a neuromorphic memory device having a crossbar array structure including input and output lines interconnected at junctions via electronic devices; programming the electronic device of the neuromorphic memory device to incrementally change a state of the electronic device by coupling a write signal to one or more of the input lines based on (i) a write command received from the controller and (ii) a write vector generated by the interface in accordance with the write command; retrieving data from the neuromorphic memory device according to a multiply-and-accumulate operation by coupling a read signal to one or more of the input lines of the neuromorphic memory device based on (i) a read command from the controller and (ii) a read vector generated by the interface in accordance with the read command; Including, programming the electronic devices further comprises performing an erase operation before incrementally changing the state of one or more of the electronic devices; the erase operation is determined by the write vector, whereby the erase operation completely erases the one or more of the electronic devices to reset a state of each of the electronic devices; the write signal is generated according to two operations based on two types of write vectors, the two operations including a first operation for determining the erase operation and a second operation for incrementally changing a state of the electronic device; the second operation includes a summing term; accumulating an addend value of the addend term, the addend value being determined by a second type of the two types of write vectors; and In response to the accumulated sum exceeding a given threshold, incrementally changing a state of the electronic device according to the accumulated sum. Executed by Hardware implementation method.

2. The method of claim 1 , wherein the write vector is generated by the interface without the interface retrieving data from the neuromorphic memory device.

3. The crossbar array structure of the neural network system provided includes N input lines and W output lines, where N>1 and W>1; The method of claim 1 , wherein the memory capacity of the neuromorphic memory device is determined by a memory matrix M of dimensions N×W.

4. The method of claim 1 , wherein programming the electronic device further comprises binarizing values ​​of components of one of the two types of vectors.

5. The method of claim 4 , wherein binarizing the value comprises calculating the value as a Heaviside function of shifted values ​​of vector components.

6. 4. The method of claim 3, wherein a second set of data is retrieved from the neuromorphic memory device according to the multiply-and-accumulate operation, whereby a read vector generated by the interface is multiplied by the memory matrix M.

7. 2. The method of claim 1, further comprising executing an algorithm by the processing unit of the controller, the algorithm causing the system to interleave the steps of: (i) programming the electronic device; and (ii) retrieving data from the neuromorphic memory device.

8. 1. A neural network system comprising: a controller including a processing unit configured to execute a neural network; a memory comprising a neuromorphic memory device having a crossbar array structure including input and output lines interconnected at junctions through electronic devices; an interface connecting the controller to the memory; Including, The system further comprises: programming the electronic device of the neuromorphic memory device to incrementally change a state of the electronic device by coupling a write signal to one or more of the input lines based on (i) a write command received from the controller and (ii) a write vector generated by the interface in accordance with the write command; Retrieving data from the neuromorphic memory device according to a multiply-and-accumulate operation by coupling a read signal to one or more of the input lines of the neuromorphic memory device based on (i) a read command from the controller and (ii) a read vector generated by the interface in accordance with the read command. It is configured as follows: the system is further configured to perform an erase operation before incrementally changing the state of the electronic device; the erase operations are determined by the write vectors such that each of the erase operations, upon operation, can only perform actions selected from the group consisting of: (i) completely erasing one of the electronic devices, and (ii) not erasing the one of the electronic devices at all; the interface comprising a read head connected to the input and output lines, and a write head connected to the input lines; The read head is embodied as a module implementing a two-layer neural network, the two-layer neural network comprising: an input layer, whose input nodes are mapped to outputs from the controller; an output layer, the output nodes of which are mapped to input lines of the crossbar array structure; Including, Neural network systems.

9. the crossbar array structure comprises N of the input lines and W of the output lines, where N>1 and W>1; the memory capacity of the neuromorphic memory device is determined by a memory matrix M of dimension N×W; 9. The neural network system of claim 8.

10. 10. The neural network system of claim 9, wherein the crossbar array structure comprises a single electronic device per junction, the electronic devices being memristive devices.

11. 9. The neural network system of claim 8, wherein the crossbar array structure further comprises pulse width modulation circuits, whereby both the read heads and the write heads are connected to the input lines through respective ones of the pulse width modulation circuits.

12. The write head is embodied as a module implementing two neural networks, each neural network comprising: an input layer, whose input nodes are mapped to outputs from the controller; 9. The neural network system of claim 8, further comprising: an output layer, wherein output nodes of the output layer are mapped to input lines of the crossbar array structure to output two types of write vectors, respectively, and based on the two types of write vectors, the system is adapted to perform two operations including a first operation of determining the erase operation and a second operation of incrementally changing a state of the electronic device.

Images