Method and apparatus for analog computing

Abstract

A method and an apparatus for processing an analog input signal and generating a corresponding analog output signal are disclosed. The input signal is supplied to an analog delay line comprising analog memory cells. A propagation signal causes the input signal to propagate through the sequence of memory cells in discrete time steps. Each memory cell is buffered and weighted by the stored analog parameters, and the sum of weighted memory cell values is accumulated as an analog signal. The result of accumulation is passed through an activation function. Multiple such blocks provide a convolutional neural network that works at extremely low power in comparison with its digital equivalent.

Description

[0001] METHOD AND APPARATUS FOR ANALOG COMPUTING

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority of Canadian Patent Application No. 3,241 ,774, filed on June 16, 2024, the disclosure of which is incorporated herein by reference.

[0004] BACKGROUND

[0005] 1. Field of the Technology

[0006] The present application relates to receiving an analog input signal and generating a corresponding analog output signal. In particular, it relates to generating the analog output signal based on stored analog parameters.

[0007] 2. Description of the Related Art

[0008] Digitally-implemented neural networks are increasingly used due to advances in algorithms and hardware. However, their power consumption, especially for large language models, is so high that entire power stations are sometimes required to supply the necessary electricity. Smaller digital neural networks consume less power, but still too much for many applications that absolutely require the lowest possible power consumption. Such applications might include monitoring environmental conditions in a remote region, using solar power to charge a battery. In remote monitoring applications, such as detecting the presence of a particular bird species to infer weather or soil quality over time, power availability is often severely limited.

[0009] In many applications, the decision as to whether or not a neural network can be used depends on the amount of power available which is an important limitation. When this amount of power is low, the option to use a neural network for advanced monitoring purposes may not be available which is a serious limitation. There is a need for at least one of a method and an apparatus that will overcome at least one of the above-identified limitations.

[0010] BRIEF SUMMARY

[0011] It is an object of the present technology to ameliorate at least some of the inconveniences present in the prior art. One or more implementations of the present technology may provide and / or broaden the scope of approaches to and / or methods of achieving the aims and objects of the present technology.

[0012] It will be appreciated that the technology disclosed herein is of great advantage since it enables small footprint and timely processing of computational workloads, including those of neural network training and inference, in applications for which available size and energy limitations would otherwise prevent the required level and speed of computation.

[0013] According to a broad aspect of the technology, there is disclosed an apparatus for receiving an analog input signal and generating a corresponding analog output signal based on stored analog parameters, the apparatus comprising an analog delay line comprising a plurality of memory cells for propagating an analog input signal through the memory cells in a time-sequenced manner under control of a propagation signal, each memory cell comprising a capacitor for storing an analog signal and a controllable transfer element for passing a stored charge to a next memory cell in response to the propagation signal, such that the analog input signal applied to a first memory cell is propagated through a defined sequence of memory cells in discrete time steps controlled by the propagation signal; one or more analog parameter storage arrays, each analog parameter storage array comprising an analog bias element for storing an analog bias value, and a plurality of analog weighting elements, each weighting element being associated with a corresponding memory cell of the analog delay line and for storing a respective analog weighting factor and for modifying a corresponding signal stored in or transferred from the associated memory cell and one or more analog computation circuits for receiving, from one or more respective analog parameter storage arrays, the respective analog bias value and the weighted signals, and for generating a corresponding analog output signal based on a combination of the respective analog bias value and the weighted signals.

[0014] According to one or more implementations of the technology, the apparatus further comprises a programming circuit for adjusting one or more of the stored parameters by configuring at least one of the analog bias elements and the one or more analog weighting elements.

[0015] According to one or more implementations of the technology, at least one of the one or more analog computation circuits is configured to perform a nonlinear function.

[0016] According to one or more implementations of the technology, the nonlinear function comprises a rectified linear unit (ReLU).

[0017] According to one or more implementations of the technology, the analog delay line comprises a one-dimensional arrangement of memory cells.

[0018] According to one or more implementations of the technology, the one or more analog computation circuits is for generating the corresponding analog output signal in the form of a voltage that corresponds to the combination of the received stored bias value and the weighted signals from the memory cells.

[0019] According to one or more implementations of the technology, each of the plurality of analog weighting elements comprises a resistive element.

[0020] According to one or more implementations of the technology, the resistive element comprises a memristor.

[0021] According to one or more implementations of the technology, the analog delay line, the one or more analog parameter storage arrays, and the one or more analog computation circuits are configured for implementing a layer in a neural network. According to one or more implementations of the technology, there is disclosed a use of the apparatus disclosed above for training the neural network ex situ.

[0022] According to one or more implementations of the technology, there is disclosed a use of the apparatus disclosed above for training the neural network in situ.

[0023] According to one or more implementations of the technology, the analog input signal comprises an analog audio signal.

[0024] According to one or more implementations of the technology, there is disclosed a use of the apparatus disclosed above for detecting a keyword or a spoken phrase from the analog audio signal.

[0025] According to a broad aspect of the technology, there is disclosed a method for processing an analog input signal and generating a corresponding analog output signal, the method comprising providing an analog input signal to an analog delay line comprising a plurality of memory cells, and propagating the analog input signal through the plurality of memory cells in a time-sequenced manner under control of a propagation signal, such that the analog input signal applied to a first memory cell is propagated through a defined sequence of memory cells in discrete time steps; modifying signals stored in or transferred from a plurality of the memory cells according to a respective analog weighting factor stored in an analog parameter storage array associated with the plurality of memory cells, thereby producing a respective plurality of weighted signals; generating the corresponding analog output signal by combining the weighted signals and providing the generated corresponding analog output signal.

[0026] According to one or more implementations of the technology, the corresponding analog output signal is generated by combining the weighted signals with an analog bias value stored in the analog parameter storage array with the analog weighting factors. According to one or more implementations of the technology, the method further comprises adjusting one or more of the stored parameters by configuring at least one of an analog bias element and one or more analog weighting elements.

[0027] According to one or more implementations of the technology, there is disclosed a use of the method disclosed above for implementing a layer in a neural network.

[0028] According to one or more implementations of the technology, the generating of the corresponding analog output signal comprises applying a nonlinear combination of the weighted signals from the memory cells.

[0029] According to one or more implementations of the technology, the nonlinear combination comprises a rectified linear unit (ReLU) function.

[0030] According to one or more implementations of the technology, the corresponding analog output has a voltage that corresponds to the combination of the weighted signals.

[0031] According to one or more implementations of the technology, there is disclosed a use of the method disclosed above wherein the neural network is a convolutional neural network.

[0032] According to one or more implementations of the technology, there is disclosed a use of the method disclosed above for training the neural network in situ.

[0033] According to one or more implementations of the technology, there is disclosed a use of the method disclosed above wherein the analog input signal comprises an audio signal.

[0034] According to one or more implementations of the technology, there is disclosed a use of the method disclosed above for detecting a keyword or a spoken phrase in the analog input signal. Implementations of the present technology each have at least one of the above- mentioned objects and / or aspects, but do not necessarily have all of them. It should be understood that some aspects of the present technology that have resulted from attempting to attain the above-mentioned object may not satisfy this object and / or may satisfy other objects not specifically recited herein.

[0035] Additional and / or alternative features, aspects and advantages of implementations of the present technology will become apparent from the following description, the accompanying drawings and the appended claims.

[0036] BRIEF DESCRIPTION OF THE DRAWINGS

[0037] For a better understanding of the present technology, as well as other aspects and further features thereof, reference is made to the following description which is to be used in conjunction with the accompanying drawings, where:

[0038] Figure 1 is a diagram which shows a voice-activated solar-powered light, including an analog neural network using one or more implementations of the technology;

[0039] Figure 2 is a flowchart which details processing steps performed when training and using the analog neural network disclosed in Figure 1 , including a processing step of storing corresponding parameters;

[0040] Figure 3 is a block diagram which illustrates the components of the voice- activated solar-powered light shown in Figure 1 ;

[0041] Figure 4 is a diagram which illustrates an embodiment of the analog neural network shown in Figure 1 , including analog signal convolution blocks and an analog signal classifier block;

[0042] Figure 5 is a diagram which illustrates details of an analog signal convolution block of the kind shown in Figure 4. Figure 6 is a diagram which illustrates details of one implementation of the analog signal convolution block shown in Figure 5;

[0043] Figure 7 is a diagram which illustrates details of one implementation of the analog signal classifier shown in Figure 4;

[0044] Figure 8 is a flowchart which details the processing step of storing parameters shown in Figure 2;

[0045] Figure 9 is a block diagram which shows an implementation of an apparatus for receiving an analog input signal and generating a corresponding analog output signal based on stored analog parameters; and

[0046] Figure 10 is a flowchart which shows an implementation of a method for processing an analog input signal and generating a corresponding analog output signal.

[0047] BRIEF DESCRIPTION OF EXAMPLE EMBODIMENTS

[0048] The examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the present technology and not to limit its scope to such specifically recited examples and conditions. It will be appreciated that those skilled in the art may devise various arrangements which, although not explicitly described or shown herein, nonetheless embody the principles of the present technology and are included within its spirit and scope.

[0049] Furthermore, as an aid to understanding, the following description may describe relatively simplified implementations of the present technology. As persons skilled in the art would understand, various implementations of the present technology may be of a greater complexity.

[0050] In some cases, what are believed to be helpful examples of modifications to the present technology may also be set forth. This is done merely as an aid to understanding, and, again, not to define the scope or set forth the bounds of the present technology. These modifications are not an exhaustive list, and a person skilled in the art may make other modifications while nonetheless remaining within the scope of the present technology. Further, where no examples of modifications have been set forth, it should not be interpreted that no modifications are possible and / or that what is described is the sole manner of implementing that element of the present technology.

[0051] Moreover, all statements herein reciting principles, aspects, and implementations of the present technology, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof, whether they are currently known or developed in the future. Thus, for example, it will be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present technology. Similarly, it will be appreciated that any flowcharts, flow diagrams, state transition diagrams, pseudo-code, and the like represent various processes which may be substantially represented in computer-readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0052] Now referring to Figure 1 , there is shown an apparatus for receiving an analog input signal and generating a corresponding analog output signal. The apparatus is a solar-powered light 101 that is switched on in response to a spoken command 102, which is the phrase "lights on". The solar-powered light 101 includes a solar cell 103 that charges a battery inside the solar-powered light 101 during the day. It will be appreciated that the amount of power received by the solar cell 103 depends on its area and efficiency, and a low cost, low area solar cell 103 is preferred. A microphone 104, always on, generates an analog input signal in response to environmental sounds. Digital systems for detecting keywords or phrases are well established, but typically consume hundreds of milliwatts of power, exceeding the sustainable energy budget of the solar- powered light 101 , and rendering the light 101 useless within a couple of hours of darkness, during which time it is most needed.

[0053] The solar-powered light 101 comprises an analog neural network integrated circuit 105 that performs keyword and phrase detection at greatly reduced power in comparison with digital systems. The reduced power required by the analog neural network 105 enables it to be used in low power, low cost, battery-operated applications, including the solar-powered light switch 101. As a result of signal processing performed by the analog neural network 105, a Light Emitting Diode (LED) array 106 is activated when the microphone picks up the spoken phrase "lights on". A support post 107 allows the solar-powered light 101 to be placed beside a footpath, and in other places where a mains supply for the light is unavailable, and it must be operated on solar power received by the solar cell 103. The solar-powered light 101 also comprises a Printed Circuit Board (PCB) 108 upon which various components are mounted, including the analog neural network integrated circuit 105.

[0054] Now referring to Figure 2, there are illustrated processing steps for the preparation and use of the solar-powered light 101 shown in Figure 1. It will be appreciated by the skilled addressee that some of the preparation steps, performed during the product development phase of the solar-powered light 101 require use of a general purpose digital computer 200. The skilled addressee will appreciate that the general purpose digital computer 200 may be of any type.

[0055] Two kinds of training may be performed when training the analog neural network and defining a set of neural network parameters for detection of the spoken phrase 102. A first kind of training is ex-situ training, where the characteristics of the analog neural network 105 are fully simulated in a digital emulation running on the general purpose computer 200.

[0056] A second kind of training is in-situ training, where training is performed by optimizing analog parameters with the help of the general purpose computer 200, but taking account of the measured errors and characteristics of the analog neural network 105. Both types of training may be performed prior to production, to produce a set of parameters that can be programmed into any particular manufactured instance of the analog neural network integrated circuit 105. Alternatively, or in addition, training may be performed individually for each analog neural network 105, prior to, or even after its inclusion in a specific product such as the solar-powered light 101. According to processing step 201 a question is asked as to whether ex- situ training is to be performed. If required, ex-situ training is performed at processing step 202 using a digital model of the analog neural network circuit 105 running on the general-purpose computer 200, to define the parameters of the analog neural network 105.

[0057] According to processing step 203, neural network parameters are stored as analog values in the analog neural network 105.

[0058] According to processing step 204, a question is asked as to whether in- situ training is to be performed. If required, in-situ training is performed at processing step 205 under control of the general-purpose computer 200 while connected to the analog neural network 105.

[0059] According to processing step 205, parameters are defined and / or refined by programming analog values stored in the analog neural network 105.

[0060] According to processing step 206, the analog neural network integrated circuit and its associated circuits are placed in the solar-powered light 101.

[0061] According to processing step 207, the solar-powered light 101 is deployed at its location. It will be appreciated that the solar-powered light 101 remains operational for as long as sufficient charge remains in its internal battery.

[0062] According to processing step 208, the spoken activation phrase 102 is detected and the LED array 106 is switched on.

[0063] According to processing step 209, the LED array 106 is switched off after five minutes. It will be appreciated by the skilled addressee that while five minutes is disclosed, the LED array 106 can be switched off after any defined number of minutes.

[0064] It will be appreciated by the skilled addressee that processing steps 208 and 209 are then repeated indefinitely, over many months and possibly years. Now referring to Figure 3, there is shown an implementation of the solar- powered light switch 101 shown in Figure 1. It will be appreciated by the skilled addressee that various alternative embodiments may be provided for the solar- powered light switch 101.

[0065] The solar-powered light switch 101 comprises the microphone 104, a preamplifier 301 and an anti-aliasing filter 302.

[0066] The microphone 104 supplies analog signals to the pre-amplifier 301 which supplies amplified analog signals to the anti-aliasing filter 302. The skilled addressee will appreciate that the microphone 104 and the pre-amplifier 301 may be of various types as known to the skilled addressee.

[0067] It will be further appreciated that the anti-aliasing filter 302 comprises a low pass filter that removes frequency components above 6kHz and enables the analog neural network to sample audio signals at 12kHz, which keeps current consumption low while capturing the essential characteristics of spoken commands.

[0068] The solar-powered light switch 101 further comprises the analog neural network 105, a comparator 305 and an ultra-low power microcontroller 307.

[0069] The output of the anti-aliasing filter 302 is supplied as the input 303 of the analog neural network 105. The analog neural network 105 has an output 304 in the form of a voltage that exceeds a threshold when the spoken command 102 is detected. The comparator 305 detects when the threshold is exceeded and supplies a detection signal 306 to the ultra-low power microcontroller 307. The skilled addressee will appreciate that the comparator 305 and the ultra-low power microcontroller 307 may be of various types. In one or more embodiments, the ultra-low power microcontroller 307 is STM32L432KC manufactured by STMicroelectronics NV (Chemin du Champ-des-Filles 39, 1228 Plan-les-Ouates, Switzerland). The skilled addressee will appreciate that various alternative implementations may be possible. The analog neural network 105 will be described in more details hereinbelow.

[0070] The solar-powered light switch 101 further comprises a semiconductor light switch 309 and the LED array 106.

[0071] The ultra-low power microcontroller 307 supplies an activation signal 308 to the semiconductor light switch 309, which then supplies power to the LED array 106. The skilled addressee will further appreciate that the semiconductor light switch 309 and the LED array 106 may be of various types. For instance, and in one or more embodiments, the LED array 106 is Cree XP G3, manufactured by CreeLED, Inc (4001 E Hwy 54, Suite 2000. Durham, NC 27713, USA). The light switch 309 is IRLZ44NPBF manufactured by Infineon Technologies AG (IFAG CP CC Am Campeon 1 -15 85579 Neubiberg Germany). The skilled addressee will appreciate that various alternative implementations may be possible.

[0072] After five minutes, in one or more embodiments, the ultra-low power microcontroller 307 switches off the semiconductor light switch 309, unless another activation command 102 has been detected.

[0073] The solar-powered light switch 101 further comprises an in-circuit programming interface 310.

[0074] It will be appreciated that the in-circuit programming interface 310 facilitates an optional connection to the general-purpose computer 200 for programming or training the analog neural network 105. It will be appreciated by the skilled addressee that this enables locale-specific parameters to be programmed into the analog neural network 105, via a Serial Protocol Interface (SPI) 311 , in one or more embodiments, so that an activation command 102 in a different language can be detected. The in-circuit programming interface 310 also facilitates in-situ training 205 to be performed even after the solar-powered light has been manufactured, should this be necessary. The Serial Protocol Interface (SPI) 311 is also used to support normal operation of the analog neural network, including initialization. The skilled addressee will appreciate that the in-circuit programming interface 310 may be of various types.

[0075] The solar-powered light switch 101 further comprises a charging circuit 321 , the solar cell 103 and a battery 313.

[0076] The charging circuit 312 receives power from the solar cell 103 and uses this to charge the battery 313, which will provide power for the LED array 106 and the entire circuit of the solar-powered light 101. The skilled addressee will appreciate that the charging circuit 312, the solar cell 103 and the battery 313 may be of various types. For instance, and in one or more implementations, the charging circuit 321 is BQ25570 manufactured by Texas Instruments (12500 T I Blvd, Ste T, Dallas, TX 75243, USA), the solar cell 103 is KS-Q103G manufactured by China Solar Ltd, (Fushen Builing, Huaxia Road, Futian District, Shenzhen, Guangdong, 518000, China) and the battery 313 is INR-18650-P26A, manufactured by E-One Moli Energy (Canada) Limited (20000 Stewart Crescent, Maple Ridge, BC, Canada, V2X 9E7).

[0077] Now referring to Figure 4, there is shown an implementation of the analog neural network 105 shown in Figure 3.

[0078] It will be appreciated that in one or more embodiments, the analog neural network 105 is implemented as a monolithic silicon integrated circuit and comprises multiple blocks.

[0079] A first analog signal convolution block 401 receives the analog input signal 303 and generates four analog output signals 402, 403, 404 and 405 that are supplied respectively to four analog signal convolution blocks 406, 407, 408, 409.

[0080] In other words, the output signal 402 is supplied as the input to the analog signal convolution block 406, the output signal 403 is supplied as the input to the analog signal convolution block 407, the output 404 is supplied as the input to the analog signal convolution block 408 and the output 405 is supplied as the input to the analog signal convolution 409. Each of the analog signal convolution blocks 406 to 409 operates similarly to the analog signal convolution block 401 , but each only generates two outputs. Eight outputs 410 to 417 are provided as analog inputs to an analog signal classifier block 418, which provides the output 304 of the analog neural network.

[0081] The skilled addressee will appreciate that various alternative implementations may be possible.

[0082] An SPI circuit 419 communicates with the microcontroller 307 via the SPI connections 311. The SPI circuit comprises an interface 420 and SPI registers 421 for controlling initialization and programming of the analog signal convolution blocks 401 , 406 to 409 and 418.

[0083] Now referring to Figure 5, there is shown an embodiment of the analog signal convolution block 401 shown in Figure 4.

[0084] It will be appreciated by the skilled addressee that the analog signal convolution blocks 406 to 409 operate very similarly to the analog convolution block 401 , and Figure 5 also describes the structure of the analog convolution blocks 406 to 409. However, for the purpose of clarity, the description will be provided for the analog convolution block 401 only.

[0085] The analog signal convolution block 401 comprises an analog delay line 501 that comprises multiple memory cells 502, 503, 504, 505 and 506. It will be appreciated by the skilled addressee that while only five memory cells are shown in Figure 5, many more are provided. An analog signal provided at the input 303 is propagated from the input 303 to the first memory cell 502 in a time-sequenced manner under control of a propagation signal 507 generated by a propagation state machine 507. If, for example, an analog value of 0.5V is provided at the input 303, this will propagate to the first memory cell 502 in response to the propagation signal 507. The input 303 may then change to 0.3V, and, in response to the next propagation signal, an electrical charge representing the analog value of 0.5V is passed to the second memory cell 503, and the first memory cell 502 now holds the value of 0.3V. Stored signals, or samples, are propagated through the defined sequence of the memory cells 502 to 506 in discrete time steps controlled by the propagation signal 507. The memory cells 502 to 506 provide analog outputs 509 to 513 to a first analog parameter storage array 514 which is programmable and non-volatile. The analog parameter storage array 514 comprises multiple analog weighting elements 515 to 519, each providing a weight value associated with a corresponding memory cell 502 to 506. The analog parameter storage array 514 also comprises an analog bias element 520 for storing an analog bias value. The weighting elements 515 to 519 modify the corresponding signal stored in or transferred from their respective associated memory cell 502 to 506. The weight values and bias value are stored as resistance states in memristors in the circuit of the analog parameter storage array 514.

[0086] The analog signal convolution block 401 further comprises a first analog computation circuit 521 that includes an accumulator 522 and an activation function 523. The accumulator 522 provides an analog sum-of-products with bias signal, where each product is the result of modifying a corresponding signal stored in or transferred from a memory cell. For example, if the voltage on memory cell 502 is A volts, and the weighting element 515 implements a factor of B, their product is AB volts. All such products are added together by the accumulator 522. The bias 520 is also included in this sum. The activation function 523 implements a Rectified Linear Unit (ReLU) non-linear transfer function. The output from the activation function 523 provides the analog output 402 previously shown in Figure 4.

[0087] The analog signal convolution block 401 also comprises a second analog parameter storage array 524 comprising weighting elements 525 to 530. For example, if the weighting element 525 implements a factor of C, the corresponding product is AC, because weighting element 525 receives the signal 507 from the first memory cell.

[0088] The analog signal convolution block 401 further comprises a second analog computation circuit 531 that comprises an accumulator 532 and an activation function 533. The accumulator 532 and activation function 533 operate in the same way as the accumulator 522 and activation function in the first computation circuit 521 . The output of the second activation function 533 provides the analog output 403 previously shown in Figure 4.

[0089] The analog signal convolution block 401 also comprises two more analog computation circuits, not shown, that provide analog outputs 404 and 405 previously shown in Figure 4. The analog signal convolution block 401 has a total of four of the analog computation circuits of the kind shown at 521 and 531. Analog signal convolution blocks 406 to 409, by contrast, only comprises two of the analog computation circuits, giving two analog outputs per block. In other respects, analog signal convolution blocks 406 to 409 operate similarly to the analog signal convolution block 401 .

[0090] The analog parameter storage arrays 514 and 524 are programmable and store analog values for weights and bias in non-volatile memory elements. A programming circuit 534 supplies programming signals 535 to the analog parameter storage arrays 514 and 524 shown in Figure 4, and to those omitted from Figure 4 for the sake of clarity.

[0091] It will be therefore appreciated that more broadly there is disclosed an apparatus 401 , shown in Figure 9, for receiving an analog input signal 303 and generating a corresponding analog output signal 402 based on stored analog parameters.

[0092] The apparatus 401 comprises an analog delay line 501 comprising a plurality of memory cells (502, 503, 504, 505 and 506) for propagating an analog input signal 303 through the memory cells (502, 503, 504, 505 and 506) in a time- sequenced manner under control of a propagation signal 507, each memory cell comprising a capacitor for storing an analog signal and a controllable transfer element for passing a stored charge to a next memory cell in response to the propagation signal 507, such that the analog input signal applied to a first memory cell is propagated through a defined sequence of memory cells in discrete time steps controlled by the propagation signal 507. While it is disclosed that the plurality of memory cells comprises memory cells 502-506, it will be appreciated that the plurality of memory cells may comprise any number of memory cells suitable with a given application. In one or more implementations, the analog delay line 501 comprises a one-dimensional arrangement of memory cells. In one or more alternative embodiments, the analog delay line 501 comprises a two-dimensional arrangement of memory cells. In fact, it will be appreciated that any arrangement of memory cells may be used.

[0093] The apparatus 401 further comprises one or more analog parameter storage arrays 514. Each analog parameter storage array 514 comprises an analog bias element 520 for storing an analog bias value, and a plurality of analog weighting elements 515-519, each weighting element being associated with a corresponding memory cell of the analog delay line 501 and for storing a respective analog weighting factor and for modifying a corresponding signal stored in or transferred from the associated memory cell. It will be appreciated by the skilled addressee that the number of analog storage arrays 514 depends on an application sought and is not limited to a given specific number.

[0094] In one or more implementations, each of the plurality of analog weighting elements 515-519 comprises a resistive element. In one or more implementations, the resistive element is a memristor.

[0095] The apparatus 401 further comprises one or more analog computation circuits 521 .

[0096] The one or more analog computation circuits 521 are for receiving, from one or more respective analog parameter storage arrays, the respective analog bias value and the weighted signals, and for generating the corresponding analog output signal 402 based on a combination of the respective analog bias value 520 and the weighted signals.

[0097] It will be appreciated by the skilled addressee that the number of analog computation circuits 521 depends on an application sought and is not limited to a given specific number.

[0098] In one or more implementations, at least one of the one or more analog computation circuits 521 is configured to perform a non-linear function. In one or more implementations, the non-linear function comprises a rectified linear unit (ReLU).

[0099] In one or more implementations, not shown, the apparatus 401 further comprises a programming circuit, not shown, for adjusting one or more of the stored parameters by configuring at least one of the analog bias elements and the one or more analog weighting elements.

[0100] In one or more implementations, the one or more analog computation circuits 521 is for generating the corresponding analog output signal 402 in the form of a voltage that corresponds to the combination of the received stored bias value and the weighted signals from the memory cells.

[0101] In one or more implementations, the analog delay line 501 , the one or more analog parameter storage arrays 514, and the one or more analog computation circuits 521 are configured for implementing a layer in a neural network. In one or more implementations, there is disclosed a use of the apparatus 401 for training a neural network in situ.

[0102] In one or more implementations, there is disclosed a use of the apparatus 401 for training a neural network ex situ.

[0103] In one or more implementations, the analog input signal 303 comprises an analog audio signal. In one or more implementations, the apparatus 401 is used for detecting a keyword or a spoken phrase from the analog audio signal 303.

[0104] Now referring to Figure 6, there is shown an implementation of the analog signal convolution block 401 shown in Figure 5.

[0105] The propagation signal comprises two parts, 507a and 507b. In one or more implementations, each propagation signal part is a non-overlapping pulse waveform which causes an analog input signal to be propagated from the input into the first memory cell 502, then to the second memory cell 503, through a defined sequence of memory cells 502 to 506 in discrete time steps defined by the non-overlapping waveforms of the propagation signal 507. While 507a is high, 507b is low, causing transistor 601 to charge capacitor 602 to the voltage on the input 303. During this time, transistor 603 is off, so that the stored charge on capacitor 604 is stable and can be used to perform analog computation. Transistors 601 and 603 are controllable transfer elements. An isolation transistor 605 improves linearity of charge storage and transfer. In the next phase of propagation, 507a is low and 507b goes high. Stored charge is passed to the next memory cell in response to the propagation signal 507. The capacitor 604 receives charge from the capacitor 602 via the transistor 603, so that the voltage previously present at the input is now stored in the first memory cell 502. The same process repeats along the analog delay line 501. Each memory cell has a buffer so that the voltage on its capacitor can be supplied for computation without discharging the capacitor and interfering with the propagation of charge in its defined sequence. For example, the first memory cell 502 has a buffer 606. This facilitates output to analog weighting elements, including 515 and 525. The defined sequence of charge transfer takes place in discrete time steps under control of the propagation signal 507.

[0106] Analog weighting element 515 comprises two memristors, which can be programmed to a particular resistance. A first memristor 607 is used to define a negative weighting factor and a second memristor is used to define a positive weighting factor. Usually, one of the two memristors will be programmed for a very high resistance, so that only the desired positive or negative weighting factor is produced, although in an embodiment both memristors can be programmed to conduct. The analog bias element is similarly provided by a pair of memristors 609 and 610, enabling a positive or negative bias value to be provided as input to the accumulator 522.

[0107] The accumulator 522 is implemented by an amplifier having a positive 611 and negative input 612. The ReLU activation function 523 is implemented by limiting a linear function to the magnitude of the power supply of the analog neural network 105. The output of the ReLU 514 provides the first output 402 of the analog signal convolution block 401 . The programming circuit 534 shown in Figure 5 comprises components shown in Figure 6 that enable the memristors to be programmed with their respective resistance values. These resistance values define the neural network parameters represented by weighting elements such as 515 to 519 and bias element 520.

[0108] It will be appreciated that the programming is performed by setting both propagation signal components 507a and 507b to zero volts and switching off a pull-up transistor 613 for the bias element 610.

[0109] Next, a forward or reverse voltage is supplied to the weighting memristor being programmed. For example, the weighting element 515 is programmed by activating a tri-state buffer 614 and supplying a programming voltage P0 to its input. In order to program the first memristor 607, another tri-state buffer 615 is activated and supplied with a programming voltage PM. Alternatively, the second memristor 608 is programmed by activating a different tri-state buffer 616 and supplying it with a programming voltage PP.

[0110] It will be appreciated that such programming is performed incrementally and by testing the resulting effect on the circuit after programming. A similar process is performed to program the analog bias element 613.

[0111] From the schematic shown in Figure 6, the skilled addressee will be able to implement the additional analog parameter storage array 524 and analog computation circuit 531 shown in Figure 5, including the required programming circuit 534.

[0112] Now referring to Figure 7, there is shown an implementation of the analog signal classifier block 418, shown in Figure 4.

[0113] Outputs 410 to 417 from the analog signal convolution blocks 406 to 409 are supplied as inputs to an analog parameter storage array 701 . This comprises analog weighting elements 702 to 709, and an analog bias element 710. An analog computation circuit 711 comprises an accumulator 712 to combine the weighted input signals and a ReLU activation function 713 that provides the output 304 of the block 418 and also of the analog neural network 105.

[0114] The skilled addressee will be immediately able to reuse elements of the circuit shown in Figure 6 to implement the analog signal classifier block 418. The computation circuit 711 in the classifier block 418 operates identically to the computation circuits 521 and 522 in the convolution blocks, reusing identical circuit structures for summation and activation.

[0115] Now referring to Figure 8, there is shown an embodiment for storing neural network parameters in the analog neural network 105, shown in Figure 2.

[0116] According to processing step 801 , a first analog weighting element or bias element memristor is selected for programming.

[0117] According to processing step 802, a test signal is propagated through the respective analog delay line 501 so the respective memristor's resistance can be measured.

[0118] According to processing step 803, a measurement is performed between the memristor's resistance and the target required to correctly represent the neural net parameter weight or bias.

[0119] According to processing step 804, a test is performed in order to determine as to whether there is no significant difference. If so, the memristor has been successfully programmed with the required parameter value. If not, control is directed to processing step 805, where the memristor's resistance is adjusted by applying a programming voltage to the memristor.

[0120] It will be appreciated that the programming voltage is applied using the tristate buffers 614, 615 and 616 shown in Figure 6, and / or other tri-state buffers in other parts of the analog neural network 105. The parameters of the analog neural network 105 are provided by the memristors in weighting elements and bias elements previously described, including weighting elements 515 to 519 and bias element 520. For example, memristors 607 and / or 608 define one parameter of the analog neural network 105. Memristors 609 and 610 define another parameter of the analog neural network 105.

[0121] After processing step 805, control is directed back to processing step 803, to determine whether further adjustment of the selected weighting element or bias element is required. Eventually, control will be directed to processing step 806, where a test is performed to determine as to whether additional weighting elements or a biasing element require programming. If so, control is directed to processing step 801. Alternatively, that completes the processing steps required for programming the parameters of the analog neural network 105.

[0122] It will be appreciated that more broadly, there is disclosed, and illustrated in Figure 10, a method for processing an analog input signal and generating a corresponding analog output signal.

[0123] According to processing step 1001 , an analog input signal is provided to an analog delay line comprising a plurality of memory cells and the analog input signal is propagated through the plurality of memory cells in a time-sequenced manner under control of a propagation signal.

[0124] It will be appreciated that the analog input signal applied to a first memory cell is propagated through a defined sequence of memory cells in discrete time steps.

[0125] According to processing step 1002, signals stored in or transferred from a plurality of the memory cells are modified according to a respective analog weighting factor stored in an analog parameter storage array associated with the plurality of memory cells, thereby producing a respective plurality of weighted signals.

[0126] According to processing step 1003, the corresponding analog output signal is generated by combining the weighted signals.

[0127] According to processing step 1004, the generated corresponding analog output signal is provided. In one or more implementations, the corresponding analog output signal is generated by combining the weighted signals with an analog bias value stored in the analog parameter storage array with the analog weighting factors.

[0128] In one or more implementations, the method further comprises adjusting one or more of the stored parameters by configuring at least one of an analog bias element and one or more analog weighting elements.

[0129] It will be appreciated that the method disclosed herein may be used for implementing a layer in a neural network as disclosed above.

[0130] In one or more implementations, the generating of the corresponding analog output signal comprises applying a non-linear combination of the weighted signals from the memory cells.

[0131] In one or more implementations, the non-linear combination comprises a rectified linear unit (ReLU) function.

[0132] In one or more implementations, the corresponding analog output has a voltage that corresponds to the combination of the weighted signals.

[0133] In one or more implementations, the neural network is a convolutional neural network.

[0134] As mentioned above, it will be appreciated that the method disclosed above may be used in one or more implementations for training a neural network in situ.

[0135] In one or more implementations, the analog input signal comprises an audio signal.

[0136] In one or more implementations, the neural network is configured for detecting a keyword or a spoken phrase in the analog input signal.

[0137] In some cases, what are believed to be helpful examples of modifications to the present technology may also be set forth. This is done merely as an aid to understanding, and, again, not to define the scope or set forth the bounds of the present technology. These modifications are not an exhaustive list, and a person skilled in the art may make other modifications while nonetheless remaining within the scope of the present technology. Further, where no examples of modifications have been set forth, it should not be interpreted that no modifications are possible and / or that what is described is the sole manner of implementing that element of the present technology.

[0138] Modifications and improvements to the above-described implementations of the present technology may become apparent to those skilled in the art. The foregoing description is intended to be exemplary rather than limiting.

Claims

CLAIMS:

1. An apparatus for receiving an analog input signal and generating a corresponding analog output signal based on stored analog parameters, the apparatus comprising:(a) an analog delay line comprising a plurality of memory cells for propagating an analog input signal through the memory cells in a time- sequenced manner under control of a propagation signal, each memory cell comprising a capacitor for storing an analog signal and a controllable transfer element for passing a stored charge to a next memory cell in response to the propagation signal, such that the analog input signal applied to a first memory cell is propagated through a defined sequence of memory cells in discrete time steps controlled by the propagation signal;(b) one or more analog parameter storage arrays, each analog parameter storage array comprising an analog bias element for storing an analog bias value, and a plurality of analog weighting elements, each weighting element being associated with a corresponding memory cell of the analog delay line and for storing a respective analog weighting factor and for modifying a corresponding signal stored in or transferred from the associated memory cell;(c) one or more analog computation circuits for receiving, from one or more respective analog parameter storage arrays, the respective analog bias value and the weighted signals, and for generating a corresponding analog output signal based on a combination of the respective analog bias value and the weighted signals.

2. The apparatus of claim 1 , further comprising a programming circuit for adjusting one or more of the stored parameters by configuring at least one of the analog bias elements and the one or more analog weighting elements.

3. The apparatus of claim 1 , wherein at least one of the one or more analog computation circuits is configured to perform a non-linear function.

4. The apparatus of claim 3, wherein the non-linear function comprises a rectified linear unit (ReLU).

5. The apparatus of claim 1 , wherein the analog delay line comprises a onedimensional arrangement of memory cells.

6. The apparatus of claim 1 , wherein the one or more analog computation circuits is for generating the corresponding analog output signal in the form of a voltage that corresponds to the combination of the received stored bias value and the weighted signals from the memory cells.

7. The apparatus of claim 1 , wherein each of the plurality of analog weighting elements comprises a resistive element.

8. The apparatus of claim 7, wherein the resistive element comprises a memristor.

9. The apparatus of claim 1 , wherein the analog delay line, the one or more analog parameter storage arrays, and the one or more analog computation circuits are configured for implementing a layer in a neural network.

10. Use of the apparatus as claimed in claim 9 for training the neural network ex situ.11 . Use of the apparatus as claimed in claim 9 for training the neural network in situ.

12. The apparatus of claim 1 , wherein the analog input signal comprises an analog audio signal.

13. Use of the apparatus as claimed in claim 12 for detecting a keyword or a spoken phrase from the analog audio signal.

14. A method for processing an analog input signal and generating a corresponding analog output signal, the method comprising:(a) providing an analog input signal to an analog delay line comprising a plurality of memory cells, and propagating the analog input signal through the plurality of memory cells in a time-sequenced manner under control of a propagation signal, such that the analog input signal applied to a first memory cell is propagated through a defined sequence of memory cells in discrete time steps;(b) modifying signals stored in or transferred from a plurality of the memory cells according to a respective analog weighting factor stored in an analog parameter storage array associated with the plurality of memory cells, thereby producing a respective plurality of weighted signals; and(c) generating the corresponding analog output signal by combining the weighted signals; and(d) providing the generated corresponding analog output signal.

15. The method of claim 14, wherein the corresponding analog output signal is generated by combining the weighted signals with an analog bias value stored in the analog parameter storage array with the analog weighting factors.

16. The method of claim 15, further comprising adjusting one or more of the stored parameters by configuring at least one of an analog bias element and one or more analog weighting elements.

17. Use of the method as claimed in claim 14 for implementing a layer in a neural network.

18. The method of claim 14, wherein the generating of the corresponding analog output signal comprises applying a non-linear combination of the weighted signals from the memory cells.

19. The method of claim 18, wherein the non-linear combination comprises a rectified linear unit (ReLU) function.

20. The method of claim 14, wherein the corresponding analog output has a voltage that corresponds to the combination of the weighted signals.21 . Use of the method as claimed in claim 17, wherein the neural network is a convolutional neural network.

22. Use of the method claimed in any of claim 19 or 20 for training the neural network in situ.

23. Use of the method as claimed in claim 17, wherein the analog input signal comprises an audio signal.

24. Use of the method as claimed in claim 23, wherein the neural network is configured for detecting a keyword or a spoken phrase in the analog input signal.

Images