Training an Oscillatory Neural Network
By encoding data as time delays and using backpropagation for weight updates, the training of oscillatory neural networks is optimized, leading to faster and more accurate performance in tasks like image recognition.
Patent Information
- Application Number
- JP2022578958
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-07-02
- Filing Date
- 2021-06-10
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-06-10
AI Technical Summary
Existing oscillatory neural networks rely heavily on mathematical pre-training of weight matrices, which is inefficient and limits their performance in tasks like image and audio classification.
A method for training oscillatory neural networks by encoding data as a vector of time delays, using programmable coupling elements to adjust network weights, and employing backpropagation for efficient weight updates based on network errors.
This approach allows for highly efficient online training of oscillatory neural networks, achieving faster training speeds and improved performance in tasks such as image recognition, particularly in challenging scenarios with high gray scale variations.
Smart Images

Figure 0007590090000007 
Figure 0007590090000008 
Figure 0007590090000009
Abstract
Description
[Technical field]
[0001] The present invention relates generally to training oscillatory neural networks. A method for training an oscillatory neural network is provided, along with a neural network system for implementing such a method.
[0002] An Oscillatory Neural Network (ONN) is an artificial neural network that uses a network of coupled oscillators. The oscillators correspond to the neurons of a neural network, and the strengths of the connections between pairs of oscillators emulate the network (synaptic) weights. Such networks can be trained to perform correlation tasks, such as image classification and speech recognition, by processing training samples in the network and adapting the matrix of network weights so that the network "learns," or memorizes, the specific correlations to be detected. A few hardware implementations for ONNs have been proposed, and these networks usually rely on mathematical pre-training of the weight matrix, for example by Hebbian learning algorithms. Summary of the Invention
[0003] One aspect of the invention provides a neural network training method. The method includes encoding data defining training samples as a vector of time delays in a network input signal and providing the input signal to an oscillatory neural network. The network comprises at least one network layer, in which a plurality of electronic oscillators interconnected through programmable coupling elements storing respective network weights generate oscillatory signals at time delays dependent on the input signal to propagate the input signal from the input to the output of the layer. The network is adapted to provide a network output signal that is substantially linearly dependent on the phase of the oscillatory signal in the last layer of the network. The method includes calculating a network error dependent on the output signal and a desired output for the training samples, and calculating an update for each network weight by backpropagation of the error, such that the weight update for the network layer depends on the vector of time delays at the input to the layer and the calculated error at the output of the layer. The method further includes programming the coupling elements to update the stored weights according to the calculated weight updates.
[0004] Another aspect of the invention provides a neural network system comprising an encoder, an oscillatory neural network as described above, and a system controller. The encoder is adapted to encode data defining training samples as a vector of time delays in a network input signal. The oscillatory neural network receives and propagates the input signal, resulting in a network output signal as described above. The system controller is adapted to calculate network errors and weight updates, and to program the connecting elements of the network as described above.
[0005] A further aspect of the present invention provides a computer program product comprising a computer readable storage medium embodying program instructions executable by a processing device to cause the processing device to perform the neural network training method described above.
[0006] Embodiments of the invention are described in more detail below, by way of illustrative and non-limiting example, with reference to the accompanying drawings, in which: [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 shows a schematic representation of a neural network system embodying the present invention. [Diagram 2] FIG. 1 illustrates the coupling of oscillators in an exemplary network layer and the corresponding weight matrix for this layer. [Diagram 3] FIG. 2 illustrates the steps of the neural network training method performed by the system of FIG. 1. [Figure 4] FIG. 2 illustrates the operation of an encoder in an embodiment of the system. [Diagram 5] FIG. 2 is a diagram showing a more detailed configuration of the network layer in a preferred embodiment. [Figure 6] FIG. 1 is a diagram showing the structure of a single layer network in an embodiment of the system. [Figure 7] FIG. 7 is a diagram showing the structure of a nonlinear element in the embodiment of FIG. 6. [Figure 8] FIG. 1 is a diagram showing a structure of a two-layer network in an embodiment of the system. [Figure 9] FIG. 2 illustrates steps of weight update calculation in a preferred embodiment. [Figure 10] FIG. 1 illustrates learning of black and white images using low contrast training samples in an embodiment of the present invention. [Figure 11] FIG. 10 compares system performance using a training method embodying the invention and a Hebbian learning algorithm. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] The present invention may be a system, a method, and / or a computer program product, which may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.
[0009] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridge structures having instructions recorded thereon, and any suitable combination thereof. As used herein, computer-readable storage media is not to be construed as being a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over a wire.
[0010] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device through a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.
[0011] The computer readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or conventional procedural programming languages, such as object oriented programming languages, such as Smalltalk®, C++, and the "C" programming language or similar programming languages. The computer readable program instructions may be executed as a stand-alone software package, exclusively on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or exclusively on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.
[0012] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0013] These computer-readable program instructions may be supplied to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to cause a machine such that instructions that execute through the processor of the computer or other programmable data processing apparatus create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.
[0014] The computer-readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process.
[0015] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the figures. For example, two blocks shown in succession may in fact be executed substantially in parallel, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or performs a combination of dedicated hardware and computer instructions.
[0016] FIG. 1 is a schematic block diagram of a neural network system embodying the present invention. The system 1 comprises an ONN 2, an input encoder (ENC) 3, and a system controller 4. The ONN 2 includes at least one network layer l i , i=1, 2,...,N, (N>=1), through which a network input signal is propagated through the network. System controller 4 is shown here to comprise a central processing unit (CPU) 5 and a memory 6. Memory 6 stores a set of training samples 7 for the ONN, and one or more program modules 8. Program modules 8 comprise program instructions executable by CPU 5 to perform the functional steps of operation described below.
[0017] In operation of the system 1, the encoder 3 receives data defining training samples from the controller 4 and encodes the sample data as a vector τ of time delays in the network input signal. The resulting input signal S IN (τ) is fed to ONN2. Each layer l of ONN2 i comprises a number of electronic oscillators interconnected through programmable coupling elements, which generate oscillating signals at time delays depending on an input signal. An example of a network layer structure is shown diagrammatically in FIG. 2, which shows the structure of the first network layer l1. This layer is composed of n oscillators O1,...,O2, which correspond to the number of time delay elements in the vector τ. n Each oscillator O j=1~n is the vector element t j Encode the input signal S IN Component s of (τ) j (t j ), which is a fully-connected ONN2 layer, where each oscillator is coupled to the others through programmable coupling elements 9. The strength of these couplings is adjustable by programming the individual elements 9, which correspond to respective network weights w. The connectivity shown corresponds to a triangular weight matrix W, with zeros along the diagonal, as shown at the bottom of the figure.
[0018] Layer i Oscillators O1, . . . , O n is the input signal S IN (τ) and the weighting matrix W for the layers. n For multi-layer networks (N>1), these oscillatory signals are relayed to the next network layer (after further processing, detailed below), resulting in the input signal S2 (τ2) for that layer, and so on through the network. Thus, the input signal S IN (τ) is the time for each layer l i is the input signal S from the previous layer. i (τ i), are propagated across successive layers of the network from the input to the output of each layer.
[0019] ONN2 is the last layer of the network. N 1, the phase detector 10 is adapted to provide a network output signal that is substantially linearly dependent on the phase of the vibration signal in the last layer and the reference signal S ref The phase difference φ between OUT (φ), which encodes the vector of phase differences φ measured by phase detector 10. The network output signal S OUT (φ) is used to compute updates to the network weights w, as described in further detail below.
[0020] FIG. 3 shows the steps of the training method performed in the system 1. In step 20, the encoder 3 trains the network input signal S IN (τ). An example of this encoding process is described below. In step 21, the network output signal S OUT To obtain (φ), the input signal S IN As shown in Figure 1, the controller 4 also stores the network layer l (τ) for use in subsequent processing. i Input signal S to i (τ i ), where S1(τ1) is the network input S IN (τ). In step 22, the controller 4 calculates the network error, represented by δ, which is OUTThe set 7 depends on the training sample (φ) and the desired output for the training sample, which corresponds to the output to be obtained for the input training sample in a fully trained network. Each training sample in set 7 can be classified according to a particular correlation to be found from that sample in the trained network, whereby the various classifications or "labels" assigned to the samples are associated with respective desired outputs. In an image processing application, for example, labels may be assigned according to the content of the training images and the desired network output for the various labels defined accordingly.
[0021] After computing the network error in step 22, the controller computes weight updates Δw for the individual network weights w in each layer of the network in step 23. OUT These weight updates can be calculated by backpropagation in a particularly efficient manner, since (φ) depends substantially linearly on the phase of the signal produced from the time-delay encoded input samples. Backpropagation results in a process for the calculation of an error corresponding to the output of each network layer, starting with the network error calculated from the network output signal and working iteratively backwards through the network layers, from the end to the beginning. Updates to the weights in each network layer can then be based on the calculated error at the output of that layer. The goal of backpropagation is to adjust each weight in the network in proportion to how much it contributes to reducing the overall error in the network. Using the present system, each layer l of the ONN2 can be iteratively calculated. i The weight update for is the propagated signal S at the input to that layer. i (τ i ) time delay τ i and the calculated error δ at the output of the layer. i The matrix ΔW of weight updates Δw for each ONN layer can be computed succinctly and efficiently as a function of ΔW and ΔW (denoted as F). i =F(τi , δ i ) is calculated by the controller 4 in step 23. In step 24, the controller 4 provides programming signals to the ONNs 2 to program the combining elements, whereby the stored weights are updated according to the respective weight updates Δw.
[0022] The process of steps 20-24 may correspond to one epoch of an iterative training process in which a given training sample is fed to the ONN2 multiple times, e.g., until the network error is reduced below a threshold level or no further improvement is obtained. The process may then be repeated for each training sample, such that the training process iterates until all samples have been processed and / or a convergence condition (e.g., a desired performance level) is achieved, as shown in step 25.
[0023] The above embodiment results in an implementation of an ONN where the network weights are trained online by repeated propagation of training samples. While the oscillatory signal in the network is described by an exponential dependence of voltage on phase and time, the network output is substantially linear in phase and thus a linear function of the input time delay. This is exploited to allow weight updates to be calculated by backpropagation in a very simple manner. Network errors can be backpropagated and weight updates are efficiently calculated for incremental tuning of the network, allowing optimization of the weights by an online training process. The system thus provides highly efficient ONN training and a noticeable improvement in the performance of the trained network.
[0024] In a system embodying the invention, a network output signal S OUT The dependence of (φ) on phase is S OUTIt must be substantially linear in the sense that the dependence of (φ) is linear to a first approximation. In general, a greater degree of linearity leads to more accurate results, and therefore the function S OUT It is preferred that the linear term be the dominant term in (φ). If there are any higher order nonlinear terms, they are ideally small compared to the linear term, e.g., less than 5% of the magnitude of the linear term, more preferably less than 1%.
[0025] In general, any desired circuit implementation can be used for the electronic oscillator O. The preferred embodiment uses a relaxation oscillator circuit, and a particularly preferred implementation is detailed below. In a multi-layer ONN2, the different network layers l i can contain different numbers of oscillators, with the number of oscillators typically decreasing across successive layers of the network. As described further below, the network may also contain additional processing stages, such as pooling and activation stages. Although a fully connected network layer is shown in Figure 2, a given oscillator may be coupled to one or more oscillators in a layer through element 9.
[0026] The coupling elements 9 that provide the synaptic connections in the network can be implemented in a commonly known manner using a variety of programmable memory circuits. Various synapse implementations are known that are based on the memristive properties of resistive memory devices such as PCM (Phase Change Memory) cells. Such devices are non-volatile memory devices that can be programmed into different internal states, exhibiting different electrical resistances, by application of a current or voltage signal to the device. The coupling elements 9 can comprise any type of memristive device based on, for example, PCM cells or other resistive memory cells such as resistive RAM (RRAM) cells, including conductive bridge RRAM cells, oxide or metal oxide RRAM cells, carbon RRAM cells, magnetoresistive random access memory (MRAM) elements, and ferroelectric random access memory (FeRAM) elements, as well as circuit devices implementing programmable memory devices, such as CMOS circuits with transistors, resistors, capacitors, or inductors or combinations thereof.
[0027] In some implementations, the system controller 4 may be integrated with the ONN2 in a system-on-chip design. In other implementations, the system controller may be implemented by a host computer operably coupled to the ONN2. The control functions may be performed exclusively by the host computer, or the control functions may be implemented by a host processor integrated with the ONN2, communicating with a control device that performs local control operations, e.g., programming of network weights, in response to control signals from the host. In general, the controller 4 may be implemented by one or more (general-purpose or special-purpose) computer / programmable data processing devices, and the functional steps of the processing operations performed by the controller may be performed by hardware or software or a combination thereof. The memory 6 may comprise one or more data storage entities, and may also comprise a main memory, e.g., a DRAM (Dynamic Random Access Memory), or other storage, or a combination thereof, physically separate from the CPU 5, as well as a cache or other memory, or both, local to the CPU 5.
[0028] A preferred implementation of the system 1 is described in more detail below. Figure 4 illustrates the operation of the encoder 3 for an image processing application, where the input sample data defines the pixels of a training image. In this simple example, four pixels are shown, with pixel values corresponding to a grey scale between white and black. The encoder 3 generates a reference signal S ref For a pixel, the pixel values are encoded into a set of DC signals s1-s4 that are switched from 0 volts to a voltage V at respective time delays t1-t4. The magnitudes of these time delays correspond to gray-scale pixel values, for example between 0 (white) and 1 (black). In the example shown, signal s1 is refwhereby t1 = 0. The time delays increase with pixel value up to a maximum time delay of t4 for black pixels. The DC signals s1-s4 correspond to the network input signal S, which encodes the time delay vector τ = (t1, t2, t3, t4). IN Any number of image pixels, or data points, defining training samples of any type, can be encoded in this manner.
[0029] Figure 5 shows the network layer i 2 shows a particularly preferred implementation for the memristive circuit 30. The circuit 30 comprises n relaxation oscillators, each comprising a field effect transistor, here a MOSFET (metal oxide semiconductor field effect transistor) 31. Each oscillator further comprises a capacitor C in parallel with a device having a negative differential resistance (NDR) 32. The NDR 32 can be implemented in a variety of ways, this embodiment using a PCM switching element based on a VO2 thin film junction. The oscillators are coupled through an array 33 of RRAM elements 34, as generally described with reference to FIG. 2. Dense arrays of these memristive elements 34 can be fabricated as integrated nanoscale systems using well-known material processing techniques.
[0030] Input signal S IN The time delay encoded in (τ) determines the timing of the signals applied to the gates of each transistor 31 in the first layer 30 of the ONN. IN (τ) component signals s1, s2, . . . , s n can now be applied directly to the gate of each transistor 31, and the oscillating signals o1, o2, . . . , o n 6 shows the structure of a single layer ONN 35 using layer circuitry 30 in a preferred embodiment. An input encoder 36 encodes image data as described with reference to FIG. IN (τ) is the number of oscillators O1 to O2 coupled by the RRAM matrix 33 of the network layer 30. nThe phase detection stage 37 detects the signals o1, . . . , o n and the reference signal S ref The phase difference vector φ between OUT (φ), which is fed to an active stage 38 which applies a predetermined non-linear function to the output signal. The active stage 38 is here implemented in hardware by a set of diodes 39 to apply a Rectified Linear Unit (ReLU) function to the output signal. The diodes 39 could for example be implemented by transistors, but are not limited to resistors R d and capacitance C d This may be conveniently implemented by the equivalent circuit shown in FIG. 7, having V f represents the forward bias voltage of the diode. The resulting signal A(φ) from the active stage 38 is provided to the controller 4 for further processing, which is described in more detail below.
[0031] The vector of phase differences φ in the ONN 35 depends on the input time delay vector and the network weights. For the image processing application of FIG. 4, the network j Time delay at t j = 0 (corresponding to a white pixel), resulting in a reference signal S ref The input signal s can be trained to produce a vibration signal oj that is in phase with j Time delay at t j =t4 (corresponding to black pixels) results in S ref results in an oscillatory signal oj with a phase difference of π (at steady state) for . Grey scale values can result in phase differences depending on the particular correlation application. For example, if only black and white images need to be detected, grey scale values can result in a phase difference of 0, or a phase difference of 0 or π depending on the pixel value relative to a defined threshold. If grey scale images need to be detected, grey scale values can result in a phase difference on a scale between 0 and π.
[0032] FIG. 8 shows a two-layer ONN 40 based on the layer circuit 30, where like components of FIG. 6 are indicated by like reference numbers. Here, an active stage 41, equipped with diodes 39, is directly connected to the output of the first network layer l1. This stage also performs signal pooling by combining the signal components into m signals required for input to the next network layer. This layer has the general structure shown in FIG. 4 and consists of m oscillators O1 to O2 coupled by a RRAM matrix 42. m The phase detection stage 43 detects the vibration signals o1, . . . , o m and the reference signal S ref The phase difference between the resulting signal S OUT (φ) is fed to an active stage 44, also implemented by diodes 39, to apply a ReLU function to the network output. The resulting signal A(φ) is fed to the controller 4 for further processing.
[0033] The structure shown in FIG. 8 can be used with any desired number of network layers. i The initial values of the network weights can be selected in various ways, such as randomly distributed values, or can be determined by mathematical pre-training of the weights offline, for example through Hebbian learning.
[0034] The vibration signal in ONN35 and 40 is
number
number
number
number
number
[0035] The network error δ is the error δ at the output of each preceding network layer. i This can be calculated by the controller 4 using the chain rule in the usual way. Then, in step 52, for each network layer l iFor the described, more preferred embodiment, the weight update calculation simplifies to Equation 4:
number
[0036] The preferred embodiment described above results in a highly efficient system for online training of hardware ONNs. ONNs can operate at extremely high speeds and with very low power consumption. The layered structure of FIG. 5 results in fine-grained control of the oscillator circuit through application of an input signal to the gate of FET 31, giving highly accurate timing control. This improves the overall accuracy of the training process, providing faster training, and enhanced performance of the trained network.
[0037] Figure 10 shows test results for a system embodying the invention using a single-layer ONN to learn a training pattern. A simple 9-pixel training image, shown on the left side of the figure, was used for this test. The image pixels were applied to a single-layer ONN with 9 oscillators as described above, with the goal of training the ONN to detect black-and-white images regardless of any gray-scale changes in the input. The training images chosen therefore represent a particularly difficult test due to the degree of gray-scale changes. The network output for the test is shown by the equivalent flattened image in the figure. After only seven training epochs, the network output matched the required black-and-white image.
[0038] The trained network weights remain fixed when applied for inference operations after training. The network output then in response to a (previously unknown) input image indicates a label that maps to the particular type or category of image that the network was trained to detect. ONNs trained by methods embodying the present invention provide enhanced inference performance compared to conventional training methods based on Hebbian learning. This is shown in FIG. 11, which compares inference results for ONNs trained to recognize black and white images by Hebbian learning and the backpropagation training scheme described above. The probability that the network will correctly identify the correct black and white version of the input image is plotted against the allowed deviation in the gray scale of the input image, up to 50% on the gray scale from white to black. Both training methods gave accurate results for gray scale deviations up to about 23%. However, the method embodying the present invention significantly outperformed Hebbian learning for the more difficult tests at higher gray scale deviations.
[0039] It will be understood that many changes and modifications can be made to the described exemplary embodiment. For example, although a preferred hardware implementation is described above, the activation stage can be implemented by software in the controller 4 in other embodiments. In some embodiments, other activation functions can be used, for example sigmoid activation. Different cost functions can also be used, provided that the derivative can be easily defined. Various modifications to the oscillator circuit of FIG. 5 can also be envisioned. For example, instead of the NDR 32, the capacitors across each transistor 31 can be omitted.
[0040] Generally, where features are described with reference to a method embodying the invention, the corresponding features can also be provided in a system / computer program product embodying the invention, and vice versa.
[0041] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used in this specification is selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.
Claims
1. 1. A computer-implemented method for training a neural network, comprising: encoding, by one or more computer processors, data defining the training samples as vectors of time delays in a network input signal; feeding, by one or more computer processors, a network input signal to an oscillatory neural network having at least one network layer, in which a plurality of electronic oscillators interconnected through programmable coupling elements storing respective network weights generate oscillatory signals at a time delay dependent on the network input signal to propagate the network input signal from an input to an output of said layer, the oscillatory neural network being adapted to provide a network output signal that is substantially linearly dependent on a phase of the oscillatory signal in the last layer of the oscillatory neural network; calculating, by one or more computer processors, a network error dependent on said network output signals and a desired output for said training samples; computing, by one or more computer processors, updates for each network weight by backpropagating the errors, such that a weight update for a network layer depends on a vector of time delays at the input to that layer and the computed errors at the output of that layer; programming, by one or more computer processors, the combining element to update the stored weights in accordance with the weight updates; The method includes:
2. The computer-implemented method of claim 1 , wherein the network output signal depends on a phase difference between a vibration signal in a last layer and a reference signal.
3. The computer-implemented method of claim 1 or 2, wherein the network error comprises a derivative of a cost function that depends on the network output signal and the desired output for the training samples.
4. 4. The computer-implemented method of claim 3, wherein the cost function depends on a difference between the desired output for the training samples and a predetermined non-linear function of the network output signal.
5. 5. The computer-implemented method of claim 4, wherein the nonlinear function includes a ReLU function, and the network output signal is provided to an active stage comprising a set of diodes to apply the ReLU function to the network output signal.
6. The computer-implemented method of any one of claims 1 to 5, wherein the oscillatory neural network comprises multiple layers.
7. 7. The computer-implemented method of claim 6, wherein the output of each layer is connected to an active stage for applying a predetermined non-linear function to the output signal of that layer.
8. 8. The computer-implemented method of claim 7, wherein the non-linear function comprises a ReLU function and the active stage comprises a set of diodes.
9. Each oscillator comprises a field effect transistor; each time delay encoded in the network input signal determines the timing of a signal applied to a gate of a transistor in a first network layer to initiate generation of an oscillating signal by the transistor; A computer-implemented method according to any one of claims 1 to 8.
10. 1. A neural network system comprising: an encoder adapted to encode data defining the training samples as a vector of time delays in a network input signal; an oscillatory neural network for receiving said network input signal, said oscillatory neural network comprising at least one network layer, in which a plurality of electronic oscillators interconnected through programmable coupling elements storing respective network weights generate oscillatory signals at a time delay dependent on said network input signal so as to propagate said network input signal from an input to an output of said layer, said oscillatory neural network being adapted to provide a network output signal substantially linearly dependent on the phase of the oscillatory signal in the last layer of said oscillatory neural network; a system controller adapted to: calculate a network error dependent on the network output signals and a desired output for the training samples; calculate updates for each network weight by backpropagating the error, such that a weight update for a network layer depends on a vector of time delays at the input to that layer and the calculated error at the output of that layer; and program the combining element to update the stored weights according to the weight updates; A neural network system comprising:
11. 11. The neural network system of claim 10, wherein the network output signal depends on a phase difference between a vibration signal in a last layer and a reference signal.
12. 12. The neural network system of claim 10 or 11, wherein the network error comprises a derivative of a cost function that depends on the difference between the desired output for the training samples and a predetermined non-linear function of the network output signal.
13. 13. The neural network system of claim 12, wherein the nonlinear function includes a ReLU function, and the network output signal is provided to an active stage comprising a set of diodes to apply the ReLU function to the network output signal.
14. The neural network system of any one of claims 10 to 13, wherein the oscillatory neural network comprises multiple layers.
15. 15. The neural network system of claim 14, wherein the output of each layer is connected to an active stage for applying a predetermined non-linear function to the output signal of that layer.
16. 16. The neural network system of claim 15, wherein the nonlinear function comprises a ReLU function and the active stage comprises a set of diodes.
17. 17. The neural network system of claim 10, wherein each oscillator comprises a field effect transistor, and the oscillatory neural network is adapted to determine the timing of signals applied to gates of the transistors such that each time delay encoded in the network input signal initiates generation of an oscillating signal by the transistor in a first network layer.
18. The neural network system of any one of claims 10 to 17, wherein the coupling elements comprise resistive memory devices.
19. A neural network system as claimed in any one of claims 10 to 18, wherein each said oscillator comprises a relaxation oscillator.
20. 1. A computer program for training an oscillatory neural network, the computer program comprising: encoding data defining the training samples as a vector of time delays in a network input signal; feeding said network input signal to an oscillatory neural network having at least one network layer, in which a plurality of electronic oscillators interconnected through programmable coupling elements storing respective network weights generate oscillatory signals at a time delay dependent on said network input signal to propagate said network input signal from an input to an output of said layer, said oscillatory neural network being adapted to provide a network output signal substantially linearly dependent on a phase of the oscillatory signal in a last layer of said oscillatory neural network; calculating a network error dependent on said network output signals and a desired output for said training samples; computing an update for each network weight by backpropagating the error, such that the weight update for a network layer depends on a vector of time delays at the input to that layer and the computed error at the output of that layer; programming the combining element to update the stored weights according to the weight updates; A computer program for executing the above.
Citation Information
Patent Citations
System and method for learning data ranking function
JP2006236345A
Multi-layer oscillating network
US20190122095A1
Nonlinear neural network oscillator
US5446828A
Information processing apparatus and program
WO2011030848A1
