Neuromorphic devices and methods for implementing neural networks

Abstract

To provide a neuromorphic device, and a method for implementing a neural network by using the neuromorphic device.SOLUTION: A method for implementing a neural network includes the steps of: learning, by a first system, respective first layers included in the neural network; learning, by a second system, a second layer of at least one layer included in the neural network; and generating output data from input data by using the learned first layers and the learned second layer of at least one layer.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a neuromorphic apparatus and a method for implementing a neural network using a neuromorphic apparatus. [Background technology]

[0002] A memory-centric neural network device refers to a computational architecture that models the biological brain. With the development of memory-centric neural network technology, research is actively underway to utilize memory-centric neural networks in various electronic systems to analyze input data and extract useful information.

[0003] Therefore, in order to use a memory-based neural network to analyze a large amount of input data in real time and extract desired information, a technology capable of efficiently processing calculations is required. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Chinese Patent Application Publication No. 109740734 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention provides a neuromorphic device and a method for implementing a neural network using the neuromorphic device. It also provides a computer-readable recording medium having a program for executing the method on a computer. The technical problems to be solved are not limited to those described above, and other technical problems may exist. [Means for solving the problem]

[0006] According to one aspect, a method for implementing a neural network includes: training each first layer included in the neural network using a first method; training at least one second layer included in the neural network using a second method; and generating output data from input data using the trained first layer and the trained at least one second layer.

[0007] A computer-readable recording medium according to another aspect includes a recording medium having a program recorded thereon for causing a computer to execute the above-described method.

[0008] According to yet another aspect, a neural network includes a plurality of convolutional layers and at least one fully-connected layer, and the convolutional layers and the at least one fully-connected layer are trained in different ways.

[0009] According to yet another aspect, a neuromorphic apparatus includes an on-chip memory including a plurality of cores, and a processor that drives a neural network by executing at least one program, wherein the processor trains each first layer included in the neural network using a first method, trains at least one second layer included in the neural network using a second method, and generates output data from input data using the trained first layer and the trained at least one second layer. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a biological neuron and a mathematical model that replicates the behavior of a biological neuron. [Figure 2] 1 is a diagram for explaining the configuration of a two-dimensional array circuit for performing neuromorphic operations. [Figure 3] 1 is a diagram for explaining the architecture of a neural network. [Figure 4] 1 is a diagram for explaining the relationship between an input feature map and an output feature map in a neural network. [Figure 5] FIG. 1 is a block diagram illustrating an example of a neuromorphic device, according to one embodiment. [Figure 6] 1 is a flowchart illustrating an example of a method for implementing a neural network, according to one embodiment. [Figure 7] 1 is a diagram illustrating an example of a neural network according to an embodiment. [Figure 8] 10 is a flowchart illustrating an example of a processor learning a first layer according to one embodiment. [Figure 9A] 10 is a diagram illustrating an example in which a processor generates a partial input vector according to an embodiment. [Figure 9B]10 is a diagram illustrating an example in which a processor generates a partial input vector according to an embodiment. [Figure 10] 10 is a diagram illustrating an example in which a processor learns a first layer according to one embodiment. [Figure 11] 10 is a diagram illustrating an example in which a processor generates an output feature map according to one embodiment. [Figure 12] 10 is a diagram illustrating an example in which a processor learns at least one second layer according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The terms used in this embodiment are generally used as much as possible, but they may vary depending on the intentions of those skilled in the art, legal precedents, or the emergence of new technologies. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the relevant description. Therefore, the terms used in this specification should be defined based on the meanings of the terms and the overall content of the specification, rather than simply by their names.

[0012] Throughout the specification, when a part "includes" a certain component, it does not mean that it excludes other components and may further include other components unless otherwise specified. Furthermore, terms such as "... unit" and "... module" used in the specification mean a unit that processes at least one function or operation, and may be realized by hardware or software, or by a combination of hardware and software.

[0013] Furthermore, terms including ordinal numbers such as "first" or "second" used in the specification may be used to describe various elements, but the elements are not limited by the terms. The terms are used to distinguish one element from another.

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and should not be construed as limited to the examples set forth herein.

[0015] Hereinafter, the present embodiment will be described in detail with reference to the drawings.

[0016] FIG. 1 is a diagram illustrating a biological neuron and a mathematical model that replicates the behavior of a biological neuron.

[0017] The biological neuron refers to a cell present in the human nervous system. The biological neuron is one of the fundamental biological computational entities. The human brain contains approximately 100 billion biological neurons and 100 trillion interconnections between the biological neurons.

[0018] Referring to Figure 1, a biological neuron 10 is a single cell. The biological neuron 10 includes a neuron cell body that contains a nucleus and various organelles, including mitochondria, numerous dendrites that emanate from the cell body, and an axon that terminates in many branching extensions.

[0019] Generally, the axon functions to transmit signals from one neuron to another, and the dendrites function to receive signals from other neurons. For example, when different neurons are connected to each other, a signal transmitted through one neuron's axon is also received by the dendrites of the other neurons. Signals are transmitted between neurons through specialized connections called synapses, and various neurons are connected to each other to form a neural network. Neurons that secrete neurotransmitters through synapses are called presynaptic neurons, and neurons that receive information transmitted through neurotransmitters are called postsynaptic neurons.

[0020] The human brain can learn and memorize vast amounts of information by transmitting and processing various signals through a neural network, which is formed by interconnecting numerous neurons. The vast number of connections between neurons in the human brain directly correlates with the massively parallel nature of biological computing. Various attempts have been made to replicate artificial neural networks and efficiently process vast amounts of information. For example, neuromorphic devices are being researched as computing systems designed to embody artificial neural networks at the neuron level.

[0021] The behavior of the biological neuron 10 is also replicated in the mathematical model 11. The mathematical model 11 corresponding to the biological neuron 10 is an example of a neuromorphic operation, which includes a multiplication operation for multiplying information from multiple neurons by synaptic weights, an addition operation (Σ) for the values ​​multiplied by the synaptic weights (ω0x0, ω1x1, ω2x2), and an operation for applying a characteristic function (b) and an activation function (f) to the result of the addition operation. The neuromorphic operation can provide a neuromorphic operation result. Here, values ​​such as x0, x1, x2, ... correspond to axon values, and values ​​such as ω0, ω1, ω2, ... correspond to synaptic weights.

[0022] FIG. 2 is a diagram for explaining the configuration of a two-dimensional array circuit for performing neuromorphic operations.

[0023] Referring to FIG. 2, the two-dimensional array circuit configuration 20 includes N (N is any natural number) axon circuits (A1 to A N ) 210, M (M is any natural number) neuron circuits (N1 to N M ) 230, and N*M synaptic arrays (S 11 Or S NM )220 included.

[0024] Synapse Array (S 11 Or S NM ) 220 synapses are connected to the axon circuit (A1 to A N ) 210 in the first direction, and a neuron circuit (N1 to N M ) 230 in the second direction and an intersection with a second direction line extending in the second direction. Here, for convenience of explanation, the first direction is illustrated as the row direction and the second direction is illustrated as the column direction, but the present invention is not limited thereto, and the first direction may be the column direction and the second direction may be the row direction.

[0025] Axonal circuit (A1 or A NEach of the axon circuits (A1 through A2) 210 may represent a circuit that replicates the axon of a biological neuron 10 (FIG. 1). The axon of a neuron performs the function of transmitting signals from the neuron to other neurons, and therefore, the axon circuits (A1 through A2) that replicate the axon of a neuron may represent a circuit that replicates the axon of a neuron. N ) 210, each of which corresponds to an activation (e.g., axons a1, a2, . . ., a N ) can be input and transmitted to the first direction line. The activation corresponds to the neurotransmitter transmitted through the neuron and is transmitted through the axon circuit (A1 to A N ) 210. On the other hand, the axon circuits (A1 to A N Each of the input / output circuits 210 may include a memory, register, or buffer for storing input information. Meanwhile, the activation may also be a binary activation having a binary value. For example, the binary activation may include one bit of information corresponding to a logical value of 0 or a logical value of 1. However, the activation is not necessarily limited thereto, and may also have a ternary value or a multi-bit value.

[0026] Synapse Array (S 11 Or S NM Each synapse in the synapse array (S) 220 may represent a circuit that replicates a synapse between neurons. 11 Or S NM ) 220 can store synaptic weights corresponding to the connection strength between neurons. For convenience of explanation, in FIG. 2, examples of synaptic weights stored at each synapse are w1, w2, . . . , w M Although the synapse is shown as a matrix, other synaptic weights may be stored in each synapse. 11 Or S NMEach synapse in the memory array 220 includes a memory element for storing synaptic weights or is coupled to other memory elements that store synaptic weights, where such memory elements may be, for example, memristors.

[0027] Synapse Array (S 11 Or S NM ) 220 are connected to the axon circuits (A1 to A N ) 210 and can output the result of a neuromorphic operation between the stored synaptic weights and the activation inputs. For example, the neuromorphic operation between the synaptic weights and the activation inputs can be, but is not limited to, a multiplication operation (i.e., an AND operation). That is, the result of the neuromorphic operation between the synaptic weights and the activation inputs can be a value obtained by any other suitable operation to replicate the strength or magnitude of activation as modulated by the strength of connections between neurons.

[0028] The axon circuit (A1 to A2) is generated by the neuromorphic operation of the synaptic weights and the activation input. N ) 210 to neuron circuit (N1 to N M ) 230 can regulate the magnitude or strength of the signal transmitted through the synaptic array (S 11 Or S NM ) 220, the strength of the connection between neurons is used to replicate the operation of adjusting the magnitude or strength of the signal transmitted to the next neuron.

[0029] Neuron circuit (N1 or N M Each of the neuronal circuits (N1 through N2) 230 may represent a circuit that replicates a neuron, including its dendrites. The dendrites of a neuron perform the function of receiving signals from other neurons. MEach of the neuron circuits (N1 to N2) 230 can receive the result of a neuromorphic operation between the synaptic weights and the activation inputs via a corresponding second direction line. M ) 230 can decide whether to output a spike or not based on the result of the neuromorphic operation. M Each neuron circuit (N1 to N2) can output a spike when the accumulated value of the neuromorphic operation is equal to or greater than a predetermined threshold. M The spikes output from 230 also correspond to activations input to the next axon circuit.

[0030] In addition, neuron circuits (N1 to N M )230 is a synaptic array (S 11 Or S NM ) 220, but is also called the post-synaptic neuron circuit, and is located at the posterior end of the axon circuit (A1 or A N ) 210 is a synaptic array (S 11 Or S NM ) 220, but is also called the pre-synaptic neuron circuit.

[0031] FIG. 3 is a diagram for explaining the architecture of a neural network.

[0032] Referring to FIG. 3, the neural network 30 is also an architecture of a deep neural network (DNN) or n-layer neural network. The DNN or n-layer neural network also corresponds to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, etc. For example, the neural network 30 may be embodied as a convolutional neural network (CNN), but is not limited thereto. Although FIG. 3 illustrates some convolutional layers constituting a convolutional neural network corresponding to an example of the neural network 30, the convolutional neural network may further include a pooling layer, a fully-connected layer, etc. in addition to the illustrated convolutional layers.

[0033] The neural network 30 may also be implemented as an architecture having multiple layers including input data, feature maps, and output data. In the neural network 30, the input data is convolved with a kernel to generate an output feature map. The generated output feature map is then used as the input feature map for the next layer and is further convolved with the kernel to generate a new output feature map. As a result of repeated convolution operations, a recognition result related to the features of the input data passed through the neural network 30 is finally output.

[0034] For example, when a 24*24 pixel image is input to the neural network 30 of FIG. 3, the input image undergoes a convolution operation with a kernel and is output as a 4-channel feature map having a size of 20*20. The 20*20 feature map then undergoes repeated convolution operations with the kernel, gradually decreasing in size until a 1*1 feature is finally output. The neural network 30 repeatedly performs convolution operations and subsampling (or pooling) operations in various layers to kernel and output robust features that can represent the entire image from the input image, and a recognition result for the input image can be derived from the output final features.

[0035] FIG. 4 is a diagram for explaining the relationship between an input feature map and an output feature map in a neural network.

[0036] Referring to FIG. 4, in a layer 40 of a neural network, a first feature map FM1 also corresponds to an input feature map, and a second feature map FM2 also corresponds to an output feature map. The feature map may refer to a data set representing various features of input data. The feature maps FM1 and FM2 may have elements of a two-dimensional matrix or elements of a three-dimensional matrix, and pixel values ​​may be defined for each element. The feature maps FM1 and FM2 have a width W (also referred to as columns), a height H (also referred to as rows), and a depth D. In this case, the depth D may correspond to the number of channels.

[0037] A convolution operation is performed on the first feature map FM1 and a kernel, resulting in a second feature map FM2. The kernel, which has weights defined for each element, is convolved with the first feature map FM1 to kernelize the features of the first feature map FM1. The kernel is convolved with a window (also called a tile) of the first feature map FM1 while shifting the first feature map FM1 using a sliding window method. During each shift, each weight included in the kernel is multiplied by each pixel value of the overlapped window in the first feature map FM1 and then added. The convolution of the first feature map FM1 with the kernel generates one channel of the second feature map FM2. Although FIG. 4 illustrates one kernel, multiple kernels may be convolved with the first feature map FM1 to generate a second feature map FM2 with multiple channels.

[0038] The second feature map FM2 may also serve as an input feature map for the next layer, for example, a pooling (or subsampling) layer.

[0039] 3 and 4, for ease of explanation, only a schematic architecture of the neural network 30 is illustrated. However, those skilled in the art will understand that the neural network 30 may be implemented with a greater or lesser number of layers, feature maps, kernels, etc. than those illustrated, and that the size of the neural network 30 may also be varied in various ways.

[0040] As described with reference to Figures 3 and 4, a typical convolutional neural network involves numerous MAC (multiply and accumulate) operations. Specifically, a typical convolutional neural network includes tens to hundreds of layers, and a large number of MAC operations must be performed to generate output data through the convolutional neural network. This has led to a growing demand for technology (weight reduction technology) that reduces the amount of calculation required by devices implementing convolutional neural networks.

[0041] As such weight reduction techniques, pruning, which removes neurons or connections that have little impact on the final output data, and weight matrix decomposition, which replaces the weight matrix of each layer with the product of multiple small matrices, have been proposed. Furthermore, QQuantized Neural Networks, Ternary Neural Networks, and Binary Neural Networks, which reduce the bit-precision of parameters (e.g., weights and activations) of each layer, have also been proposed. However, the above-mentioned weight reduction techniques have the problem of reducing the accuracy of the final output data.

[0042] Also, back propagation has been proposed as a common neural network training method. However, with back propagation, the closer to the initial layer of the neural network, the closer the gradient becomes to 0 (i.e., gradient vanishing). Since weight updates in back propagation depend on the gradient, the learning effect of the neural network is reduced.

[0043] A neural network according to one embodiment is proposed as a model with a lower computational load and a higher learning efficiency than existing neural networks. Specifically, the neural network according to one embodiment performs autonomous or unsupervised learning using a self-organizing map for at least one layer. This eliminates the gradient vanishing problem associated with backpropagation, improving learning efficiency. Furthermore, at least one layer of the trained neural network generates output feature map data based on a self-organizing map. Therefore, the neural network according to one embodiment can generate output feature map data using only addition and subtraction instead of MAC operations, while significantly reducing the computational load.

[0044] A neural network and a neuromorphic apparatus embodying the neural network according to one embodiment will now be described with reference to FIGS. 5 to 12. FIG.

[0045] FIG. 5 is a block diagram illustrating an example of a neuromorphic device, according to one embodiment.

[0046] 5, a neuromorphic device 500 may include a processor 510 and an on-chip memory 520. Only components related to the present embodiment are shown in the neuromorphic device 500 shown in FIG. 5. Therefore, it will be apparent to those skilled in the art that the neuromorphic device 500 may further include other general-purpose components in addition to the components shown in FIG.

[0047] The principle of the neuromorphic device 500 is the same as that described with reference to Figures 1 and 2. Therefore, even if omitted below, the contents described with reference to Figures 1 and 2 also apply to the neuromorphic device 500 of Figure 5.

[0048] The neuromorphic device 500 may also be incorporated into digital systems requiring low-power neural network operation, such as, but not limited to, smartphones, drones, tablet devices, AR (augmented reality) devices, IoT (internet of things) devices, autonomous vehicles, robotics, and medical devices.

[0049] The neuromorphic device 500 may include multiple on-chip memories 520, each of which may also be comprised of multiple cores. The cores may include multiple presynaptic cells, multiple postsynaptic cells, and synapses, i.e., memory cells, providing connections between the multiple presynaptic cells and the multiple postsynaptic cells. In one embodiment, the cores may also be embodied as resistive crossbar memory arrays (RCA).

[0050] The external memory 530 is hardware that stores various data processed by the neuromorphic device 500, and can store data that has been processed by the neuromorphic device 500 and data to be processed by the neuromorphic device 500. The external memory 530 can also store applications, drivers, etc. that are run by the neuromorphic device 500. The external memory 530 may include random access memory (RAM) such as dynamic random access memory (DRAM) and static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), compact disc read only memory (CD-ROM), Blu-ray or other optical disc storage, hard disk drive (HDD), solid static drive (SSD), or flash memory.

[0051] The processor 510 controls the overall functions for operating the neuromorphic device 500. For example, the processor 510 controls the overall operation of the neuromorphic device 500 by executing a program stored in the on-chip memory 520 within the neuromorphic device 500. The processor 510 may also be implemented as an array of multiple logic gates, or as a combination of a general-purpose microprocessor and a memory storing a program executable by the microprocessor. Those skilled in the art will understand that the processor 510 may also be implemented as other types of hardware.

[0052] The processor 510 may be implemented by, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc., provided within the neuromorphic device 500. The processor 510 reads / writes various data from the external memory 530 and executes the neuromorphic device 500 using the read / written data.

[0053] Hereinafter, with reference to FIGS. 6 to 12, an example of the operation of the processor 510 will be specifically described.

[0054] FIG. 6 is a flowchart illustrating an example of a method for implementing a neural network, according to one embodiment.

[0055] Referring to Figure 6, the method for implementing a neural network comprises steps that are processed in time sequence in the processor 510 shown in Figure 5. Therefore, even if omitted below, it will be understood that the content described above with respect to the processor 510 of Figure 5 also applies to the method for implementing a neural network of Figure 6.

[0056] In step 610, the processor 510 trains each first layer included in the neural network using a first method.

[0057] According to one embodiment, the neural network includes multiple layers, which may also be represented by a first layer and a second layer. The processor 510 trains the first layer and the second layer using different methods. For example, the processor 510 may train the first layer using a first method and train the second layer using a second method.

[0058] An example of a neural network according to one embodiment will now be described with reference to FIG.

[0059] FIG. 7 is a diagram illustrating an example of a neural network according to an embodiment.

[0060] 7, the neural network 70 includes a first layer 710 and at least one second layer 720. For example, the first layer 710 may be a convolutional layer, and the second layer 720 may be a fully connected layer, but is not limited thereto. Also, the at least one second layer 720 may be located at the rear end of the first layer 710.

[0061] For example, the first layer 710 is a layer that extracts features from input data 730, and the second layer 720 is a layer that performs classification and identification based on output feature map data (OFM) 732.

[0062] The neural network 70 may also be configured with only the first layer 710, without the second layer 720. If the neural network 70 does not include the second layer 720, the learning by the second method described below may be omitted.

[0063] Input data 730 is input to the neural network 70, and output data 740 is ultimately generated. In addition, output feature map data 731 and 732 are generated by layers 710 and 720 that make up the neural network 70, respectively. An example of the operation of the neural network 70 has been described with reference to Figures 3 and 4. Therefore, a detailed description of the neural network 70 will be omitted below.

[0064] The processor 510 trains each of the first layers 710 using a first method. For example, the processor 510 trains the first layer (convolutional layer #1) 711 included in the first layer 710 using input data 730. Then, the processor 510 trains the layer next to the first layer 710 using output feature map data 731 of the trained first layer 711. Through this process, the processor 510 can train the first layer 710. Meanwhile, if necessary, a pooling layer 712 may be further included between each layer included in the first layer 710, and the pooling layer 712 may manipulate the output feature map data 731 according to a predetermined criterion. Also, if necessary, the layers included in the first layer 710 and the pooling layer 712 may be connected to each other in series or in parallel.

[0065] The first method is also related to autonomous learning. For example, the processor 510 may learn the first layer 710 using a self-organizing map, but is not limited thereto. Here, autonomous learning refers to a method of learning based on an input pattern without a target pattern. That is, autonomous learning refers to learning that occurs when only input data is provided as learning data without output data.

[0066] The self-organizing map is a neural network model used for data clustering and visualization of the results. The self-organizing map consists of an input layer with nodes whose number is equal to the dimension of the input data, and an output layer with neurons whose number is equal to the number of clustering target classes. Each neuron in the output layer has a weight represented by a vector with the same dimension as the input data. The similarity between each neuron's weight and the input data is calculated, and the input data is classified and clustered with the most similar neuron in the output layer. The neurons in the output layer are arranged in a one-, two-, or three-dimensional structure. During the learning process, not only are the weights of the neurons updated, but similar neurons are also arranged adjacent to each other. This characteristic of the output layer (i.e., the tendency for similar neurons to be arranged adjacent to each other) is effective for visualizing the clustering results.

[0067] The manner in which the processor 510 learns each of the first layers 710 is described below with reference to Figures 8 to 11 .

[0068] The processor 510 trains at least one second layer 720 using a second method. For example, the processor 510 can train the second layer 720 using output feature map data 732 of the final layer (convolutional layer #N) 713 included in the first layer 710.

[0069] The second method is also related to supervised learning. For example, the processor 510 may train the second layer 720 using a backpropagation method, but is not limited thereto. Here, the supervised learning refers to learning performed in a situation where input data and output data based on the input data are provided as training data.

[0070] The manner in which the processor 510 trains the at least one second layer 720 is described below with reference to FIG.

[0071] The processor 510 may train the neural network 70 using multiple methods, thereby improving the effectiveness of training the neural network 70. In particular, the processor 510 may train the first layer 710 of the neural network 70 using autonomous learning, thereby eliminating the problem of training using the backpropagation method (i.e., gradient vanishing).

[0072] The neural network 70 may also include a first layer 710 based on a self-organizing map. Here, the output feature map data of each first layer 710 is generated without a MAC operation. This can solve the problem of an increased computational load caused by a MAC operation in an existing convolutional layer.

[0073] Referring back to FIG. 6, in step 620, the processor 510 trains at least one second layer included in the neural network using a second method.

[0074] The processor 510 can use the output feature map data 732 of the final layer 713 included in the first layer 710 of the neural network 70 to train at least one second layer 720. For example, the processor 510 can train at least one second layer 720 by a backpropagation method, but is not limited to this.

[0075] In step 630, the processor 510 generates output data from input data using the trained first layer and at least one trained second layer.

[0076] Specifically, the processor 510 applies input data 730 to the trained first layer 710 to generate output feature map data 732. The processor 510 then applies the output feature map data 732 to at least one trained second layer 720 to generate output data 740.

[0077] FIG. 8 is a flowchart illustrating an example in which a processor learns a first layer according to one embodiment.

[0078] 8, the processor 510 sequentially and independently learns each of the first layers 710. For example, if the first layer 710 includes L layers (where L is a natural number), the processor 510 independently learns each layer in the order of the first layer 711, the next layer, ..., and the final layer 713 (i.e., the Lth layer).

[0079] In step 810, the processor 510 determines whether the current layer is the initial layer 711 in the first layer 710. If the current layer is the initial layer 711, the process proceeds to step 820; otherwise, the process proceeds to step 860.

[0080] In operation 820, the processor 510 generates partial input vectors using input data 730 of the first layer 711. Here, the input data 730 refers to data that is first input to the neural network 70 and is used for training the first layer 710. As described with reference to FIG. 7, the neural network 70 is configured in the order of the first layer 710 → the second layer 720. Therefore, the processor 510 can train the first layer 711 using the input data 730. Hereinafter, an example of the processor 510 generating partial input vectors will be described with reference to FIGS. 9A and 9B.

[0081] 9A and 9B are diagrams illustrating an example in which a processor generates a partial input vector according to one embodiment.

[0082] 9A illustrates a first set 910 included in input data 900, and FIG. 9B illustrates all sets 910, 920, and 930 that make up the input data 900. For ease of explanation, it is assumed that each of the sets 910, 920, and 930 has a pixel size of N*M (where N and M are natural numbers) and a channel size of C (where C is a natural number). It is also assumed that the input data 730 is composed of a total of Q sets 910, 920, and 930 (where Q is a natural number).

[0083] 9A, the processor 510 scans the first set 910 by moving up, down, left, and right by one pixel using a scan window 911. Here, it is assumed that the scan window 911 has a pixel size of K*K (where K is a natural number) and a channel size of C.

[0084] Therefore, there are a total of K*K*C pixels in each scan region, and the values ​​stored in the K*K*C pixels become elements of the partial input vector. In other words, the processor 510 stores the K*K*C elements P1, P2, P3, . . . , P K*K*C A partial input vector can be generated that includes:

[0085] The processor 510 generates a K*K*C dimensional partial input vector for each scan region included in the first set 910. Therefore, when the processor 510 scans the entire region of the first set 910 in the scan window 911, a total of M*N partial input vectors V1, V2, . . . , V M*N is generated.

[0086] 9B, input data 900 includes a total of Q sets 910, 920, and 930. Therefore, when processor 510 scans input data 900 with scan window 911, processor 510 scans a total of M*N*Q partial input vectors V1, V2, . . . , V M*N*Q can be generated.

[0087] The process of generating the partial input vectors for the next layer by the processor 510, which will be described later with reference to step 860, is the same as that described with reference to FIGS. 9A and 9B.

[0088] Referring again to FIG. 8, in step 830, the processor 510 uses a self-organizing map to train the first layer 711 based on the partial input vector.

[0089] The processor 510 can cluster the partial input vectors using a self-organizing map. For example, the processor 510 can search for an output neuron having a weight most similar to the partial input vector. The processor 510 can then update the weight of at least one neuron located within a predetermined range around the searched output neuron. Hereinafter, with reference to FIG. 10 , an example of the processor 510 learning the first layer 711 will be described.

[0090] FIG. 10 is a diagram illustrating an example in which a processor learns a first layer according to one embodiment.

[0091] 10 illustrates the structure of a self-organizing map 1000 for learning the first layer. Specifically, the self-organizing map 1000 is composed of an input layer 1010 and an output layer 1020. The input layer 1010 has input nodes I1, I2, I3, . . . , I, which are the same number as the dimensions of the partial input vector 1030. K*K*C Includes:

[0092] The output layer 1020 has the same number of output neurons O as the number of clustering target classes. 11 ,O 12 ,O RR In FIG. 10, for convenience of explanation, the number of clustering target classes is illustrated as RR, but it is not limited to this. 11 ,O 12 ,O RR Each performs the same operation as a single kernel in a convolutional layer.

[0093] Also, the output neuron O 11 ,O 12 ,O RR Each node is connected to all input nodes I1, I2, I3,...,I K*K*C and the connected weight W j:r1r2 It has the following.

[0094] The processor 510 searches for an output neuron having a weight most similar to the partial input vector 1030. Then, the processor 510 can learn the first layer by updating the weight of at least one neuron located within a predetermined range around the searched output neuron. For example, the processor 510 may input a total of M*N*Q partial input vectors V included in the partial input vector 1030 into the self-organizing map 1000. i By inputting the partial input vectors 1030 one by one, the first layer can be trained so that the partial input vectors 1030 are clustered. For example, the processor 510 can train the first layer using the following Equations 1 to 6.

[0095] First, the processor 510 can calculate the similarity between the partial input vector and the connection weight using the following Equation 1.

[0096] Formula 1

[0097]

number

[0098] The similarity (E r1r2 ) is calculated, the processor 510 can calculate the coordinates (win1, win2) of the output neuron that is most similar to the partial input vector using the following Equation 2.

[0099] Formula 2

[0100]

number

[0101] Thereafter, the processor 510 can update the weight of at least one neuron located within a predetermined range around the searched output neuron according to the following Equations 3 to 6.

[0102] For example, the processor 510 may use the following Equation 3 to configure the output neurons O such that the output layer 1020 more closely resembles the partial input vectors: 11 ,O 12 ,ORR At least one of them can be updated.

[0103] Equation 3

[0104]

number

[0105] Equation 4

[0106]

number

[0107] In addition, in the above formula 3, ∂(t) is the output neuron O 11 ,O 12 ,O RR In the above equation, the range of output neurons whose weights are updated is determined by the following equations 5 and 6.

[0108] Formula 5

[0109]

number

[0110]

number

[0111] According to the above-mentioned formulas 3 to 6, the output neuron O 11 ,O 12 ,O RR are also arranged so that those with similar properties are adjacent to each other.

[0112] The output neuron O that has completed learning according to the above-mentioned formulas 1 to 6 11 ,O 12 ,O RR is the connection weight W corresponding to the main pattern of the partial input vector 1030. j:r1r2 , and output neurons corresponding to similar patterns are arranged adjacent to each other. Thus, processor 510 completes learning for first layer 711.

[0113] The process of learning the next layer by the processor 510, which will be described later with reference to step 870, is the same as that described with reference to FIG.

[0114] Referring again to FIG. 8, in step 840, the processor 510 generates output feature map data 731 for the trained first layer 711.

[0115] The processor 510 uses input data 730 to generate partial input data. Here, the input data 730 refers to data that is initially input to the neural network 70 and is used to generate output feature map data 731. The processor 510 can then generate the output feature map 731 by calculating the similarity between the partial input vector and the updated weights. An example of how the processor 510 generates the output feature map 731 will be described below with reference to FIG. 11 .

[0116] FIG. 11 is a diagram illustrating an example of a processor generating an output feature map according to one embodiment.

[0117] Figure 11 illustrates an example of output feature map data 1100. For ease of explanation, it is assumed that the input data from which the output feature map data 1100 of Figure 11 is generated is the example of Figure 9A.

[0118] The processor 510 uses the input data to generate partial input vectors. In the example of FIG. 9A, the processor 510 generates a total of M*N partial input vectors V1, V2, . . . , V M*N can be generated.

[0119] The processor 510 calculates the partial input vectors V1, V2, . . . , V M*N For example, the processor 510 can calculate the similarity between the partial input vector and the updated connection weight using the following Equation 7.

[0120] Formula 7

[0121]

number

[0122] According to the above-mentioned Equation 7, the processor 510 calculates the partial input vectors V1, V2, . . . , V M*N The processor 510 can calculate the similarity between the partial input vectors V1, V2, . . . , V M*N and a similarity vector S that indicates the similarity between the updated output neuron. i S i =[E 11 ,E 12 ,···,E 1R ,E 21 ,E 22 ,···,E RR ] can be constructed as follows.

[0123] When each partial input vector passes through the self-organizing map, a similarity vector S of the same dimension as the number of output neurons is generated. i The processor 510 generates the affinity vector S i is treated as the partial output feature map data S1 with R*R channels and 1*1 pixel size.

[0124] Through the above process, the processor 510 generates the same output feature map data S1, S2, . . . , S from all the partial input vectors. M*Nand generate output feature map data S1,S2,…,S M*N with the coordinates of the input data from which the partial input vectors were generated, allowing the processor 510 to generate output feature map data 1100 with R*R channels and M*N pixel size, which is used as the input feature map data for the next layer.

[0125] The process of generating output feature map data for the next layer by the processor 510, which will be described later with reference to step 880, is the same as that described with reference to FIG.

[0126] 8, in step 850, processor 510 determines whether a next layer exists in first layer 710. Here, the next layer refers to a layer that needs to be learned in first layer 710. If a next layer exists, the process proceeds to step 860; otherwise, the process ends learning of first layer 710.

[0127] In steps 860 to 880, the processor 510 learns the next layer. In other words, the processor 510 learns the layer next to the layer whose learning has been completed (i.e., the layer whose input feature map data is the output feature map data of the layer whose learning has been completed).

[0128] Specifically, the detailed processes of steps 860 to 880 are the same as the detailed processes of steps 820 to 840. Therefore, detailed description of steps 860 to 880 will be omitted below.

[0129] As described with reference to FIGS. 8 to 11, the processor 510 learns all layers included in the first layer 710 in order and independently.

[0130] As described with reference to step 620, the processor 510 may train at least one second layer 720 using the backpropagation method. Hereinafter, an example of the processor 510 training at least one second layer 720 will be described with reference to FIG.

[0131] FIG. 12 is a diagram illustrating an example in which a processor learns at least one second layer according to one embodiment.

[0132] 12 illustrates an example of at least one second layer 1220 included in the neural network 1200. The processor 510 uses the final output feature map data 1230 of the first layer 1210 to train the at least one second layer 1220.

[0133] For example, the processor 510 may train at least one second layer 1220 using a backpropagation method. For convenience of explanation, the final output feature map data 1230 may be represented as a set of activations i 0 , . . . ,i n Also, at least one second layer 1220 includes multiple layers, and activations o0, . . . , o m Assume that is output.

[0134] Activation o0,···,o m When generated, activation o0,···,o m can be compared with the expected result to generate an error δ. For example, the error δ is the difference between the expected result and the activations o0, ,o m Therefore, the learning of the neural network 1200 must be performed in a direction that reduces the error δ.

[0135] In order for the error δ to be small, the final error δ0, ,δ mThe activations used in the intermediate operations already performed must be updated by propagating δ in the opposite direction to the forward propagation direction (i.e., backpropagating). For example, the final error δ0, , δ m is calculated by the weight and the intermediate error δ (1,0) ,···,δ (1,l) is generated. The intermediate error δ (1,0) ,···,δ (1,l) becomes the input for generating the intermediate error of the next layer, and the above operation is performed again. Through this process, the error δ is propagated in the opposite direction to the forward propagation direction, and the activation gradient used for activation update is calculated.

[0136] The backpropagation process described above can be expressed in mathematical terms as shown in Equation 8 below.

[0137] Formula 8

[0138]

number

[0139] As described above, once backpropagation has been performed for all second layers 1220, the weights are updated based on the results of the backpropagation. Specifically, the activation gradients calculated by backpropagation are used to calculate the weight gradients used for weight updating. The weight update can be expressed as Equation 9 below.

[0140] Formula 9

[0141]

number

[0142] Through the backpropagation and weight updates described above, the second layer 1220 is learned.

[0143] Based on the foregoing, the neural networks 70, 1200 and the neuromorphic device 500 embodying the neural networks 70, 1200 according to one embodiment have the following advantages:

[0144] First, the convolutional layer included in a conventional convolutional neural network operates based on MAC operations between input data and weights. However, the neural network 70, 1200 includes a first layer 710, 1210 based on a self-organizing map, which can operate using only addition and subtraction. Therefore, the neural network 70, 1200 can reduce the number of bit-wise operations (i.e., the amount of operations) required on digital computing hardware, as shown in Table 1 below.

[0145] Table 1

[0146] [Table 1] According to Table 1 above, the conditions (layer-related parameters) of the conventional convolutional layer and the first layers 710 and 1210 are the same. In this case, if the bit precision of the pixel values ​​of the input data and the kernel weights (weights of the output neurons) is 32 bits, the first layers 710 and 1210 each have a 1 / 32 times reduction in the amount of calculation compared to the conventional convolutional layer.

[0147] The computational load of some representative models of conventional convolutional neural networks is summarized in Table 2 below.

[0148] Table 2

[0149] [Table 2] According to Table 2 above, in conventional convolutional neural networks, the computational load of the convolutional layer accounts for approximately 83 to 99% of the computational load of the entire neural network. In other words, most of the computational load generated in a convolutional neural network is accounted for by the computational load of the convolutional layer. As shown in Table 1 above, while the computational load of each of the first layers 710 and 1210 is significantly less than that of a conventional convolutional layer, the total computational load of the neural network 70 or 1200 is significantly less than that of a conventional convolutional neural network.

[0150] Furthermore, conventional convolutional neural networks (i.e., convolutional neural networks based on MAC operations) require the calculation of an activation function such as a ReLu function, a Sigmoid function, or a tanh function. However, the neural networks 70, 1200 do not require the calculation of an activation function. Therefore, the neural networks 70, 1200 not only further reduce the amount of calculation, but also greatly facilitate the implementation of dedicated hardware (e.g., neuromorphic device 500) that drives the neural networks 70, 1200.

[0151] In addition, conventional convolutional neural networks perform learning based on a backpropagation method, which sequentially updates the weights of each layer from the final layer to the first layer. In this case, the closer to the first layer, the smaller the amount of weight update. Therefore, in the case of a convolutional neural network that includes tens or hundreds of layers, there is a problem in that the frontmost layers (i.e., layers close to the first layer) are hardly trained. As a result, the accuracy of classification and recognition by the convolutional neural network varies depending on how the initial weights are set.

[0152] However, the neural network 70, 1200 is independent from the first layer 710, 1210 to the last layer and undergoes sufficient training, so that all layers included in the neural network 70, 1200 can effectively extract features and perform classification and recognition. Therefore, the neural network 70, 1200 has the advantage of improving the accuracy of results compared to conventional convolutional neural networks.

[0153] Furthermore, because the first layer 710, 1210 is trained using a self-organizing map, output neurons are arranged so that neurons with similar properties are adjacent to each other. Therefore, if the neural network 70, 1200 includes a pooling layer, channel-wise pooling is possible. However, conventional convolutional neural networks do not support channel-wise pooling. Therefore, the output feature map data of the neural network 70, 1200 can be smaller in data size than the output feature map data of conventional convolutional neural networks. This can reduce the overall size of the neural network 70, 1200 model.

[0154] Furthermore, the first layer 710, 1210 is trained using a self-organizing map, and even if the weights and output feature map data of the first layer 710, 1210 are converted to binary, the classification accuracy is superior to that of a conventional convolutional neural network. Therefore, the neural network 70, 1200 can reduce the amount of calculation and the overall model size while maintaining a high level of classification accuracy.

[0155] The above-described method can be created as a computer-executable program and can be implemented by a general-purpose digital computer that runs the program using a computer-readable recording medium. The data structure used in the above-described method can also be recorded on a computer-readable recording medium by various means. Examples of computer-readable recording media include magnetic recording media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optically readable media (e.g., CD-ROM, DVD (digital versatile disc)).

[0156] Those skilled in the art in the art to which the present invention pertains will understand that the present invention may be embodied in various modified forms without departing from the essential characteristics described above. Therefore, the disclosed method should be considered from an illustrative rather than a restrictive perspective, and the scope of protection is defined in the claims, not the above description, and should be interpreted to include all variations within the scope of equivalents thereto. [Explanation of symbols]

[0157] 500 Neuromorphic Devices 510 processor 520 On-chip Memory 530 external memory

Claims

1. A method for implementing a neural network, comprising: training each first layer included in the neural network using a first method relating to unsupervised learning using a self-organizing map; training at least one second layer included in the neural network using a second method involving supervised learning by backpropagation; generating output data from input data using the trained first layer and the trained at least one second layer; The method, wherein the first layers include convolutional layers and the at least one second layer includes a fully connected layer.

2. The step of learning by the first method includes: generating a partial input vector in the first layer using input data of the first layer; learning the first layer based on the partial input vectors using the self-organizing map; and utilizing the learned first layer to generate output feature map data for the first layer.

3. The step of learning the first layer includes: searching for an output neuron having a weight most similar to the partial input vector; 3. The method of claim 2, further comprising updating the weight of at least one neuron located within a predetermined range around the retrieved output neuron.

4. The step of generating output feature map data of the first layer comprises: generating a partial input vector using the input data; and calculating a similarity between the partial input vector and the updated weight.

5. The method of claim 4 , further comprising: using the output feature map data of the first layer to train a layer subsequent to the first layer.

6. The step of generating output data comprises: generating output feature map data by applying the input data to the trained first layer; and generating output data by applying the output feature map data to the trained at least one second layer.

7. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to any one of claims 1 to 6.

8. In a neuromorphic device, an on-chip memory containing multiple cores; a processor that executes at least one program to drive the neural network; The processor: training each first layer included in the neural network using a first method related to unsupervised learning using a self-organizing map, training at least one second layer included in the neural network using a second method related to supervised learning using backpropagation, and generating output data from input data using the trained first layer and the trained at least one second layer; The neuromorphic device, wherein the first layer includes convolutional layers and the at least one second layer includes a fully connected layer.

9. The processor:

9. The neuromorphic device of claim 8, further comprising: generating partial input vectors using input feature map data of an initial layer among the first layers; learning the initial layer based on the partial input vectors using the self-organizing map; and generating output feature map data of the initial layer using the learned initial layer.

10. The processor:

10. The neuromorphic device according to claim 9, wherein an output neuron having a weight most similar to the partial input vector is searched for, and the weight of at least one neuron located within a predetermined range centered on the searched output neuron is updated.

11. The processor: The neuromorphic device of claim 10 , further comprising: generating a partial input vector using the input data; and calculating a similarity between the partial input vector and the updated weights.

12. The processor: The neuromorphic device according to claim 8 , wherein the output feature map data of the first layer is used to train the layer next to the first layer.

13. The processor:

13. The neuromorphic device of claim 12, wherein output feature map data is generated by applying the input data to the trained first layer, and output data is generated by applying the output feature map data to the trained at least one second layer.

Images