Apparatus for implementing neural network and operation method thereof

Abstract

A neuromorphic device implementing a neural network may include a memory in which at least one program is stored, an on-chip memory including a crossbar array circuit, and at least one processor that drives the neural network by executing at least one program. At least one processor can store binary weight values ​​in synapse circuits included in the crossbar array circuit, obtain an input feature map from memory, and convert the input feature map into time domain binary vectors. Additionally, at least one processor can provide the time domain binary vectors as input values ​​to the crossbar array circuit and output an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors.

Description

Technology Field

[0001] This invention relates to a device for implementing a neural network and a method of operation thereof. Background Technology

[0002] Memory-centric neural network devices reference a computational architecture modeled after the biological brain. As memory-centric neural network technology advances, active research is underway to utilize these networks in various types of electronic systems to analyze input data and extract useful information.

[0003] Therefore, in order to analyze large amounts of input data in real time using memory-centric neural networks and extract desired information, technology capable of efficiently processing computations is required. The problem to be solved

[0004] The invention provides an apparatus and method for generating chemical structures using a neural network. Additionally, the invention provides a computer-readable recording medium storing a program for executing the above method on a computer. The technical problems to be solved are not limited to those described above, and other technical problems may exist. means of solving the problem

[0005] As a technical means for achieving the technical problem described above, the first aspect of the present disclosure may be a neuromorphic device for implementing a neural network, comprising: a memory in which at least one program is stored; an on-chip memory including a crossbar array circuit; and at least one processor for driving the neural network by executing the at least one program, wherein the at least one processor stores binary weight values ​​in synapse circuits included in the crossbar array circuit, obtains an input feature map from the memory, converts the input feature map into temporal domain binary vectors, provides the temporal domain binary vectors as input values ​​to the crossbar array circuit, and outputs an output feature map by performing a convolution operation between the binary weight values ​​and the temporal domain binary vectors.

[0006] A second aspect of the present disclosure provides a neural network device for implementing a neural network, comprising: a memory in which at least one program is stored; and at least one processor for driving the neural network by executing the at least one program, wherein the at least one processor obtains binary weight values ​​and an input feature map from the memory, converts the input feature map into time domain binary vectors, and outputs an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors.

[0007] A third aspect of the present disclosure provides a method for implementing a neural network in a neuromorphic device, comprising: storing binary weight values ​​in synapse circuits included in a crossbar array circuit; obtaining an input feature map from memory; converting the input feature map into time domain binary vectors; providing the time domain binary vectors as input values ​​to the crossbar array circuit; and outputting an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors.

[0008] A fourth aspect of the present disclosure may provide a method for implementing a neural network in a neural network device, comprising: obtaining binary weight values ​​and an input feature map from memory; converting the input feature map into time domain binary vectors; and outputting an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors.

[0009] A fifth aspect of the present disclosure may provide a computer-readable recording medium having a program for executing the methods of the third and fourth aspects on a computer. Effects of the invention

[0010] According to the problem-solving means of the present disclosure described above, by using binary weight values ​​and time domain binary vectors, the model size and amount of computation can be reduced.

[0011] In addition, according to one of the other means for solving the problem of the present disclosure, by performing a time-axis XNOR operation between a binary weight value and a time-domain binary vector, it is possible to secure learning performance and final classification / recognition accuracy at a level similar to that of a neural network using multi-bit data. Brief explanation of the drawing

[0012] Figure 1 is a diagram illustrating a biological neuron and a mathematical model that simulates the operation of the biological neuron. FIGS. 2a and 2b are drawings for explaining the operation method of a neuromorphic device according to one embodiment. FIGS. 3a and 3b are drawings for comparing vector-matrix multiplication and operations performed on a crossbar array according to one embodiment. FIG. 4 is a diagram illustrating an example in which a convolution operation is performed in a neuromorphic device according to one embodiment. FIG. 5 is a diagram illustrating operations performed in a neural network according to one embodiment. FIGS. 6a to 6c are drawings for illustrating an example of converting initial weight values ​​into binary weight values ​​according to one embodiment. FIGS. 7a and 7b are drawings for illustrating an example of converting an input feature map into time domain binary vectors according to one embodiment. FIG. 8 is a diagram illustrating the application of binary weight values ​​and time domain binary vectors to a batch normalization process according to one embodiment. FIG. 9 is a block diagram of a neural network device using a von Neumann structure according to one embodiment. FIG. 10 is a block diagram of a neuromorphic device using an in-memory structure according to one embodiment. FIG. 11 is a flowchart illustrating a method for implementing a neural network in a neural network device according to one embodiment. FIG. 12 is a flowchart illustrating a method for implementing a neural network in a neuromorphic device according to one embodiment. FIG. 13 is a block diagram illustrating the hardware configuration of a neural network device according to one embodiment. FIG. 14 is a block diagram illustrating the hardware configuration of a neuromorphic device according to one embodiment. Specific details for implementing the invention

[0013] Phrases such as "in some embodiments" or "in one embodiment" appearing in various places in this specification do not necessarily refer to the same embodiment.

[0014] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0015] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0016] The present disclosure will be described in detail below with reference to the attached drawings.

[0017] Figure 1 is a diagram illustrating a biological neuron and a mathematical model that simulates the operation of the biological neuron.

[0018] Biological neurons refer to cells present in the human nervous system. Biological neurons are one of the basic biological computational entities. The human brain contains approximately 100 billion biological neurons and about 100 trillion interconnections between biological neurons.

[0019] Referring to FIG. 1, the biological neuron (10) is a single cell. The biological neuron (10) comprises a cell body (neuron cell body) containing a nucleus and various organelles. The various organelles include mitochondria, a number of dendrites radiating from the cell body, and an axon terminating at many branching extensions.

[0020] Generally, axons perform the function of transmitting signals from one neuron to another, while dendrites perform the function of receiving signals from other neurons. For example, when different neurons are connected, a signal transmitted through a neuron's axon can be received by another neuron's dendrite. In this case, signals between neurons are transmitted through specialized connections called synapses, and multiple neurons are connected to form a neural network. Based on the synapse, neurons that secrete neurotransmitters are referred to as pre-synaptic neurons, and neurons that receive information transmitted through neurotransmitters can be referred to as post-synaptic neurons.

[0021] Meanwhile, the human brain is capable of learning and remembering vast amounts of information by transmitting and processing various signals through neural networks formed by the interconnection of a large number of neurons. Since the vast number of connections between neurons within the human brain is directly related to the massively parallel nature of biological computing, various attempts have been made to efficiently process vast amounts of information by mimicking artificial neural networks. For example, neuromorphic devices are being researched as computing systems designed to implement artificial neural networks at the neuron level.

[0022] Meanwhile, the operation of a biological neuron (10) can be simulated by a mathematical model (11). A mathematical model (11) corresponding to a biological neuron (10) may include, as an example of a neuromorphic operation, a multiplication operation that multiplies synaptic weights on information from multiple neurons, an addition operation (Σ) on values ​​(ω0x0, ω1x1, ω2x2) on which synaptic weights are multiplied, and an operation that applies a characteristic function (b) and an activation function (f) to the result of the addition operation. A neuromorphic operation result may be provided by the neuromorphic operation. Here, values ​​such as x0, x1, x2, ... correspond to axon values, and values ​​such as ω0, ω1, ω2, ... correspond to synaptic weights.

[0023] FIGS. 2a and 2b are drawings for explaining the operation method of a neuromorphic device according to one embodiment.

[0024] Referring to FIG. 2a, the neuromorphic device may include a crossbar array circuit unit. The crossbar array circuit unit may include a plurality of crossbar array circuits, and each crossbar array circuit may be implemented as a Resistive Crossbar Memory Array (RCA). Specifically, each crossbar array circuit may include an input node (210) corresponding to a presynaptic neuron, a neuron circuit (220) corresponding to a postsynaptic neuron, and a synapse circuit (230) providing a connection between the input node (210) and the neuron circuit (220).

[0025] In one embodiment, the crossbar array circuit of the neuromorphic device includes four input nodes (210), four neuron circuits (220), and 16 synapse circuits (230), but the number of these can be varied. When the number of input nodes (210) is N (where N is a natural number greater than or equal to 2) and the number of neuron circuits (220) is M (where M is a natural number greater than or equal to 2 and may be equal to or different from N), N*M synapse circuits (230) can be arranged in a matrix form.

[0026] Specifically, a wiring (21) connected to an input node (210) and extending in a first direction (e.g., horizontal direction), and a wiring (22) connected to a neuron circuit (220) and extending in a second direction (e.g., vertical direction) that intersects the first direction may be provided. Hereinafter, for convenience of explanation, the wiring (21) extending in the first direction will be referred to as a row line, and the wiring (22) extending in the second direction will be referred to as a column line. A plurality of synapse circuits (230) may be arranged at each intersection of the row line (21) and the column line (22) to connect the corresponding row line (21) and the corresponding column line (22) to each other.

[0027] The input node (210) performs the role of generating a signal, such as a signal corresponding to specific data, and sending it to the row wiring (21), and the neuron circuit (220) performs the role of receiving and processing a synaptic signal that has passed through the synapse circuit (230) via the column wiring (22). The input node (210) corresponds to an axon, and the neuron circuit (220) corresponds to a neuron. However, whether it is a presynaptic neuron or a postsynaptic neuron may be determined by its relative relationship with other neurons. For example, if the input node (210) receives a synaptic signal in relation to another neuron, it may function as a postsynaptic neuron. Similarly, if the neuron circuit (220) sends a signal in relation to another neuron, it may function as a presynaptic neuron.

[0028] The connection between the input node (210) and the neuron circuit (220) can be made through a synapse circuit (230). Here, the synapse circuit (230) is a device whose electrical conductance or weight changes according to an electrical pulse, such as voltage or current, applied to both ends.

[0029] The synapse circuit (230) may include, for example, a variable resistor element. The variable resistor element is a device capable of switching between different resistance states depending on the voltage or current applied to both ends, and may have a single-layer or multi-layer structure including various materials capable of having multiple resistance states, such as metal oxides like transition metal oxides or perovskite-based materials, phase change materials like chalcogenide-based materials, ferroelectric materials, ferromagnetic materials, etc. The operation of the variable resistor element and / or the synapse circuit (230) changing from a high resistance state to a low resistance state may be called a set operation, and the operation of changing from a low resistance state to a high resistance state may be called a reset operation.

[0030] The operation of the neuromorphic device is described below with reference to FIG. 2b. For convenience of explanation, the row wiring (21) is referred to in order from top to bottom as the first row wiring (21A), the second row wiring (21B), the third row wiring (21C), and the fourth row wiring (21D), and the column wiring (22) is referred to in order from left to right as the first column wiring (22A), the second column wiring (22B), the third column wiring (22C), and the fourth column wiring (22D).

[0031] Referring to FIG. 2b, in the initial state, all of the synapse circuits (230) may be in a state of relatively low conductivity, i.e., a high resistance state. If some of the synapse circuits (230) are in a low resistance state, an initialization operation to make them high resistance state may be additionally required. Each of the synapse circuits (230) may have a predetermined threshold value required for resistance and / or conductivity change. More specifically, if a voltage or current smaller than a predetermined threshold value is applied across each synapse circuit (230), the conductivity of the synapse circuit (230) does not change, and if a voltage or current larger than a predetermined threshold value is applied to the synapse circuit (230), the conductivity of the synapse circuit (230) may change.

[0032] In this state, in order to perform the operation of outputting specific data as a result of a specific column wiring (22), an input signal corresponding to specific data may be introduced into the row wiring (21) in response to the output of the input node (210). At this time, the input signal may be represented by the application of an electrical pulse to each row wiring (21). For example, when an input signal corresponding to the data '0011' is introduced into the row wiring (21), an electrical pulse may not be applied to the row wiring (21) corresponding to '0', e.g., the first and second row wirings (21A, 21B), and an electrical pulse may be applied only to the row wiring (21) corresponding to '1', e.g., the third and fourth row wirings (21C, 21D). At this time, the column wiring (22) may be driven with an appropriate voltage or current for output.

[0033] For example, if a column wiring (22) to output specific data is already determined, this column wiring (22) is driven so that a synapse circuit (230) located at the intersection with a row wiring (21) corresponding to '1' receives a voltage greater than or equal to the voltage required for a set operation (hereinafter, set voltage), and the remaining column wiring (22) can be driven so that the remaining synapse circuit (230) receives a voltage smaller than the set voltage. For example, if the magnitude of the set voltage is Vset and the column wiring (22) to output data of '0011' is set to the third column wiring (22C), the magnitude of the electrical pulse applied to the third and fourth row wirings (21C, 21D) may be greater than Vset and the voltage applied to the third column wiring (22C) may be 0V, so that the first and second synapse circuits (230A, 230B) located at the intersection of the third column wiring (22C) and the third and fourth row wirings (21C, 21D) receive a voltage greater than Vset. Accordingly, the first and second synapse circuits (230A, 230B) may be in a low-resistance state. The conductivity of the first and second synapse circuits (230A, 230B) in the low-resistance state may gradually increase as the number of electrical pulses increases. The magnitude and width of the applied electrical pulse may be substantially constant. In order for the remaining synapse circuits (230), excluding the first and second synapse circuits (230A, 230B), to receive a voltage smaller than Vset, the voltage applied to the remaining column wiring, i.e., the first, second, and fourth column wirings (22A, 22B, 22D), may have a value between 0V and Vset, for example, 1 / 2Vset. Accordingly, the resistance state of the remaining synapse circuits (230), excluding the first and second synapse circuits (230A, 230B), may not change.

[0034] As another example, the column wiring (22) to output specific data may not be determined. In this case, by applying an electrical pulse corresponding to the specific data to the row wiring (21) and measuring the current flowing through each column wiring (22), the column wiring (22) that reaches a predetermined threshold current first, such as the third column wiring (22C), may become the column wiring (22) that outputs the specific data.

[0035] By the method described above, different data can be output to different column wiring (22).

[0036] FIGS. 3a and 3b are drawings for comparing vector-matrix multiplication and operations performed on a crossbar array according to one embodiment.

[0037] First, referring to FIG. 3a, the convolution operation between the input feature map and the weight values ​​can be performed using vector-matrix multiplication. For example, the pixel data of the input feature map can be represented by matrix X (310), and the weight values ​​can be represented by matrix W (311). The pixel data of the output feature map can be represented by matrix Y (312), which is the result of the multiplication operation between matrix X (310) and matrix W (311).

[0038] Referring to FIG. 3b, a vector multiplication operation can be performed using a non-volatile memory element of a crossbar array. As described in comparison with FIG. 3a, pixel data of an input feature map can be received as an input value of the non-volatile memory element, and the input value can be a voltage (320). Additionally, weight values ​​can be stored in the synapse, i.e., the memory cell, of the non-volatile memory element, and the weight values ​​stored in the memory cell can be a conductance (321). Accordingly, the output value of the non-volatile memory element can be expressed as a current (322), which is the result of a multiplication operation between the voltage (320) and the conductance (321).

[0039] FIG. 4 is a diagram illustrating an example in which a convolution operation is performed in a neuromorphic device according to one embodiment.

[0040] The neuromorphic device may receive pixels of an input feature map (410), and the crossbar array circuit (400) of the neuromorphic device may be implemented as Resistive Crossbar Memory Arrays (RCA).

[0041] The neuromorphic device can receive an input feature map in the form of a digital signal and can convert the input feature map into a voltage in the form of an analog signal using a Digital Analog Converter (DAC, 420). In one embodiment, the neuromorphic device can convert the pixel values ​​of the input feature map into a voltage using the DAC (420) and then provide the voltage as an input value (401) of a crossbar array circuit (400).

[0042] Additionally, learned weight values ​​may be stored in the crossbar array circuit (400) of the neuromorphic device. The weight values ​​may be stored in the memory cells of the crossbar array circuit, and the weight values ​​stored in the memory cells may be conductances (402). At this time, the neuromorphic device may produce an output value by performing a vector multiplication operation between the input value (401) and the conductance (402), and the output value may be expressed as a current (403). That is, the neuromorphic device may output a result value identical to the result of a convolution operation between the input feature map and the weight values ​​using the crossbar array circuit (400).

[0043] Since the current (403) output from the crossbar array circuit (400) is an analog signal, the neuromorphic device may use an Analog-to-Digital Converter (ADC, 430) to use the current (403) as an input feature map of another crossbar array circuit. The neuromorphic device may convert the current (403), which is an analog signal, into a digital signal using the ADC (430). In one embodiment, the neuromorphic device may use the ADC (430) to convert the current (403) into a digital signal with the same number of bits as the pixels of the input feature map (410). For example, if the pixels of the input feature map (410) are 4-bit data, the neuromorphic device may use the ADC (430) to convert the current (403) into 4-bit data.

[0044] The neuromorphic device can apply an activation function to the digital signal converted by the ADC (430) using an activation unit (440). As activation functions, the Sigmoid function, the Tanh function, and the ReLU (Rectified Linear Unit) function may be used, but the activation functions that can be applied to the digital signal are not limited to these.

[0045] A digital signal to which an activation function has been applied can be used as an input feature map of another crossbar array circuit (450). When a digital signal to which an activation function has been applied is used as an input feature map of another crossbar array circuit (450), the process described above can be applied in the same way to the other crossbar array circuit (450).

[0046] FIG. 5 is a diagram illustrating operations performed in a neural network according to one embodiment.

[0047] Referring to FIG. 5, the neural network (500) has a structure including an input layer, hidden layers and an output layer, and can perform operations based on received input data (e.g., I1 and I2) and generate output data (e.g., O1 and O2) based on the results of the operation.

[0048] For example, as illustrated in FIG. 5, the neural network (500) may include an input layer (Layer 1), two hidden layers (Layer 2 and Layer 3), and an output layer (Layer 4). Since the neural network (500) includes more layers capable of processing valid information, the neural network (500) can process more complex data sets than a neural network having a single layer. Meanwhile, although the neural network (500) is illustrated as including four layers, this is merely an example, and the neural network (500) may include fewer or more layers or fewer or more channels. That is, the neural network (500) may include layers of various structures different from those illustrated in FIG. 5.

[0049] Each layer included in the neural network (500) may include multiple channels. A channel may correspond to multiple artificial nodes known as a neuron, a processing element (PE), a unit, or similar terms. For example, as shown in FIG. 5, Layer 1 may include two channels (nodes), and Layer 2 and Layer 3 may each include three channels. However, this is merely an example, and each layer included in the neural network (500) may include various numbers of channels (nodes).

[0050] The channels included in each layer of the neural network (500) can be connected to each other to process data. For example, one channel can receive data from other channels and perform calculations, and can output the results of the calculations to other channels.

[0051] Each input and output of each channel can be referred to as an input feature map and an output feature map, respectively. An input feature map may include multiple input activations, and an output feature map may include multiple output activations. That is, a feature map or activation may be an output of a channel and, at the same time, a parameter corresponding to the input of channels included in the next layer.

[0052] Meanwhile, each channel can determine its own activation based on activations and weight values ​​received from the channels included in the previous layer. The weight value is a parameter used to calculate the output activation at each channel and may be a value assigned to the connection relationship between the channels.

[0053] Each of the channels can be processed by a computational unit or processing element that receives an input and outputs an output activation, and the input and output of each channel can be mapped. For example, is an activation function, and is the weight value from the k-th channel included in the (i-1)-th layer to the j-th channel included in the i-th layer, and is the bias of the j-th channel included in the i-th layer, and Let be the activation of the j-th channel of the i-th layer. It can be calculated using the following mathematical formula 1.

[0054]

[0055] As shown in FIG. 5, the activation of the first channel (CH 1) of the second layer (Layer 2) is It can be expressed as. Also, is according to mathematical formula 1 It may have a value. However, the mathematical formula 1 described above is merely an example to explain the activation and weight values ​​used to process data in the neural network (500), and is not limited thereto. The activation may be a value obtained by applying batch normalization and an activation function to the sum of activations received from the previous layer.

[0056] FIGS. 6a to 6c are drawings for illustrating an example of converting initial weight values ​​into binary weight values ​​according to one embodiment.

[0057] Referring to FIG. 6a, the input layer (601), the output layer (602), and the initial weight values ​​(W 11 , W 12 , ..., W 32 , W 33 ) is illustrated. Three input activations (I1, I2, I3) may correspond to each of the three neurons in the input layer (601), and three output activations (O1, O2, O3) may correspond to each of the three neurons in the output layer (602). Additionally, the nth input activation (I n ) and the m-th output activation(O m ) is the initial weight value W nm This can be applied.

[0058] The initial weight values ​​(610) of FIG. 6b are the initial weight values ​​(W) according to FIG. 6a. 11 , W 12 , ..., W 32 , W 33 It is a representation of ) as a matrix.

[0059] Initial weight values ​​(610) can be determined during the learning process of the neural network. In one embodiment, the initial weight values ​​(610) can be represented as 32-bit floating-point numbers.

[0060] Initial weight values ​​(610) can be converted into binary weight values ​​(620). Binary weight values ​​(620) can have a size of 1 bit. In the present invention, by using binary weight values ​​(620) instead of initial weight values ​​(610) during the inference process of a neural network, the model size and amount of computation can be reduced. For example, if 32-bit initial weight values ​​(610) are converted into 1-bit binary weight values ​​(620), the model size can be compressed by 1 / 32.

[0061] In one embodiment, based on the maximum and minimum values ​​of the initial weight values ​​(610), the initial weight values ​​(610) can be converted into binary weight values ​​(620). In another embodiment, based on the maximum and minimum values ​​of the initial weight values ​​that can be input to the neural network, the initial weight values ​​(610) can be converted into binary weight values ​​(620).

[0062] For example, the maximum value of initial weight values ​​that can be input to a neural network may be 1.00 and the minimum value may be -1.00. If the initial weight value is 0.00 or greater, it may be converted to a binary weight value of 1, and if the initial weight value is less than 0.00, it may be converted to a binary weight value of -1.

[0063] Additionally, the binary weight values ​​(620) may be multiplied by the average value (630) of the absolute values ​​of the initial weight values ​​(610). By multiplying the binary weight values ​​(620) by the average value (630) of the absolute values ​​of the initial weight values ​​(610), a result similar to that obtained when using the initial weight values ​​(610) can be obtained even when using the binary weight values ​​(620).

[0064] For example, assuming there are 1024 neurons in the previous layer and 512 neurons in the current layer, the 512 neurons in the current layer each have 1024 initial weight values ​​(610), and after the average of the absolute values ​​of the 1024 32-bit floating-point initial weight values ​​(610) for each neuron is calculated, the result of the calculation can be multiplied by the binary weight values ​​(620).

[0065] Specifically, the average value of the absolute values ​​of the initial weight values ​​(610) used to calculate the predetermined output activation (O1, O2, O3) can be multiplied by the initial weight values ​​(610) used to calculate the predetermined output activation (O1, O2, O3).

[0066] For example, referring to FIG. 6a, in the process of calculating the first output activation (O1), an initial weight value W 11 , W 21 and W 31 This can be used. Initial weight value W 11 , W 21 and W 31 Each is a binary weight value W 11 ', W 21 ' and W 31 Converted to ', and binary weight value W 11 ', W 21 ' and W 31 In 'in the initial weight value W 11 , W 21 and W 31 The average value of the absolute values This can be multiplied.

[0067] In the same way, the binary weight value W 12 ', W 22 ' and W 32 In 'in the initial weight value W 12 , W 22 and W 32 The average value of the absolute values This can be multiplied. Also, the binary weight value W 13 ', W 23 ' and W 33 In 'in the initial weight value W 13 , W 23 and W 33 The average value of the absolute values This can be multiplied.

[0068] Referring to FIG. 6c, the initial weight values ​​(610), binary weight values ​​(620), and the average value (630) of the absolute values ​​of the initial weight values ​​(610) are shown as specific numbers. In FIG. 6c, the initial weight values ​​(610) are expressed in decimal form for convenience of explanation, but it is assumed that the initial weight values ​​(610) are 32-bit floating-point numbers.

[0069] Figure 6c illustrates that if the initial weight value is 0.00 or greater, it is converted to a binary weight value of 1, and if the initial weight value is less than 0.00, it is converted to a binary weight value of -1.

[0070] Also, the initial weight value W 11 , W 21 and W 31 The average value of the absolute values ​​is '0.28', and the initial weight value W 12 , W 22 and W 32 The average value of the absolute values ​​is '0.37', and the initial weight value W 13 , W 23 and W 33 It is shown that the average value of the absolute values ​​is '0.29'.

[0071] FIGS. 7a and 7b are drawings for illustrating an example of converting an input feature map into time domain binary vectors according to one embodiment.

[0072] The input feature map can be converted into multiple time-domain binary vectors. The input feature map may include multiple input activations, and each of the multiple input activations can be converted into a time-domain binary vector.

[0073] The input feature map can be converted into multiple time-domain binary vectors based on quantization levels. In one embodiment, the range between the maximum and minimum values ​​of input activations that can be input to a neural network can be divided into N (N is a natural number) quantization levels. For example, a sigmoid function or a tanh function may be used to distinguish the quantization levels, but is not limited thereto.

[0074] For example, referring to Fig. 7a, when the quantization level is set to 9 and the maximum and minimum values ​​of input activations that can be input to the neural network are 1.0 and -1.0, respectively, the quantization level can be divided into '1.0, 0.75, 0.5, 0.25, 0, -0.25, -0.5, -0.75, -1.0'.

[0075] Meanwhile, although the spacing between quantization levels is shown as being equal in Fig. 7a, the spacing between quantization levels may be set non-linearly.

[0076] When the quantization level is set to N, the time domain binary vector can have N-1 elements. For example, referring to Fig. 7a, when the quantization level is set to 9, the time domain binary vector can have 8 elements (t1, t2, ..., t7, t8).

[0077] Based on which of the N quantization levels the input activation belongs to, the input activation can be converted into a time domain binary vector. For example, if a given input activation has a value of 0.75 or higher, the given input activation can be converted into a time domain binary vector '+1, +1, +1, +1, +1, +1, +1, +1'. Also, for example, if a given input activation has a value of less than -0.25 and greater than or equal to -0.5, the given input activation can be converted into a time domain binary vector '+1, +1, +1, -1, -1, -1, -1'.

[0078] Referring to FIG. 7b, an example is illustrated in which each of the multiple input activations included in the input feature map (710) is converted into a time binary vector. Since the first activation has a value less than 0 and greater than or equal to -0.25, it can be converted into a time domain binary vector '-1, -1, -1, -1, +1, +1, +1, +1', and since the second activation has a value less than 0.5 and greater than or equal to 0.25, it can be converted into a time domain binary vector '-1, -1, +1, +1, +1, +1, +1, +1'. Additionally, since the third activation has a value less than -0.75 and greater than or equal to -1.0, it can be converted into a time domain binary vector '-1, -1, -1, -1, -1, -1, -1, +1', and since the fourth activation has a value greater than or equal to 0.75, it can be converted into a time domain binary vector '+1, +1, +1, +1, +1, +1, +1, +1'.

[0079] On the other hand, if each input activation of a layer of a neural network is converted into a binary value using the conventional method, the information contained in the input activations is lost, and information transfer between layers may not occur normally.

[0080] On the other hand, as in the present disclosure, if each of the input activations of each layer of a neural network is converted into a time-domain binary vector, the original input activations can be approximated based on multiple binary values.

[0081] FIG. 8 is a diagram illustrating the application of binary weight values ​​and time domain binary vectors to a batch normalization process according to one embodiment.

[0082] Generally, in neural network algorithm models, the input activation (the initial input value or the output value of the previous layer) and the initial weight value (32-bit floating-point) are multiplied and the results summed (MAC: multiply and accumulate), after which a separate bias value is additionally added for each neuron. Then, batch normalization is performed on the resulting values ​​for each neuron. Finally, the result is input into an activation function, and the output of the activation function is passed as the input to the next layer.

[0083] This can be expressed as in Equation 2. In Equation 2, I n is input activation, W nm is the initial weight value, B m is the bias value, α m β is the initial scale value for batch normalization. m is the batch normalization bias value, f is the activation function, O m means output activation.

[0084]

[0085]

[0086] Referring to FIG. 8, input activation I n (810) is a time domain binary vector I b nIt can be converted to (t)(820). The temporal binary vector generator input activation I n (810) time domain binary vector I b n It can be converted to (t)(820).

[0087] As described above in FIGS. 7a and 7b, input activation I according to a preset quantization level n (810) is a time domain binary vector I b n It can be converted to (t)(820). Meanwhile, the time domain binary vector I b n (t)(820) The number of elements included in each can be determined by the number of quantization levels. For example, if the number of quantization levels is N, the time domain binary vector I b n (t)(820) The number of elements included in each can be N-1.

[0088] Meanwhile, input activation I n (810) This time domain binary vector I b n When converted to (t)(820), time domain binary vector I b n The result of the operation using (t)(820) is a time-domain binary vector I b n It can be amplified by the number of elements T (850) included in (t) (820). Accordingly, the time domain binary vector I b n When using (t)(820), the result of the operation must be divided by the number of elements T(850) to obtain the same result as the original MAC operation result. A detailed explanation of this will be provided using mathematical formulas 5 and 6.

[0089] As described above in FIGS. 6a to 6c, the initial weight value W nm is the binary weight value W b nm It can be converted to (830). For example, the initial weight value W through the sign function nm is the binary weight value W b nm It can be converted to (830).

[0090] Time domain binary vector I b n (t)(820) and binary weight value W b nm Convolution operations between (830) can be performed. In one embodiment, time domain binary vector I b n (t)(820) and binary weight value W b nm XNOR operations and addition operations between (830) can be performed.

[0091] Time domain binary vector I b n (t)(820) and binary weight value W b nm After performing XNOR operations between (830) and summing all the results, the original multi-bit input activation I n (810) and binary initial weight value W nm It exhibits the same pattern of increase and decrease as the result of performing a convolution operation between them.

[0092] Time domain binary vector I b n (t)(820) and binary weight value W b nm (830) The operation between them can be expressed as in mathematical formula 3 below.

[0093]

[0094] An intermediate activation Xm (840) can be produced as a result of the convolution operation. Intermediate activation X m (840) can be expressed as shown in mathematical equation 4 below.

[0095]

[0096] Intermediate Activation X m (840) contains the average value S of the absolute values ​​of the initial weight values. m (860) can be multiplied

[0097] Also, intermediate activation X m (840) contains the time domain binary vector I b n (t)(820) can be divided by the number of elements T(850) contained in each. Time domain binary vector I b n (t)(820) The number of elements included in each can be determined by the number of quantization levels. Time domain binary vector I b n Since the result of the operation using (t)(820) is amplified by the number of elements T(850), the intermediate activation X m By dividing (840) by the number of elements T (850), the same result as the original MAC operation result can be obtained.

[0098] Intermediate Activation X m (840) The average value of the absolute values ​​of the initial weight values ​​S m (860) is multiplied, and the time domain binary vector I b n (t)(820) is divided by the number of elements T(860) included in each, thereby output activation O m (870) can be produced. Output activation O m (870) can be expressed as shown in mathematical formula 5 below.

[0099]

[0100] In one embodiment, when batch normalization is performed, the average value S of the absolute values ​​of the initial weight values ​​for the initial scale value of the batch normalization m (860) is multiplied, and the time domain binary vector I b n The number of elements T (850) included in (t) (820) is divided, thereby changing the modified scale value α" m (modified scale value) can be calculated.

[0101] In the neural network algorithm model according to Equation 2, the binary weight value W b nm , time domain binary vector I b nt , and modified scale value α" m When applying, mathematical formula 2 can be expressed as mathematical formula 6 below.

[0102]

[0103]

[0104]

[0105]

[0106] In the present disclosure, an initial weight value W is represented as a multi-bit floating-point number. nm Binary weight value W with +1 or -1 b nm By converting to (830), the model size and amount of computation can be reduced.

[0107] In the present disclosure, the binary weight value W b nm (830) The average value of the absolute values ​​of the initial weight values ​​S m By multiplying by (860), the binary weight value W b nm Even when using (830), the initial weight value W nmYou can obtain a result value similar to that obtained when using .

[0108] Meanwhile, the average value S of the absolute values ​​of the initial weight values m (860) can be included in the batch normalization operation as in Equation 6 above (M m X α m ), additional model parameters are generated, so there is no loss in reducing the model size, nor is there any loss in reducing the amount of computation. In other words, compared to Equation 2, it can be confirmed that the operation can be performed in Equation 6 without adding separate parameters or separate procedures.

[0109] In this disclosure, multi-bit input activation I n After quantizing (810) into a lower bit size of 2 to 3 bits, it becomes a time-domain binary vector I having multiple elements b n It can be converted to (t)(820). In addition, in the present disclosure, the binary weight value W b nm (830) and time domain binary vector I b n By performing time-axis XNOR operations between (t)(820), it is possible to obtain learning performance and final classification / recognition accuracy similar to that of a 32-bit floating-point neural network based on binary MAC operations.

[0110] Meanwhile, the number of elements T (860) can be included in the batch normalization operation as in Equation 6 above (α m X 1 / T), additional model parameters are generated, so there is no loss in reducing the model size or reducing the amount of computation. In other words, compared to Equation 2, it can be confirmed that the calculation can be performed in Equation 6 without adding separate parameters or separate procedures.

[0111] For example, 32-bit input activation I n (810) is a time domain binary vector I with T elements b n When converting to (t)(820), the model size can be compressed by T / 32.

[0112] FIG. 9 is a block diagram of a neural network device using a von Neumann structure according to one embodiment.

[0113] Referring to FIG. 9, the neural network device (900) may include an external input receiving unit (910), a memory (920), a time domain binary vector generating unit (930), a convolution operation unit (940), and a neural operation unit (950).

[0114] The neural network device (900) illustrated in FIG. 9 shows only the components related to the embodiments thereof. Therefore, it is obvious to a person skilled in the art that the neural network device (900) may include other general-purpose components in addition to the components illustrated in FIG. 9.

[0115] The external input receiver (910) can receive information related to the neural network model, input image (or audio) data, etc. from the outside. Various information and data received from the external input receiver (910) can be stored in memory (920).

[0116] In one embodiment, the memory (920) may be divided into a first memory that stores an input feature map and a second memory that stores binary weight values, real other parameters, model structure definition variables, etc. Meanwhile, the binary weight values ​​stored in the memory (920) may be values ​​converted from the initial weight values ​​(e.g., 32-bit floating-point) after the neural network has completed training.

[0117] The time domain binary vector generation unit (930) can receive an input feature map from memory (920). The time domain binary vector generation unit (930) can convert the input feature map into time domain binary vectors. The input feature map may include a plurality of input activations, and the time domain binary vector generation unit (930) can convert each of the plurality of input activations into a time domain binary vector.

[0118] Specifically, the time domain binary vector generation unit (930) can convert an input feature map into a plurality of time domain binary vectors based on quantization levels. In one embodiment, when the range between the maximum and minimum values ​​of an input activation that can be input to a neural network is divided into N (N is a natural number) quantization levels, the time domain binary vector generation unit (930) can convert the input activation into a time domain binary vector having N-1 elements.

[0119] The convolution operation unit (940) can receive binary weight values ​​from memory (920). Additionally, the convolution operation unit (940) can receive multiple time domain binary vectors from the time domain binary vector generation unit (930).

[0120] The convolution operation unit (940) may include an adder, and the convolution operation unit (940) may perform a convolution operation between binary weight values ​​and a plurality of time domain binary vectors.

[0121] The neural operation unit (950) can receive the result of a convolution operation between binary weight values ​​and a plurality of time domain binary vectors from the convolution operation unit (940). Additionally, the neural operation unit (950) can receive a batch normalization modification scale value, a batch normalization bias value, and an activation function from the memory (920).

[0122] In the neural operation unit (950), batch normalization and pooling are performed and an activation function may be applied, but the operations that can be performed and applied in the neural operation unit (950) are not limited thereto.

[0123] Meanwhile, the modified scale value of batch normalization can be calculated by multiplying the initial scale value by the average of the absolute values ​​of the initial weight values ​​and dividing by the number of elements T included in the time domain binary vector.

[0124] Batch normalization is performed in the neural operation unit (950), and an activation function is applied so that an output feature map can be output. The output feature map may include multiple output activations.

[0125] FIG. 10 is a block diagram of a neuromorphic device using an in-memory structure according to one embodiment.

[0126] Referring to FIG. 10, the neuromorphic device (1000) may include an external input receiving unit (1010), a memory (1020), a time domain binary vector generating unit (1030), an on-chip memory (1040), and a neural operation unit (1050).

[0127] The neuromorphic device (1000) illustrated in FIG. 10 only shows components related to the embodiments thereof. Therefore, it is obvious to a person skilled in the art that the neuromorphic device (1000) may include other general-purpose components in addition to the components illustrated in FIG. 10.

[0128] The external input receiver (1010) can receive information related to the neural network model, input image (or audio) data, etc. from the outside. Various information and data received from the external input receiver (1010) can be stored in memory (1020).

[0129] The memory (1020) can store input feature maps, real number and other parameters, model structure definition variables, etc. Unlike the neural network device (900) of FIG. 9, binary weight values ​​can be stored in on-chip memory (1040) rather than in memory (1020), and details will be described later.

[0130] The time domain binary vector generation unit (1030) can receive an input feature map from memory (1020). The time domain binary vector generation unit (1030) can convert the input feature map into time domain binary vectors. The input feature map may include a plurality of input activations, and the time domain binary vector generation unit (1030) can convert each of the plurality of input activations into a time domain binary vector.

[0131] Specifically, the time domain binary vector generation unit (1030) can convert an input feature map into a plurality of time domain binary vectors based on quantization levels. In one embodiment, when the range between the maximum and minimum values ​​of an input activation that can be input to a neural network is divided into N (N is a natural number) quantization levels, the time domain binary vector generation unit (1030) can convert the input activation into a time domain binary vector having N-1 elements.

[0132] The on-chip memory (1040) may include an input section (1041), a crossbar array circuit (1042), and an output section (1043).

[0133] The crossbar array circuit (1042) may include a plurality of synapse circuits (e.g., variable resistors). Binary weight values ​​may be stored in the plurality of synapse circuits. The binary weight values ​​stored in the plurality of synapse circuits may be values ​​converted from the initial weight values ​​(e.g., 32-bit floating-point) after the neural network has completed training.

[0134] The input unit (1041) can receive a plurality of time domain binary vectors from the time domain binary vector generation unit (1030).

[0135] When a plurality of time domain binary vectors are received at the input unit (1041), the crossbar array circuit (1042) can perform a convolution operation between the binary weight values ​​and the plurality of time domain binary vectors.

[0136] The output unit (1043) can transmit the convolution operation result to the neural operation unit (1050).

[0137] The neural operation unit (1050) can receive the result of a convolution operation between binary weight values ​​and a plurality of time domain binary vectors from the output unit (1043). Additionally, the neural operation unit (1050) can receive a batch normalization modification scale value, a batch normalization bias value, and an activation function from the memory (1020).

[0138] In the neural operation unit (1050), batch normalization and pooling are performed and an activation function may be applied, but the operations that can be performed and applied in the neural operation unit (1050) are not limited thereto.

[0139] Meanwhile, the modified scale value of batch normalization can be calculated by multiplying the initial scale value by the average of the absolute values ​​of the initial weight values ​​and dividing by the number of elements T included in the time domain binary vector.

[0140] Batch normalization is performed in the neural operation unit (1050), and an activation function is applied so that an output feature map can be output. The output feature map may include multiple output activations.

[0141] FIG. 11 is a flowchart illustrating a method for implementing a neural network in a neural network device according to one embodiment.

[0142] Referring to FIG. 11, in step 1110, the neural network device can obtain binary weight values ​​and an input feature map from memory.

[0143] In step 1120, the neural network device can convert the input feature map into time domain binary vectors.

[0144] In one embodiment, the neural network device can convert the input feature map into time-domain binary vectors based on the quantization level of the input feature map.

[0145] Specifically, the neural network device divides the range between the maximum and minimum values ​​that can be input to the neural network into N (N is a natural number) quantization levels, and can convert each of the activations of the input feature map into a time domain binary vector based on which of the N quantization levels each activation belongs to.

[0146] Meanwhile, a neural network device can distinguish the range between the maximum and minimum values ​​that can be input to the neural network into linear quantization levels or non-linear quantization levels.

[0147] In step 1130, the neural network device can output an output feature map by performing a convolution operation between binary weight values ​​and time domain binary vectors.

[0148] A neural network device can output an output feature map by performing batch normalization on the result of a convolution operation.

[0149] In one embodiment, the neural network device can calculate a modified scale value for the initial scale value of batch normalization by multiplying the average of the absolute values ​​of the initial weight values ​​and dividing by the number of elements included in each of the time domain binary vectors. The neural network device can perform batch normalization based on the modified scale value.

[0150] The neural network device performs a multiplication operation in which it multiplies each of the bias values ​​applied to the neural network by an initial scale value, and can reflect the result of the multiplication operation in the output feature map.

[0151] In addition, the neural network device can output an output feature map by performing batch normalization on the result of the convolution operation and applying an activation function to the result of the batch normalization.

[0152] FIG. 12 is a flowchart illustrating a method for implementing a neural network in a neuromorphic device according to one embodiment.

[0153] Referring to FIG. 12, in step 1210, the neuromorphic device can store binary weight values ​​in synapse circuits included in a crossbar array circuit.

[0154] In step 1220, the neuromorphic device can obtain an input feature map from memory.

[0155] In step 1230, the neuromorphic device can convert the input feature map into time-domain binary vectors.

[0156] In one embodiment, the neuromorphic device can convert an input feature map into time-domain binary vectors based on a quantization level for the input feature map.

[0157] Specifically, the neuromorphic device divides the range between the maximum and minimum values ​​that can be input to a neural network into N (N is a natural number) quantization levels, and can convert each activation of the input feature map into a time-domain binary vector based on which of the N quantization levels each activation belongs to.

[0158] Meanwhile, neuromorphic devices can distinguish the range between the maximum and minimum values ​​that can be input to a neural network into linear or non-linear quantization levels.

[0159] In step 1240, the neuromorphic device can provide time domain binary vectors as input values ​​to the crossbar array circuit.

[0160] In step 1250, the neuromorphic device can output an output feature map by performing a convolution operation between binary weight values ​​and time domain binary vectors.

[0161] A neuromorphic device can output an output feature map by performing batch normalization on the result of a convolution operation.

[0162] In one embodiment, the neuromorphic device can calculate a modified scale value for the initial scale value of batch normalization by multiplying it by the average of the absolute values ​​of the initial weight values ​​and dividing by the number of elements included in each of the time domain binary vectors. The neuromorphic device can perform batch normalization based on the modified scale value.

[0163] The neuromorphic device performs a multiplication operation in which it multiplies each of the bias values ​​applied to the neural network by an initial scale value, and can reflect the result of the multiplication operation in the output feature map.

[0164] In addition, the neuromorphic device can output an output feature map by performing batch normalization on the result of the convolution operation and applying an activation function to the result of the batch normalization.

[0165] FIG. 13 is a block diagram illustrating the hardware configuration of a neural network device according to one embodiment.

[0166] The neural network device (1300) can be implemented in various types of devices such as PCs (personal computers), server devices, mobile devices, and embedded devices. Specific examples include smartphones, tablet devices, AR (Augmented Reality) devices, IoT (Internet of Things) devices, autonomous vehicles, robotics, and medical devices that perform speech recognition, image recognition, and image classification using neural networks, but are not limited thereto. Furthermore, the neural network device (1300) can be a dedicated hardware accelerator (HW accelerator) installed in such devices, and the neural network device (1300) can be a hardware accelerator such as an NPU (neural processing unit), TPU (Tensor Processing Unit), or Neural Engine, which is a dedicated module for operating neural networks, but is not limited thereto.

[0167] Referring to FIG. 13, the neural network device (1300) includes a processor (1310) and a memory (1320). Only the components related to the embodiments are shown in the neural network device (1300) illustrated in FIG. 9. Therefore, it is obvious to a person skilled in the art that the neural network device (1300) may include other general-purpose components in addition to those shown in FIG. 13.

[0168] The processor (1310) serves to control overall functions for executing the neural network device (1300). For example, the processor (1310) controls the neural network device (1300) overall by executing programs stored in memory (1320) within the neural network device (1300). The processor (1310) may be implemented as a CPU (central processing unit), GPU (graphics processing unit), AP (application processor), etc., provided within the neural network device (1300), but is not limited thereto.

[0169] Memory (1320) is hardware that stores various data processed within the neural network device (1300). For example, memory (1320) can store data processed and data to be processed in the neural network device (1300). Additionally, memory (1320) can store applications, drivers, etc. to be driven by the neural network device (1300). Memory (1320) may include RAM (random access memory) such as DRAM (dynamic random access memory) and SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), CD-ROM, Blu-ray or other optical disc storage, HDD (hard disk drive), SSD (solid state drive), or flash memory.

[0170] The processor (1310) reads / writes neural network data, such as image data, feature map data, weight value data, etc., from memory (1320) and executes the neural network using the read / write data. When the neural network is executed, the processor (1310) repeatedly performs convolution operations between the input feature map and the weight value to generate data regarding the output feature map. At this time, the amount of computation for the convolution operation may be determined by various factors such as the number of channels of the input feature map, the number of channels of the weight value, the size of the input feature map, the size of the weight value, and the precision of the value.

[0171] The actual neural network running in the neural network device (1300) can be implemented with a more complex architecture. Accordingly, the processor (1310) performs operations with a very large number of operations (operation count), ranging from hundreds of millions to tens of billions, and the frequency with which the processor (1310) accesses memory (1320) for operations is inevitably increased dramatically. Due to this burden of operations, the processing of the neural network may not be smooth in mobile devices such as smartphones, tablets, and wearable devices, or embedded devices, which have relatively low processing performance.

[0172] The processor (1310) can perform convolution operations, batch normalization operations, pooling operations, activation function operations, etc. In one embodiment, the processor (1310) can perform matrix multiplication operations, transformation operations, and transposition operations to obtain multihead self-attention. In the process of obtaining multihead self-attention, the transformation operations and transposition operations may be performed after or before the matrix multiplication operations.

[0173] The processor (1310) obtains binary weight values ​​and an input feature map from memory (920) and can convert the input feature map into time domain binary vectors. Additionally, the processor (1310) can output an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors.

[0174] FIG. 14 is a block diagram illustrating the hardware configuration of a neuromorphic device according to one embodiment.

[0175] Referring to FIG. 14, the neuromorphic device (1400) may include a processor (1410) and an on-chip memory (1420). Only the components related to the embodiments are shown in the neuromorphic device (1400) illustrated in FIG. 14. Therefore, it is obvious to a person skilled in the art that the neuromorphic device (1400) may include other general-purpose components in addition to those shown in FIG. 14.

[0176] The neuromorphic device (1400) can be installed in digital systems requiring low-power neural network operation, such as smartphones, drones, tablet devices, AR (Augmented Reality) devices, IoT (Internet of Things) devices, autonomous vehicles, robotics, and medical devices, but is not limited thereto.

[0177] A neuromorphic device (1400) may include a plurality of on-chip memories (1420), and each of the on-chip memories (1420) may be composed of a plurality of crossbar array circuits. A crossbar array circuit may include a plurality of presynaptic neurons, a plurality of postsynaptic neurons, and synapse circuits, i.e., memory cells, that provide respective connections between the plurality of presynaptic neurons and the plurality of postsynaptic neurons. In one embodiment, the crossbar array circuit may be implemented as Resistive Crossbar Memory Arrays (RCAs).

[0178] The external memory (1430) is hardware that stores various data processed by the neuromorphic device (1400), and the external memory (1430) can store data processed by the neuromorphic device (1400) and data to be processed. Additionally, the external memory (1430) can store applications, drivers, etc. to be driven by the neuromorphic device (1400). The external memory (1430) may include RAM (random access memory) such as DRAM (dynamic random access memory) and SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), CD-ROM, Blu-ray or other optical disc storage, HDD (hard disk drive), SSD (solid state drive), or flash memory.

[0179] The processor (1410) serves to control the overall functions for operating the neuromorphic device (1400). For example, the processor (1410) controls the neuromorphic device (1400) overall by executing programs stored in the on-chip memory (1420) within the neuromorphic device (1400). The processor (1410) may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc., provided within the neuromorphic device (1400), but is not limited thereto. The processor (1410) reads / writes various data from the external memory (1430) and executes the neuromorphic device (1400) using the read / written data.

[0180] The processor (1410) can generate multiple binary feature maps by binarizing the pixel values ​​of the input feature map based on multiple threshold values. The processor (1410) can provide the pixel values ​​of the multiple binary feature maps as input values ​​to a crossbar array circuit unit. The processor (1410) can convert the pixel values ​​into an analog signal (voltage) using a Digital Analog Converter (DAC).

[0181] The processor (1410) can store weight values ​​to be applied to the crossbar array circuit unit in the synapse circuits included in the crossbar array circuit unit. The weight values ​​stored in the synapse circuits may be conductances. Additionally, the processor (1410) can produce output values ​​of the crossbar array circuit unit by performing a multiplication operation between the input values ​​and the kernel values ​​stored in the synapse circuits.

[0182] The processor (1410) can generate pixel values ​​of an output feature map by merging the output values ​​calculated from the crossbar array circuit unit. Meanwhile, since the output values ​​calculated from the crossbar array circuit unit (or the results obtained by multiplying the calculated output values ​​by weight values) are in the form of analog signals (current), the processor (1410) can convert the output values ​​into digital signals using an Analog-to-Digital Converter (ADC). Additionally, the processor (1410) can apply an activation function to the output values ​​converted into digital signals by the ADC.

[0183] The processor (1410) can store binary weight values ​​in synapse circuits included in the crossbar array circuit and obtain an input feature map from external memory (1630). Additionally, the processor (1410) can convert the input feature map into time domain binary vectors and provide the time domain binary vectors as input values ​​for the crossbar array circuit. Additionally, the processor (1410) can output an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors.

[0184] The embodiments may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and inseparable media. Additionally, a computer-readable medium may include both computer storage media and communication media. A computer storage medium includes both volatile and non-volatile, removable and inseparable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. A communication medium typically includes computer-readable instructions, data structures, other data of modulated data signals such as program modules, or other transmission mechanisms, and includes any information transmission medium.

[0185] Additionally, in this specification, "part" may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.

[0186] The foregoing description of this specification is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0187] The scope of the present embodiment is defined by the claims set forth below rather than by the detailed description above, and should be interpreted to include all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents.

Claims

Claim 1 A neuromorphic device implementing a neural network, comprising: a memory in which at least one program is stored; and an on-chip memory including a crossbar array circuit. A neuromorphic device comprising at least one processor that drives a neural network by executing at least one program, wherein the at least one processor stores binary weight values ​​converted from the initial weight values ​​based on the maximum and minimum values ​​of the initial weight values ​​in synapse circuits included in the crossbar array circuit, obtains an input feature map from the memory, converts each of the activations of the input feature map into temporal domain binary vectors represented as sequences of -1 and +1 elements based on which quantization level each of the activations of the input feature map belongs to among N (N is a natural number) quantization levels, provides the temporal domain binary vectors as input values ​​of the crossbar array circuit, and outputs an output feature map by performing a convolution operation between the binary weight values ​​and the temporal domain binary vectors. Claim 2 A neuromorphic device according to claim 1, wherein at least one processor outputs an output feature map by performing batch normalization on the result of the convolution operation. Claim 3 A neuromorphic device according to claim 2, wherein at least one processor calculates a modified scale value by multiplying the initial scale value of the batch normalization by the average value of the absolute values ​​of the initial weight values ​​and dividing by the number of elements included in each of the time domain binary vectors, and performs the batch normalization based on the modified scale value. Claim 4 delete Claim 5 In claim 1, the at least one processor is a neuromorphic device that divides the range between a maximum value and a minimum value that can be input to the neural network into the N quantization levels. Claim 6 In claim 5, the neuromorphic device wherein at least one processor distinguishes the range between a maximum value and a minimum value that can be input to the neural network into non-linear quantization levels. Claim 7 A neuromorphic device according to claim 3, wherein at least one processor performs a multiplication operation of multiplying each of the bias values ​​applied to the neural network by the initial scale value and reflecting the result of the multiplication operation in the output feature map. Claim 8 A neuromorphic device according to claim 3, wherein at least one processor performs batch normalization on the result of the convolution operation and outputs an output feature map by applying an activation function to the result of the batch normalization. Claim 9 A neural network device for implementing a neural network, comprising: a memory storing at least one program; and at least one processor for driving the neural network by executing the at least one program, wherein the at least one processor obtains binary weight values ​​converted from the initial weight values ​​based on the maximum and minimum values ​​of the input feature map and the initial weight values ​​from the memory, converts each of the activations of the input feature map into time domain binary vectors represented as sequences of -1 and +1 elements based on which quantization level each of the activations of the input feature map belongs to among N (N is a natural number) quantization levels, and outputs an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors. Claim 10 A neural network device according to claim 9, wherein at least one processor outputs an output feature map by performing batch normalization on the result of the convolution operation. Claim 11 A neural network device according to claim 10, wherein at least one processor calculates a modified scale value by multiplying the initial scale value of the batch normalization by the average value of the absolute values ​​of the initial weight values ​​and dividing by the number of elements included in each of the time domain binary vectors, and performs the batch normalization based on the modified scale value. Claim 12 delete Claim 13 In claim 9, the at least one processor is a neural network device that divides the range between a maximum value and a minimum value that can be input to the neural network into the N quantization levels. Claim 14 In claim 13, the neural network device wherein at least one processor distinguishes the range between a maximum value and a minimum value that can be input to the neural network into non-linear quantization levels. Claim 15 A neural network device according to claim 11, wherein at least one processor performs a multiplication operation of multiplying each of the bias values ​​applied to the neural network by the initial scale value, and reflects the result of the multiplication operation in the output feature map. Claim 16 A neural network device according to claim 11, wherein at least one processor performs batch normalization on the result of the convolution operation and outputs an output feature map by applying an activation function to the result of the batch normalization. Claim 17 A method for implementing a neural network in a neuromorphic device, comprising: a step of storing, under the control of a processor provided in the neuromorphic device, binary weight values ​​converted from initial weight values ​​based on the maximum and minimum values ​​of the initial weight values ​​in synapse circuits included in a crossbar array circuit; a step of acquiring an input feature map from memory in the processor; a step of converting each of the activations of the input feature map into time domain binary vectors represented as sequences of -1 and +1 elements based on which quantization level each of the activations of the input feature map belongs to among N (N is a natural number) quantization levels, under the control of the processor; a step of providing the time domain binary vectors as input values ​​to the crossbar array circuit under the control of the processor; and a step of outputting an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors under the control of the processor. Claim 18 A method for implementing a neural network in a neural network device, comprising: a step of obtaining binary weight values ​​converted from initial weight values ​​from memory based on maximum and minimum values ​​of an input feature map and initial weight values, under the control of a processor provided in the neural network device; a step of converting each of the activations of the input feature map into time domain binary vectors represented as sequences of -1 and +1 elements based on which quantization level each of the activations of the input feature map belongs to among N (N is a natural number) quantization levels, under the control of the processor; and a step of outputting an output feature map by performing a convolution operation between the binary weight values ​​and the time domain binary vectors under the control of the processor. Claim 19 A computer-readable recording medium having a program for executing the method of claim 17 on a computer. Claim 20 A computer-readable recording medium having a program for executing the method of claim 18 on a computer.

Images