Computational device and method of operation for multidimensional vector neural networks

Abstract

To provide a computing device for multidimensional vector neural network and a method of operating the same.SOLUTION: A disclosed computing device for a multidimensional vector neural network includes: a plurality of input lines to which a plurality of multidimensional input vectors are input; a plurality of output lines that intersect the plurality of input lines; a plurality of memory cells that are disposed at intersecting points between the plurality of input lines and the plurality of output lines and store weight elements included in a plurality of multidimensional weight vectors; a plurality of selectors that transmit a value output from each of the plurality of output lines to any one of a plurality of adders; and the plurality of adders that accumulate values transmitted from the plurality of selectors in a predetermined number of cycles. The weight elements included in each of the plurality of multidimensional weight vectors are stored in reference memory cells that connect a corresponding one reference input line and corresponding two or more reference output lines.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The following embodiments relate to a computing device for a multidimensional vector neural network and a method for operating the same. [Background technology]

[0002] A neuromorphic processor is capable of performing neural network computations. For example, a neuromorphic processor can be realized that includes neuronal and synaptic circuits. Such a neuromorphic processor can be used as a neural network device to drive various neural networks such as CNNs (Convolutional Neural Networks), RNNs (Recurrent Neural Networks), and FNNs (Feedforward Neural Networks), and can be utilized in various fields, including data classification and image recognition. [Overview of the project] [Problems that the invention aims to solve]

[0003] The objective of this invention is to efficiently perform multidimensional vector-based MAC operations, such as those involving complex numbers and quaternions, on analog in-memory computing hardware. [Means for solving the problem]

[0004] An arithmetic unit for a multidimensional vector neural network according to one embodiment includes a plurality of input lines into which a plurality of multidimensional input vectors are input, a plurality of output lines intersecting the plurality of input lines, a plurality of memory cells arranged at the intersections of the plurality of input lines and the plurality of output lines and storing weight elements included in the plurality of multidimensional weight vectors, a plurality of selectors that transmit the values ​​output from each of the plurality of output lines to one of a plurality of adders, and a plurality of adders that accumulate the values ​​transmitted from the plurality of selectors over a predetermined number of cycles, wherein the weight elements included in each of the plurality of multidimensional weight vectors are stored in a reference memory cell that connects a corresponding reference input line and two or more corresponding reference output lines.

[0005] In a computing device according to one embodiment, the input elements included in each of the plurality of multidimensional input vectors can be sequentially input to the same reference input line during the cycle.

[0006] In one embodiment of the computing device, the number of cycles may be the same as the number of input elements included in each of the plurality of multidimensional input vectors.

[0007] In a calculation device according to one embodiment, the output elements included in each of a plurality of multidimensional output vectors can be determined based on the values ​​accumulated in each of the reference adders corresponding to the reference output line.

[0008] In one embodiment of the arithmetic unit, the reference selector among the plurality of selectors corresponding to the reference output line can divide the values ​​output from each of the reference output lines into the output elements of each of the multidimensional output vectors and transmit them to other reference adders.

[0009] In a calculation device according to one embodiment, the number of the plurality of output lines can be determined based on the number of multidimensional weight vectors and the number of weight value elements contained in each of the multidimensional weight vectors.

[0010] In a computing device according to one embodiment, any one of the weight elements included in each of the plurality of multidimensional weight vectors can be stored in a reference memory cell connected to a different reference output line from the other one.

[0011] In an arithmetic device according to one embodiment, the plurality of multidimensional input vectors, the plurality of multidimensional weight vectors, and the plurality of multidimensional output vectors determined based on the values ​​accumulated in each of the plurality of adders may be complex vectors including real and imaginary elements.

[0012] In an arithmetic unit according to one embodiment, the value output from a first reference output line connected to a memory cell storing real number elements among the reference output lines is transmitted via a first reference selector to a second reference adder that accumulates imaginary number elements while retaining its sign, and the value output from a second reference output line connected to a memory cell storing imaginary number elements among the reference output lines is transmitted via a second reference selector to a first reference adder that accumulates real number elements with its sign reversed.

[0013] In an arithmetic device according to one embodiment, the plurality of multidimensional input vectors, the plurality of multidimensional weight vectors, and the plurality of multidimensional output vectors determined based on the values ​​accumulated in each of the plurality of adders may be a quaternion vector containing a plurality of imaginary elements or a quaternion vector containing a real element and a plurality of imaginary elements.

[0014] An arithmetic unit according to one embodiment may further include a plurality of second selectors that determine the sign of the value output from each of the plurality of output lines based on the type of multidimensional weight vector to be applied to the calculation to be performed in each cycle, the type of weight element stored in the plurality of memory cells, and the type of input element input to the plurality of input lines, and transmit the value whose sign has been determined to the corresponding selector.

[0015] In an arithmetic unit according to one embodiment, the elements included in each of the partial calculation result vectors accumulated by the plurality of adders during a part of the cycle can be sequentially input to the plurality of input lines during the remainder of the cycle.

[0016] In one embodiment of the arithmetic unit, the predetermined number of cycles is 2 in response to the plurality of multidimensional weight vectors being complex number vectors, and may be 7 or 8 in response to the plurality of multidimensional weight vectors being quaternion vectors.

[0017] A method of operation for a computing device for a multidimensional vector neural network according to one embodiment includes the steps of: inputting input elements contained in each of a plurality of multidimensional input vectors into a plurality of input lines; transmitting values ​​output from each of a plurality of output lines intersecting the plurality of input lines to one of a plurality of adders via a plurality of selectors; and accumulating the values ​​transmitted from the plurality of selectors via the plurality of adders over a predetermined number of cycles, wherein the weight elements contained in each of a plurality of multidimensional weight vectors applied to the multidimensional vector neural network can be stored in a reference memory cell that connects a corresponding reference input line and two or more corresponding reference output lines. [Effects of the Invention]

[0018] According to the present invention, multidimensional vector neural operations such as complex number-based MAC operations and / or quaternion-based MAC operations can be realized using low-power in-memory computing. Furthermore, complex number-based MAC operations and / or quaternion-based MAC operations can be performed on a crossbar array while maintaining the effect of reducing the model size of the multidimensional neural network. [Brief explanation of the drawing]

[0019] [Figure 1] This is a diagram illustrating the architecture of a neural network according to one embodiment. [Figure 2] This is a diagram illustrating the operations performed by a neural network according to one embodiment. [Figure 3] This figure shows an in-memory computing circuit according to one embodiment. [Figure 4] This figure illustrates an operation performed by a complex neural network according to one embodiment. [Figure 5] This is a diagram illustrating the structure and operation of a computing device for a complex neural network according to one embodiment. [Figure 6] This is a diagram illustrating the structure and operation of a computing device for a complex neural network according to one embodiment. [Figure 7] This is a diagram illustrating the structure and operation of a computing device for a complex neural network according to one embodiment. [Figure 8] This figure illustrates an operation performed by a quaternion neural network according to one embodiment. [Figure 9] This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 10] This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 11]This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 12] This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 13] This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 14] This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 15] This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 16] This figure illustrates the structure and operation of a computing device for a quaternion neural network according to one embodiment. [Figure 17] This figure illustrates the structure and operation of the computing device for a quaternion neural network according to another embodiment. [Figure 18] This figure shows the operation method of a computing device according to one embodiment. [Figure 19] This figure shows an electronic device according to one embodiment. [Modes for carrying out the invention]

[0020] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and can be modified in various ways. Therefore, the embodiments are not limited to the specific disclosure, and the scope of this specification includes modifications, equivalents, or substitutions that are part of the technical idea.

[0021] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0022] When it is mentioned that one component is “linked” or “connected” to another component, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.

[0023] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to presuppose the existence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0024] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0025] The embodiments will be described in detail below with reference to the attached drawings. When describing with reference to the drawings, the same components will be given the same reference numerals regardless of the reference numerals used in the drawings, and redundant explanations for them will be omitted.

[0026] Figure 1 is a diagram illustrating the architecture of a neural network according to one embodiment.

[0027] The neural network 100 can be represented as a mathematical model using nodes and edges. The neural network 100 may be an architecture of a Deep Neural Network (DNN) or an n-layers neural network. DNNs or n-layers neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks, and restricted Boltzmann machines. For example, the neural network 100 is implemented as a convolutional neural network (CNN), but is not limited to this. The neural network 100 shown in Figure 1 is a part of a convolutional neural network. Therefore, the neural network 100 may be a convolutional layer, a pooling layer, a fully connected layer, etc. However, for convenience, the following explanation will assume that neural network 100 corresponds to the convolutional layer of a convolutional neural network.

[0028] In a convolutional layer, the first feature map (FM1) corresponds to the input feature map, and the second feature map (FM2) corresponds to the output feature map. For example, the first feature map (FM1) represents a dataset that expresses various features of the input data, and the second feature map (FM2) represents a dataset that expresses various features of the output data determined by the convolution operation performed by applying a weight map (WM) to the first feature map (FM1). Feature maps FM1 and FM2 may be high-dimensional matrices of two or more dimensions and each has its own activation parameter. If feature maps FM1 and FM2 correspond to, for example, three-dimensional feature maps, then feature maps FM1 and FM2 have a width W (or column), a height H (or row), and a depth C. Here, the depth C corresponds to the number of channels.

[0029] In the convolution layer, a convolution operation is performed on the first feature map FM1 and the weight map WM, resulting in the generation of the second feature map FM2. The weight map WM filters the first feature map FM1 and may be called a weight filter or weight kernel. For example, the depth of the weight map WM, in other words, the number of channels, may be the same as the depth of the first feature map FM1, in other words, the number of channels. The weight map WM may be shifted by traversing the first feature map FM1 as a sliding window. Between each shift, each weight contained in the weight map WM is multiplied and added with all feature values ​​in the region overlapping with the first feature map FM1. The convolution of the first feature map FM1 and the weight map WM generates one channel of the second feature map FM2.

[0030] Figure 1 shows a single weight map WM, but in reality, multiple weight maps may be convolved with the first feature map FM1 to generate multiple channels of the second feature map FM2. On the other hand, the second feature map FM2 of the convolutional layer can become the input feature map of the next layer. For example, the second feature map FM2 may become the input feature map of a pooling layer or the input feature map of a subsequent convolutional layer. However, it is not limited to these.

[0031] Figure 2 is a diagram illustrating the operations performed by a neural network according to one embodiment.

[0032] The neural network 200 has a structure including an input layer, a hidden layer, and an output layer, and performs calculations based on the received input data (e.g., I1 and I2), and generates output data (e.g., O1 and O2) based on the results of the calculations.

[0033] As previously described, the neural network 200 may be a DNN or an n-layer neural network containing two or more hidden layers. For example, as shown in Figure 2, the neural network 200 may be a DNN containing an input layer Layer 1, two hidden layers Layer 2 and Layer 3, and an output layer Layer 4. When the neural network 200 is embodied as a DNN architecture, it can process more complex data sets than a single-layer neural network because it contains many layers capable of processing useful information. On the other hand, although the neural network 200 is illustrated as containing four layers, this is merely illustrative, and the neural network 200 may contain fewer or more layers, or fewer or more channels. That is, the neural network 200 can contain layers of various structures different from those shown in Figure 2.

[0034] Each layer in the neural network 200 may contain multiple channels. Each channel may contain or represent multiple artificial nodes, known as neurons, processing elements (PEs), units, or similar terms. Although nodes are referred to as "artificial nodes" or "neurons," such references are not intended to imply any connection to how neural network architectures are computationally mapped, how information is intuitively perceived, or how human neurons operate. In other words, the terms "artificial nodes" or "neurons" are merely technical terms indicating hardware implementation nodes of the neural network. As shown in Figure 2, Layer 1 may contain two channels (or nodes), and Layers 2 and 3 may each contain three channels. However, this is merely an example, and each layer in the neural network 200 may contain a different number of channels (or nodes).

[0035] The channels contained within each layer of the neural network 200 can be linked together to process data. For example, one channel can receive data from other channels, perform calculations on it, and output the result of those calculations to yet another channel.

[0036] The output value of a channel can be referred to as the activation or the value determined by the channel's predetermined activation function. The input and output of each channel may also be referred to as the input activation and output activation, respectively. That is, activation can be the output of one channel and, simultaneously, a parameter corresponding to the input of a channel in the next layer via a corresponding connection to the next layer. On the other hand, each channel can determine its own activation based on the activations and weights received from channels in previous layers. Weights are values ​​assigned to the connection relationships between channels, used as parameters to calculate the output activation in each channel. For example, the output from a channel in a previous layer can be provided as input to a channel in the next layer via a weighted connection between the channel in the previous layer and the channel in the next layer. The weights of the weighted connection can be adjusted in various ways during neural network training until the neural network learns to achieve its desired goal. Additional connections to channels in subsequent layers may be provided to the aforementioned exemplary recurrent connections, which provide bias connection values ​​via weighted or unweighted connections, and / or may be weighted. During training and implementation, such connections and connection weights can be arbitrarily implemented, removed, and modified to generate and obtain the final neural network.

[0037] Each channel can be processed by an arithmetic unit or processing element that receives an input and outputs an activation, and the input-output of each channel is mapped. The arithmetic unit can execute an activation function on the node. For example, σ is an activation function, and w jk iis the weight from the k-th channel in the (i - 1)-th layer to the j-th channel in the i-th layer, and b j i is the bias of the j-th channel in the i-th layer, and a j i When taking the activation of the j-th channel in the i-th layer as the activation a j i can be calculated using the following formula (1).

[0038]

Equation

[0039]

Equation

[0040] As described above, in the neural network 200, a large number of data sets are propagated between a plurality of interconnected channels, and the calculation process is performed through the layers. In such a calculation process, a large number of multiply-accumulate (MAC) operations are executed together with a large number of memory access operations for loading the activation and weight, which are the operands of the MAC operation, at an appropriate time.

[0041] On the other hand, a typical digital computer separates the arithmetic unit and memory, and can use a von Neumann structure that includes a common data bus for data transmission between the two separated blocks. Therefore, in the process of implementing a neural network 200, in which data movement and calculations are repeatedly performed, using a typical digital computer, a lot of time is spent on data transmission, and excessive power is consumed. To overcome these technical problems, an in-memory computing circuit has been proposed as an architecture that integrates the memory and arithmetic unit for MAC calculations into one, thereby reducing the time required for data transmission and lowering power consumption. The in-memory computing circuit will be described in more detail below with reference to Figure 3.

[0042] Figure 3 shows an in-memory computing circuit according to one embodiment.

[0043] The in-memory computing circuit 300 includes an analog crossbar array 310 and ADCs (Analog to Digital Converters) 320. However, the in-memory computing circuit 300 shown in Figure 3 only illustrates the components relating to this embodiment. Therefore, it will be obvious to those skilled in the art that the in-memory computing circuit 300 may further include other general-purpose components in addition to those shown in Figure 3.

[0044] The analog crossbar array 310 includes a plurality of row lines 311, a plurality of column lines 312, and a plurality of memory cells 313. The plurality of row lines 311 are used to receive input data and may be referred to as a plurality of input lines. For example, if the plurality of row lines 311 are N row lines (where N is any natural number), then the N row lines have voltages V1, V2, ..., V corresponding to input activation. NA value is applied. Multiple column lines 312 intersect with multiple row lines 311. For example, if there are M column lines 312 (where M is any natural number), the multiple column lines 312 and the multiple row lines 311 intersect at N × M intersections.

[0045] On the other hand, multiple memory cells 313 are located at the intersections of multiple row lines 311 and multiple column lines 312. Each of the multiple memory cells 313 is implemented as a non-volatile memory such as ReRAM (Resistive RAM), MRAM (Magnetic RAM), or eFlash to store weights, but is not necessarily limited to this. In other non-restrictive embodiments, each of the multiple memory cells 313 may be a volatile memory such as SRAM (Static Random Access Memory).

[0046] In the example shown in Figure 3, the multiple memory cells 313 have a conductance G corresponding to the weight. 11 ..., G NM It has such that when a voltage corresponding to input activation is applied to each of the multiple row lines 311, a current of magnitude I = V × G according to Ohm's law can be output through each memory cell 313. The currents output from memory cells arranged along one column line combine with each other, so the sum of the currents I1, ..., I along the multiple column lines 312 is M The output is the sum of currents I1, ..., I M This corresponds to the result of MAC calculation performed in an analog manner. Therefore, the multiple column lines 312 may be referred to as multiple output lines.

[0047] One or more ADCs320 receive the results of analog MAC calculations output from the analog crossbar array 310 (in other words, the sum of currents I1, ..., I1). MThe signal is converted into a digital signal. The result of the MAC operation, converted into a digital signal, is output from the ADCs320 and used in the subsequent neural network computation process.

[0048] As shown in Figure 3, the in-memory computing circuit 300 according to this embodiment is an effective hardware architecture for low-power driving of neural network models by performing MAC calculations with low power. The in-memory computing circuit 300 simultaneously stores weight values ​​and performs MAC calculations in the analog crossbar array 310, thus reducing energy consumption due to the movement of model parameters. Compared to the von Neumann digital architecture, it can significantly reduce the energy consumed for driving and calculating neural networks. The operation of the analog crossbar array 310 involves input values ​​being input to each row line 311, which are then transmitted identically to each of the multiple memory cells located on the same row line. These memory cells are then multiplied with the weight values ​​stored in them, and the results of the multiplication calculations for each memory cell are summed up for each column line 312 and output.

[0049] The analog crossbar array 310 shown in Figure 3 is based on a real-valued neural network that performs learning and inference based on real-valued MAC operations, which are commonly used in the field of artificial intelligence. For convenience of explanation, in this specification, the real-valued neural network may be referred to as an RVNN (real-valued neural network).

[0050] There are neural networks that perform learning and inference based on multidimensional vector MAC operations, which are different from the above. Neural networks that perform complex-value-based MAC operations are called CVNNs (complex-valued neural networks), and neural networks that perform quaternion-based MAC operations are called QNNs (quaternion neural networks). In unrestricted embodiments, CVNNs and QNNs show higher learning performance than RVNNs in application fields such as speech recognition, image recognition, and spatial recognition.

[0051] The following provides a detailed explanation of the efficient structure and operation method of a computing device that performs complex number-based or quaternion-based MAC operations, with reference to the drawings.

[0052] Figure 4 is a diagram illustrating the operations performed by a complex neural network according to one embodiment.

[0053] Referring to Figure 4, a complex-based neural network is shown. The input vectors are 2-dimensional vectors, and each input vector X i The real number element R Xi and the imaginary element I Xi It includes (where i is a natural number). Also, the output vector is a 2D vector, and each output vector Y j The real number element R Yj and the imaginary element I Yj This includes (where j is a natural number). Also, the weight vectors are 2D vectors, and each weight vector W ij This indicates the connection weight between the i-th input node and the j-th output node, and is represented by the real element R. Wij and the imaginary element I Wij It may include.

[0054] A complex number is a two-dimensional vector represented by one real element and one imaginary element i. For example, a complex number C1 = a1 + b 1i and C2 = a2 + b 2iMultiplication and addition between them are performed as follows:

[0055] C1+C2=(a1+a2)+(b1+b2)i C1 × C2 = (a1 × a2 - b1 × b2) + (a1 × b2 + a2 × b1) i The complex number-based MAC operation uses a complex number input P. c l =a p l +b p l i and complex number weight W c lm =a w lm +b w lm It is executed based on multiplication and addition between i.

[0056]

number

[0057] In an RVNN, since each input node receives and outputs only one-dimensional real values, it is possible to separate the real and imaginary elements of N complex signals obtained via the Fourier transform and input their respective values ​​to 2N input nodes, or to input only the real element values ​​of each complex signal to N input nodes and perform learning and inference. Because the RVNN operates without the correlation between the real and imaginary elements of each complex signal—in other words, the correlation between the intensity and phase of the sound wave signal—learning and inference performance may be low, or a considerable model size and a lot of computational power are required to reconstruct the correlation between the real and imaginary elements that the complex sound wave signal originally possessed in the RVNN and improve learning and inference performance.

[0058] On the other hand, the CVNN according to this embodiment can take complex numbers as inputs and output them at each input node. Unlike a typical RVNN, it can directly input complex numbers obtained via the Fourier transform and perform learning and inference. In other words, because it can perform learning and inference while preserving the correlation between the real and imaginary elements of the complex number signal, the CVNN can achieve even higher learning and inference performance compared to a typical RVNN.

[0059] Since most signals existing in natural systems, such as sound wave signals, are defined as multidimensional vectors, neural networks based on multidimensional vector MAC operations (e.g., CVNN, QNN) can be expected to achieve even higher learning and inference performance compared to RVNNs. However, as explained in equation (2), complex number-based MAC operations require four times more computation per MAC operation compared to RVNNs. To address the technical issues of complex number-based MAC operations, the structure and operation method of a computing device that efficiently performs complex number-based MAC operations will be described in detail below according to an embodiment.

[0060] Figures 5 to 7 illustrate the structure and operation of a computing device for a complex neural network according to one embodiment.

[0061] Referring to Figure 5, the arithmetic unit that performs complex number-based MAC arithmetic includes a crossbar array, a plurality of selectors (shown as "Sel" in Figure 5), and a plurality of adders. The crossbar array may include N input lines into which a plurality of 2D input vectors 510 are input, 2M output lines intersecting the input lines, and N × 2M memory cells located at the intersections of the input and output lines.

[0062] In complex-based MAC arithmetic, the commutative property holds between the real and imaginary elements of a two-dimensional input vector and the real and imaginary elements of a two-dimensional weight vector, as shown in equation (2). To realize such complex-based MAC arithmetic as an arithmetic unit including a crossbar array, first, the weight elements contained in each two-dimensional weight vector can be stored in a reference memory cell that connects a corresponding reference input line and two corresponding reference output lines. For example, weight vector W 11 The real number element R W11 The weight vector W is stored in a memory cell located at the intersection between the input line of the first row and the output line of the first column. 11 In this case, the imaginary element I W11 These are stored in a memory cell located at the intersection between the input line of the first row and the output line of the second column. In summary, the weight elements (e.g., real and imaginary elements) contained in each of the multiple two-dimensional weight vectors can be stored in a reference memory cell connected to different reference output lines.

[0063] On the other hand, each of the multiple input lines corresponds to one input node, while two of the multiple output lines correspond to one output node. If there are M output nodes, the crossbar array may contain 2M output lines. Of the two output lines 520 corresponding to one output node, one memory cell connected to one may store the real elements of the weight vector connected to that output node, and the other memory cell connected to the other may store the imaginary elements of the weight vector connected to that output node. A pair of two output lines corresponds to one output node.

[0064] N input lines may be connected to N two-dimensional input vectors 510. The input elements contained in each input vector 510 are sequentially connected to the same reference input line. For example, the real element R of the first input vector X1. X1 This is input to the first low input line in the first cycle. The imaginary element I of the first input vector X1 X1 This is input to the input line of the first row in the second cycle. The input vector 510 is input to the corresponding input line between the two cycles. The number of cycles to which the input vector 510 is input (e.g., total or determined number) may be the same as the number of dimensions of the input vector 510.

[0065] The input elements of the input vector 510, which are sequentially input to the input line over two cycles, are multiplied with the weight elements stored in the memory cell, and the sum of the multiplication is determined and output by the output line. The calculation results output from each output line can be transmitted to one of several adders via several selectors 530. For example, the calculation result of one of the reference output lines corresponding to the first output node can be transmitted to one of two adders via the corresponding first selector, and the calculation result of the other of the reference output lines can be transmitted to the other of the two adders via the corresponding second selector.

[0066] Multiple adders can accumulate the results of operations transmitted via a selector between two cycles and determine the output elements contained in the output vector. For example, the result of operations accumulated between two cycles in the first adder is the real element R of the first output vector. Y1 It may be determined that the result of the operation accumulated between two cycles in the second adder is the imaginary element I of the first output vector. Y1 It may be decided that way.

[0067] The operation of the arithmetic unit that performs complex number-based MAC calculations will be explained in detail with reference to Figures 6 and 7.

[0068] Referring to Figure 6, the operation of the arithmetic unit in the first cycle of a complex-based MAC operation is shown. In the first cycle, the real elements of each 2D input vector are input to the corresponding input line. The real elements of each 2D input vector input to the input line are multiplied by the weight elements stored in the memory cell. For example, the real element R of the first input vector input to the input line of the first row X1 This is a weight element R stored in multiple memory cells connected to the first low input line. W11 , I W11 ,..., R W1M , I W1M These can also be multiplied together.

[0069] Each of the selectors can transmit the calculation result output from each output line to one of the multiple adders. For example, the calculation result 610 output from the output line of the first column is R X1 ×R W11 +R X2 ×R W21 +R X3 ×R W31 +...+R XN ×R WN1The result can be the MAC operation result for the real elements of each input vector and the real elements of each weight vector with respect to the first output node. The first selector 630 can transmit the operation result 610 to the first adder 650. The operation result 620 output from the output line of the second column is R X1 ×I W11 +R X2 ×I W21 +R X3 ×I W31 +...+R XN ×I WN1 This could be the result of a MAC operation on the real elements of each input vector and the imaginary elements of each weight vector with respect to the first output node. The second selector 640 can transmit the operation result 620 to the second adder 660.

[0070] In the above explanation, the calculation results and their transmission behavior performed in the output lines of the first and second columns can be similarly applied to the output lines of the remaining columns; therefore, a more detailed explanation is omitted.

[0071] Referring to Figure 7, the operation of the arithmetic unit in the second cycle of a complex-based MAC operation is shown. In the second cycle, the imaginary elements of each 2D input vector may be input to the corresponding input line. The imaginary elements of each 2D vector input to the input line are multiplied by the weight elements stored in the memory cell. For example, the imaginary element I of the first input vector input to the input line of the first row X1 This is a weight element R stored in multiple memory cells connected to the first low input line. W11 , I W11 ,..., R W1M , I W1M These can also be multiplied together.

[0072] Each of the plurality of selectors can transmit the calculation result output from each output line to any one of the plurality of adders. For example, the calculation result 710 output from the output line of the first column can be I X1 ×R W11 +I X2 ×R W21 +I X3 ×R W31 +...+I XN ×R WN1 and can be the MAC calculation result for the imaginary element of each input vector and the real element of each weight vector regarding the first output node. The first selector 730 (for example, the first selector 630 shown in FIG. 6) can transmit the calculation result 710 to the second adder 760 (for example, the second adder 660 shown in FIG. 6). Also, the calculation result 720 output from the output line of the second column can be I X1 ×I W11 +I X2 ×I W21 +I X3 ×I W31 +...+I XN ×I WN1 and can be the MAC calculation result for the imaginary element of each input vector and the imaginary element of each weight vector regarding the first output node. Since the calculation result 720 is the sum of multiplications between imaginary elements, the second selector 740 (for example, the second selector 640 shown in FIG. 6) can invert the sign of the calculation result 720 and transmit it to the first adder 750 (for example, the first adder 650 shown in FIG. 6).

[0073] The first adder 750 accumulates the calculation results transmitted between two cycles, and the real element of the first output vector can be determined as the result accumulated by the first adder 750. The second adder 760 accumulates the calculation results transmitted between two cycles, and the imaginary element of the first output vector can be determined as the result accumulated by the second adder 760.

[0074] For example, the operation of complex number-based MAC executed by an arithmetic unit can be expressed by the following mathematical formula (3).

[0075]

Number

[0076] For the sake of convenience of explanation, the operation of the first output node has been described, but the above explanation can also be similarly applied to the operation of the remaining output nodes. Thus, even with only two cycle operations of the arithmetic unit, the complex number-based MAC operation result can be efficiently obtained, thereby improving a general arithmetic unit that can obtain the complex number-based MAC operation result with a larger number of cycles and / or operations.

[0077] FIG. 8 is a diagram for explaining an operation executed by a quaternion neural network according to an embodiment.

[0078] Referring to FIG. 8, a quaternion-based neural network is shown. The input vector is a 4-dimensional vector, and each input vector X i has real number elements R Xi and imaginary number elements I Xi , J Xi , K Xi (where i is a natural number). Also, the output vector is a 4-dimensional vector, and each output vector Y j has real number elements R Yj and imaginary number elements I Yj , J Yj , K YjThis includes (where j is a natural number). Also, the weight vectors are 4-dimensional vectors, and each weight vector W ij This indicates the connection weight between the i-th input node and the j-th output node, and is represented by the real element R. Wij and the imaginary element I Wij , J Wij , K Wij It may include.

[0079] A quaternion is a four-dimensional vector represented by one real element and three imaginary elements i, j, and k, where the three imaginary elements i, j, and k are orthogonal to each other. Therefore, the quaternion Q1 = a1 + b 1i +c 1j +d 1k And Q2 = a2 + b 2i +c 2j +d 2k Multiplication and addition between them can be performed, for example, as follows:

[0080] Q1+Q2=(a1+a2)+(b1+b2)i+(c1+c2)j+(d1+d2)k Q1×Q2=(a1×a2-b1×b2-c1×c2-d1×d2)+(a1×b2+b1×a2+c1×d2-d1×c2)i+(a1×c2-b1×d2+c1×a2+d1×b2)j+(a1×d2+b1×c2-c1×b2+d1×a2)k The quaternion-based MAC operation uses a quaternion input a p l +b p l i+c p l j+d p l k and quaternion weight W q lm= a w lm+ b w lm i+c w lm j+d w lm k, conjugate of quaternion weights W q lm * =a wlm -b w lm I C w lm jd w lm It is performed based on multiplication and addition between k terms and can be expressed as shown in equation (4) below.

[0081]

number

[0082] On the other hand, in the embodiment, since each input node of the QNN can receive and output quaternion values, the x, y, and z values ​​of 3D spatial coordinate data can be represented by the three imaginary elements i, j, and k of the quaternion, and learning and inference can be performed while maintaining the correlation between the x, y, and z values. In other words, since learning and inference can be performed while maintaining the correlation between the x, y, and z values ​​that the 3D spatial coordinate data has, the QNN can achieve even higher learning and inference performance compared to RVNN.

[0083] In image recognition, QNN can achieve even higher learning and inference performance compared to RVNN by representing the R, G, and B values ​​of each pixel with three imaginary elements i, j, and k of a quaternion, and performing learning and inference while maintaining the correlation between the R, G, and B values ​​(where the correlation between the R, G, and B values ​​represents detailed color information of each pixel).

[0084] Since various signals existing in natural systems, such as 3D spatial coordinate data and pixel values, are mostly defined as multidimensional vectors, higher learning and inference performance can be expected in neural networks based on multidimensional vector MAC operations (e.g., CVNN, QNN) compared to general RVNNs. However, as explained in equation (4), quaternion-based MAC operations can perform 32 times more operations per MAC operation compared to RVNNs. To address the technical issues of quaternion-based MAC operations, the structure and operation method of a computing device that efficiently performs quaternion-based MAC operations according to an embodiment will be described in detail below.

[0085] Figures 9 to 16 are diagrams illustrating the structure and operation of a computing device for a quaternion neural network according to one embodiment.

[0086] An arithmetic unit that performs quaternion-based MAC operations includes a crossbar array, multiple sign selectors, multiple adder selectors, and multiple adders. The crossbar array may include N input lines into which multiple 4-dimensional input vectors are input, 4M output lines intersecting the input lines, and N × 4M memory cells located at the intersections of the input and output lines. Since a quaternion contains one real element and three imaginary elements, 4M output lines can be used to determine M output vectors.

[0087] In quaternion-based MAC arithmetic, as shown in equation (3), a commutative law holds between the real elements and three imaginary elements of the 4-dimensional input vector, and between the real elements and three imaginary elements of the weight vector. To implement such quaternion-based MAC arithmetic in an arithmetic unit including a crossbar array, first, the weight elements contained in each of the 4-dimensional weight vectors can be stored in a reference memory cell that connects a corresponding reference input line and four corresponding reference output lines. For example, weight vector W 11 The real number element R W11The weight vector W may be stored in a memory cell located at the intersection between the input line of the first row and the output line of the first column. 11 The imaginary element I W11 The weight vector W may be stored in a memory cell located at the intersection between the input line of the first row and the output line of the second column. 11 The imaginary element J W11 The weight vector W may be stored in a memory cell located at the intersection between the input line of the first row and the output line of the third column. 11 The imaginary element K W11 These may be stored in a memory cell located at the intersection between the input line of the first row and the output line of the fourth column. In summary, the weight elements (e.g., real elements and three imaginary elements) contained in each of the multiple four-dimensional weight vectors may be stored in a reference memory cell connected to different reference output lines. In some embodiments, to maximize the operational efficiency of the crossbar array operation, the values ​​obtained by dividing each of the real elements and three imaginary elements of each weight vector by the magnitude of the weight vector (e.g., the result of multiplying the weight vector by its conjugate weight vector) may be stored in the memory cell.

[0088] On the other hand, while each of the multiple input lines corresponds to one input node, four of the multiple output lines may correspond to one output node. If there are M output nodes, the crossbar array may contain 4M output lines. Of the four output lines corresponding to one output node, the memory cell connected to the first output line may store the real element of the weight vector connected to that output node, the memory cell connected to the second output line may store the imaginary element i of the weight vector connected to that output node, the memory cell connected to the third output line may store the imaginary element j of the weight vector connected to that output node, and the memory cell connected to the fourth output line may store the imaginary element k of the weight vector connected to that output node. A pair of four output lines corresponds to one output node.

[0089] Unlike the example of an arithmetic unit performing complex-based MAC operations, an arithmetic unit performing quaternion-based MAC operations may further include a plurality of sign selectors and a plurality of adder selectors. The operation result output on each output line can have its sign determined by the corresponding sign selector and be transmitted to one of the plurality of adders by the corresponding adder selector. According to an unrestrictive embodiment, an arithmetic unit for performing quaternion-based MAC operations may include components of an arithmetic unit for performing complex-based MAC operations, and thus may perform both complex-based MAC operations and quaternion-based MAC operations.

[0090] The operation of an arithmetic unit that performs quaternion-based MAC operations over eight cycles according to one embodiment will be described in detail with reference to Figures 9 and 16.

[0091] Referring to Figure 9, the operation of the arithmetic unit in the first cycle of a quaternion-based MAC operation is shown. In the first cycle, the real elements of each of the four-dimensional input vectors are input to the corresponding input lines. The real elements of each four-dimensional input vector input to the input line may be multiplied by the weight elements stored in the memory cell. For example, the real elements R of the first input vector input to the input line of the first row. X1 This is a weight element R stored in multiple memory cells connected to the first low input line. W11 , I W11 , J W11 , K W11 ,..., R W1M , I W1M , J W1M , K W1M These can also be multiplied together.

[0092] Each of the sign selectors determines the sign of the calculation result output from each output line, and each of the adder selectors can transmit the calculation result output from each output line to one of the adders. For example, the calculation result 910 output from the output line of the first column is R X1 ×R W11 +R X2 ×R W21 +R X3 ×R W31 +...+R XN ×R WN1 The result can be the MAC operation result for the real elements of each input vector and the real elements of each weight vector with respect to the first output node. The first sign selector determines the sign of the operation result 910 to be +, and the first adder selector can transmit the operation result 910 to the first adder. Also, the operation result 920 output from the output line of the second column is R X1 ×I W11 +R X2 ×I W21 +R X3 ×I W31 +...+R XN ×I WN1The result can be the MAC operation result for the real elements of each input vector and the imaginary element i of each weight vector with respect to the first output node. The second sign selector determines the sign of the operation result 920 to be +, and the second adder selector can transmit the operation result 920 to the second adder. Also, the operation result 930 output from the output line of the third column is R X1 ×J W11 +R X2 ×J W21 +R X3 ×J W31 +...+R XN ×J WN1 This can be the MAC operation result for the real elements of each input vector and the imaginary element j of each weight vector with respect to the first output node. The third sign selector determines the sign of the operation result 930 to +, and the third adder selector can transmit the operation result 930 to the third adder. Also, the operation result 940 output from the output line of the fourth column is R X1 ×K W11 +R X2 ×K W21 +R X3 ×K W31 +...+R XN ×K WN1 This could be the MAC operation result for the real elements of each input vector and the imaginary element k of each weight vector with respect to the first output node. The fourth sign selector determines the sign of the operation result 940 to be +, and the fourth adder selector can transmit the operation result 940 to the fourth adder.

[0093] In the above explanation, the calculation results and their transmission behavior performed in the output lines of the first to fourth columns can be similarly applied to the output lines of the remaining columns; therefore, a more detailed explanation is omitted.

[0094] Referring to Figure 10, the operation of the arithmetic unit is shown in the second cycle of the quaternion-based MAC operation. In the second cycle, the imaginary element i of each 4-dimensional input vector is input to the corresponding input line. The imaginary element i of each 4-dimensional input vector input to the input line is multiplied by the weight element stored in the memory cell. For example, the imaginary element i of the first input vector input to the input line of the first row X1 This is a weight element R stored in multiple memory cells connected to the first low input line. W11 , I W11 , J W11 , K W11 ,..., R W1M , I W1M , J W1M , K W1M These can also be multiplied together.

[0095] Each of the sign selectors determines the sign of the calculation result output from each output line, and each of the adder selectors can transmit the calculation result output from each output line to one of the adders. For example, the calculation result 1010 output from the output line of the first column is I X1 ×R W11 +I X2 ×R W21 +I X3 ×R W31 +...+I XN ×R WN1 This can be the result of a MAC operation on the imaginary element i of each input vector and the real element of each weight vector with respect to the first output node. The first sign selector determines the sign of the operation result 1010 to be +, and the first adder selector can transmit the operation result 1010 to the second adder. Also, the operation result 1020 output from the output line of the second column is I X1 ×I W11 +I X2 ×I W21 +I X3 ×I W31 +...+I XN ×I WN1The result can be the MAC operation result for the imaginary element i of each input vector and the imaginary element i of each weight vector with respect to the first output node. The second sign selector determines the sign of the operation result 1020 to -, and the second adder selector can transmit the operation result 1020 to the first adder. Also, the operation result 1030 output from the output line of the third column is I X1 ×J W11 +I X2 ×J W21 +I X3 ×J W31 +...+I XN ×J WN1 The result can be the MAC operation result for the imaginary element i of each input vector and the imaginary element j of each weight vector with respect to the first output node. The third sign selector determines the sign of the operation result 1030 to -, and the third adder selector can transmit the operation result 1030 to the fourth adder. Also, the operation result 1040 output from the output line of the fourth column is I X1 ×K W11 +I X2 ×K W21 +I X3 ×K W31 +...+I XN ×K WN1 This could be the result of a MAC operation on the imaginary element i of each input vector and the imaginary element k of each weight vector with respect to the first output node. The fourth sign selector determines the sign of the operation result 1040 to be +, and the fourth adder selector can transmit the operation result 1040 to the third adder.

[0096] In the above explanation, the calculation results and their transmission behavior performed in the output lines of the first to fourth columns can be similarly applied to the output lines of the remaining columns; therefore, a more detailed explanation is omitted.

[0097] Referring to Figure 11, the operation of the arithmetic unit is shown in the third cycle of a quaternion-based MAC operation. In the third cycle, the imaginary element j of each 4-dimensional input vector can be input to the corresponding input line. The imaginary element j of each 4-dimensional input vector input to the input line may be multiplied by a weight element stored in the memory cell. For example, the imaginary element j of the first input vector input to the input line of the first row. X1 This is a weight element R stored in multiple memory cells connected to the first low input line. W11 , I W11 , J W11 , K W11 ,..., R W1M , I W1M , J W1M , K W1M These can also be multiplied together.

[0098] Each of the sign selectors determines the sign of the calculation result output from each output line, and each of the adder selectors can transmit the calculation result output from each output line to one of the adders. For example, the calculation result 1110 output from the output line of the first column is J X1 ×R W11 +J X2 ×R W21 +J X3 ×R W31 +...+J XN ×R WN1 The result can be the MAC operation result for the imaginary element j of each input vector and the real element of each weight vector with respect to the first output node. The first sign selector determines the sign of the operation result 1110 to be +, and the first adder selector can transmit the operation result 1110 to the third adder. Also, the operation result 1120 output from the output line of the second column is J X1 ×I W11 +J X2 ×I W21 +J X3 ×I W31 +...+J XN ×I WN1This can be the result of a MAC operation on the imaginary element j of each input vector and the imaginary element i of each weight vector with respect to the first output node. The second sign selector determines the sign of the operation result 1120 to be +, and the second adder selector can transmit the operation result 1120 to the fourth adder. Also, the operation result 1130 output from the output line of the third column is J X1 ×J W11 +J X2 ×J W21 +J X3 ×J W31 +...+J XN ×J WN1 The result can be the MAC operation result for the imaginary element j of each input vector and the imaginary element j of each weight vector with respect to the first output node. The third sign selector determines the sign of the operation result 1130 to -, and the third adder selector can transmit the operation result 1130 to the first adder. Also, the operation result 1140 output from the output line of the fourth column is J X1 ×K W11 +J X2 ×K W21 +J X3 ×K W31 +...+J XN ×K WN1 This could be the result of a MAC operation on the imaginary element j of each input vector and the imaginary element k of each weight vector with respect to the first output node. The fourth sign selector determines the sign of the operation result 1140 to -, and the fourth adder selector can pass the operation result 1140 to the second adder.

[0099] In the above explanation, the calculation results and their transmission behavior performed in the output lines of the first to fourth columns can be similarly applied to the output lines of the remaining columns; therefore, a more detailed explanation is omitted.

[0100] Referring to Figure 12, the operation of the arithmetic unit is shown in the fourth cycle of a quaternion-based MAC operation. In the fourth cycle, the imaginary element k of each four-dimensional input vector can be input to the corresponding input line. The imaginary element k of each four-dimensional input vector input to the input line may be multiplied by a weight element stored in the memory cell. For example, the imaginary element K of the first input vector input to the input line of the first row. X1 This is a weight element R stored in multiple memory cells connected to the first low input line. W11 , I W11 , J W11 , K W11 ,..., R W1M , I W1M , J W1M , K W1M These can also be multiplied together.

[0101] Each of the sign selectors determines the sign of the calculation result output from each output line, and each of the adder selectors can transmit the calculation result output from each output line to one of the adders. For example, the calculation result 1210 output from the output line of the first column is K X1 ×R W11 +K X2 ×R W21 +K X3 ×R W31 +...+K XN ×R WN1 The result can be the MAC operation result for the imaginary element k of each input vector and the real element of each weight vector with respect to the first output node. The first sign selector determines the sign of the operation result 1210 to be +, and the first adder selector can transmit the operation result 1210 to the fourth adder. Also, the operation result 1220 output from the output line of the second column is K X1 ×I W11 +K X2 ×I W21 +K X3 ×I W31 +...+K XN ×I WN1The result can be the MAC operation result for the imaginary element k of each input vector and the imaginary element i of each weight vector with respect to the first output node. The second sign selector determines the sign of the operation result 1220 to -, and the second adder selector can transmit the operation result 1220 to the third adder. Also, the operation result 1230 output from the output line of the third column is K X1 ×J W11 +K X2 ×J W21 +K X3 ×J W31 +...+K XN ×J WN1 The result can be the MAC operation result for the imaginary element k of each input vector and the imaginary element j of each weight vector with respect to the first output node. The third sign selector determines the sign of the operation result 1230 to -, and the third adder selector can transmit the operation result 1230 to the second adder. Also, the operation result 1240 output from the output line of the fourth column is K X1 ×K W11 +K X2 ×K W21 +K X3 ×K W31 +...+K XN ×K WN1 This could be the result of a MAC operation on the imaginary element k of each input vector and the imaginary element k of each weight vector with respect to the first output node. The fourth sign selector determines the sign of the operation result 1240 to be +, and the fourth adder selector can transmit the operation result 1240 to the first adder.

[0102] In the above explanation, the calculation results and their transmission behavior performed in the output lines of the first to fourth columns can be similarly applied to the output lines of the remaining columns; therefore, a more detailed explanation is omitted.

[0103] Multiple adders can accumulate the results of operations transmitted via adder selectors during the first to fourth cycles, with the sign determined via a sign selector, and determine the intermediate operation elements included in the intermediate operation vector. For example, in the first adder, the results of operations accumulated during the first to fourth cycles are the real number element 'R' of the first intermediate operation vector. Y1 Determined to be the result of the operation accumulated between the first and fourth cycles in the second adder, the imaginary element 'I' of the first intermediate operation vector is determined to be the result of the operation in the second adder. Y1 This is determined. Also, in the third adder, the result of the operation accumulated between the first and fourth cycles is the imaginary element 'J' of the first intermediate operation vector. Y1 It is determined that in the fourth adder, the result of the operation accumulated between the first and fourth cycles is the imaginary element 'K' of the first intermediate operation vector. Y1 It will be decided.

[0104] The arithmetic unit can complete the multiplication operation between the quaternion weight vector and the quaternion input vector within four cycles. Based on equation (4), the arithmetic unit can multiply the intermediate operation vector obtained by the multiplication operation within four cycles by the conjugate quaternion weight vector. The conjugate quaternion vector has the same magnitude as the quaternion vector (e.g., the absolute values ​​of the real elements and the three imaginary elements), (where the opposite sign is determined by the multiple sign selectors in Figures 13 to 16), but only the signs of the three imaginary elements are opposite. Therefore, the crossbar array used in the multiplication operation between the quaternion weight vector and the quaternion input vector described above can be used as is. However, the operation of the selectors may differ from one another.

[0105] The real number element 'R' contained in each intermediate operation vector XN and the imaginary element 'I XN ,'J XN ,'K XN These are input to the input line sequentially during the remaining 4 cycles. In a non-restrictive embodiment, the real number element 'RY1 and the imaginary element 'I Y1 ,'J Y1 ,'K Y1 These are the real number elements 'R', respectively. XN and the imaginary element 'I XN ,'J XN ,'K XN Corresponds to the output of the 1st to 4th cycles 'R'. Ym ,'I Ym ,'J Ym , and K Ym ;m={1, 2, 3, ..., M} is the input 'R' for the 5th to 8th cycles. Xn ,'I Xn ,'J Xn , and K Xn ;n is mapped to {1, 2, 3, ..., N}. Therefore, if M=N, even in the case of a crossbar array for complex-based MAC operations, quaternion-based MAC operations can be performed on a single crossbar array without the need for a separate storage device.

[0106] Referring to Figure 13, one embodiment shows the operation of the arithmetic unit in the fifth cycle of a quaternion-based MAC operation. In the fifth cycle, each real element of the four-dimensional intermediate operation vector is input to the corresponding input line and can be multiplied by the weight elements stored in the memory cell. Each of the sign selectors determines the sign of the operation result output from each output line, and each of the adder selectors can transmit the operation result output from each output line to one of the adders. The sign selectors and adder selectors operate as shown by the solid lines in Figure 13, so a more detailed explanation is omitted.

[0107] Referring to Figure 14, the operation of the arithmetic unit in the sixth cycle of the quaternion-based MAC operation is shown. In the sixth cycle, the imaginary element 'i' of each four-dimensional intermediate operation vector is input to the corresponding input line and multiplied by the weight element stored in the memory cell. Each of the sign selectors determines the sign of the operation result output from each output line, and each of the adder selectors can transmit the operation result output from each output line to one of the adders. The sign selectors and adder selectors operate as shown by the solid lines in Figure 14, so a more detailed explanation is omitted.

[0108] Referring to Figure 15, the operation of the arithmetic unit in the seventh cycle of a quaternion-based MAC operation is shown. In the seventh cycle, the imaginary element 'j' of each four-dimensional intermediate operation vector is input to the corresponding input line and multiplied by the weight element stored in the memory cell. Each of the sign selectors determines the sign of the operation result output from each output line, and each of the adder selectors can transmit the operation result output from each output line to one of the adders. The sign selectors and adder selectors operate as shown by the solid lines in Figure 15, so a more detailed explanation is omitted.

[0109] Referring to Figure 16, the operation of the arithmetic unit in the eighth cycle of the quaternion-based MAC operation is shown. In the eighth cycle, the imaginary element 'k' of each four-dimensional intermediate operation vector is input to the corresponding input line and multiplied by the weight element stored in the memory cell. Each of the sign selectors determines the sign of the operation result output from each output line, and each of the adder selectors can transmit the operation result output from each output line to one of the adders. The sign selectors and adder selectors operate as shown by the solid lines in Figure 16, so a more detailed explanation is omitted.

[0110] Multiple adders can accumulate the results of operations transmitted via the adder selector between the 5th and 8th cycles, with the sign determined via the sign selector, to determine the output elements contained in the 4D output vector. For example, in the first adder, the results accumulated between the 5th and 8th cycles are the real element R of the first output vector. Y1 It is determined that in the second adder, the result of the operation accumulated between the 5th and 8th cycles is the imaginary element I of the first output vector. Y1 It may be determined as follows. Also, in the third adder, the result of the operation accumulated between the 5th and 8th cycles is the imaginary element J of the first output vector. Y1 It is determined that in the fourth adder, the accumulated result of the operation between cycles 5 and 8 is the imaginary element K of the first output vector. Y1 It may be decided that way.

[0111] In this way, the quaternion-based MAC calculation result can be efficiently obtained with only 8 cycles of operation of the arithmetic unit. Therefore, it is possible to improve upon general arithmetic units that can obtain the quaternion-based MAC calculation result with a larger number of cycles and / or calculations.

[0112] Figure 17 is a diagram illustrating the structure and operation of the computing device for a quaternion neural network according to another embodiment.

[0113] Referring to Figure 17, a computing device that performs 3D vector-based MAC operations is shown. While 3D vector-based MAC operations are used in various applications such as spatial recognition and image recognition, quaternion vectors are 4-dimensional, so MAC operations can be performed using quaternion vectors with real elements set to "0" and three imaginary elements. In other words, multiple input vectors, multiple weight vectors, and multiple output vectors may be quaternion vectors with real elements set to "0".

[0114] The arithmetic unit that performs MAC operations based on three-dimensional vectors includes a crossbar array, multiple sign selectors, multiple adder selectors, and multiple adders. The crossbar array may include N input lines into which multiple three-dimensional input vectors 1710 are input, 3M output lines intersecting the input lines, and N × 3M memory cells located at the intersections of the input and output lines.

[0115] Each weight element contained in a three-dimensional weight vector can be stored in a reference memory cell that connects a corresponding reference input line and three corresponding reference output lines. For example, weight vector W 1、1 The imaginary element I W1、1 The weight vector W may be stored in a memory cell located at the intersection between the input line of the first row and the output line of the first column. 1、1 The imaginary element J W1、1 The weight vector W may be stored in a memory cell located at the intersection between the input line of the first row and the output line of the second column. 1、1 The imaginary element K W1、1 These may be stored in a memory cell located at the intersection between the input line of the first row and the output line of the third column. In other words, the weight elements contained in each of the multiple three-dimensional weight vectors (e.g., three imaginary elements) may be stored in a reference memory cell connected to a different reference output line.

[0116] On the other hand, while each of the multiple input lines corresponds to one input node, three of the multiple output lines may correspond to one output node. If there are M output nodes, the crossbar array may contain 3M output lines. A memory cell connected to the first of the three output lines 1720 corresponding to one output node may store the imaginary element i of the weight vector connected to that output node, a memory cell connected to the second output line may store the imaginary element j of the weight vector connected to that output node, and a memory cell connected to the third output line may store the imaginary element k of the weight vector connected to that output node. A pair of three output lines can correspond to one output node.

[0117] Unlike an arithmetic unit that performs complex number-based MAC operations, an arithmetic unit that performs three-dimensional vector-based MAC operations may include a plurality of sign selectors and a plurality of adder selectors 1730. The operation result output on each output line can be assigned a + or - sign by the corresponding sign selector and transmitted to one of the plurality of adders by the corresponding adder selector. According to an unrestrictive embodiment, an arithmetic unit for performing three-dimensional vector-based MAC operations may include components of an arithmetic unit for performing complex number-based MAC operations, and thus may perform complex number-based MAC operations and three-dimensional vector-based MAC operations.

[0118] Each of the multiple sign selectors can determine the sign of the calculation result output from the output line by the control signal CONJ_ON (conjunction on). For example, if the control signal is "0" (or low), the sign is determined to be +, and if the control signal is "1" (or high), the sign is determined to be -, although the sign may be determined in the opposite way depending on the embodiment. The control signal can be determined based on whether either a weight vector or a conjugate weight vector is applied to the calculation performed in the relevant cycle, and which vector elements the multiplication performed on the relevant output line is between (for example, whether it is a multiplication between imaginary element i and imaginary element i).

[0119] Each of the multiple adder selectors can determine which of the multiple adders transmits the calculation result output from its corresponding output line via control signals I_ON, J_ON, and K_ON. The control signal I_ON is "1" if the input element input to the relevant cycle is the imaginary element i, and may be "0" otherwise. The control signal J_ON is "1" if the input element input to the relevant cycle is the imaginary element j, and may be "0" otherwise. The control signal K_ON is "1" if the input element input to the relevant cycle is the imaginary element k, and may be "0" otherwise.

[0120] A 3D vector-based MAC operation using a quaternion with real elements of "0" can be expressed, for example, as shown in equation (5) below.

[0121]

number

[0122]

Number

[0123] In the above formula (5), the multiplication operation between the weight vector and the input vector

[0124]

Number

[0125]

Number

[0126] In the first cycle, the imaginary elements i of each three-dimensional input vector 1710 are input to the corresponding input lines. The imaginary element i of each three-dimensional input vector input to the input line is multiplied by the weight element stored in the memory cell. For example, the imaginary element I of the first input vector input to the input line of the first row X1 is the weight element I stored in the plurality of memory cells connected to the input line of the first row W11 , J W11 , K W11 ,..., I W1M , J W1M , K W1M and are multiplied respectively.

[0127] The operation result 1721 output from the output line of the first column is I in formula (6) WNM ·I XNIt corresponds to, and a - sign may be applied by the corresponding sign selector. The operation result 1722 output from the output line of the second column is J in Equation (6) WNM ·I XN It corresponds to, and a - sign may be applied by the corresponding sign selector. The operation result 1723 output from the output line of the third column is K in Equation (6) WNM ·I XN It corresponds to, and a + sign may be applied by the corresponding sign selector.

[0128] In the second cycle, each imaginary element j of the three - dimensional input vector 171 is input to the corresponding input line. Each imaginary element j of the three - dimensional input vector input to the input line may be multiplied by the weight element stored in the memory cell.

[0129] The operation result 1721 output from the output line of the first column is I in Equation (6) WNM ·J XN It corresponds to, and a + sign may be applied by the corresponding sign selector. The operation result 1722 output from the output line of the second column is J in Equation (6) WNM ·J XN It corresponds to, and a - sign may be applied by the corresponding sign selector. The operation result 1723 output from the output line of the third column is K in Equation (6) WNM ·J XN It corresponds to, and a - sign may be applied by the corresponding sign selector.

[0130] In the third cycle, each imaginary element k of the three - dimensional input vector 1710 is input to the corresponding input line. Each imaginary element k of the three - dimensional input vector input to the input line is multiplied by the weight element stored in the memory cell.

[0131] The operation result 1721 output from the output line of the first column is I in Equation (6) WNM ·K XNThis applies, and the - sign may be applied by the corresponding sign selector. The calculation result 1722 output from the output line of the second column is J in formula (6). WNM ·K XN This applies, and the + sign may be applied by the corresponding sign selector. The calculation result 1723 output from the output line of the third column is given by equation (6) K WNM ·K XN This applies, and the negative sign may be applied by the corresponding sign selector.

[0132] The values ​​accumulated in each adder during the first to third cycles may be included in the intermediate operation vector as intermediate operation elements. As can be seen from equation (6), the intermediate operation vector may be a four-dimensional vector containing one real element and three imaginary elements. The real element of the intermediate operation vector is R * YM In the case of the imaginary element i, * YM The imaginary element j is J * YM The imaginary element k is K * YM It can be shown.

[0133] The multiplication operation between an intermediate operation vector and a conjugate weight vector can be represented, for example, by the following number (7).

[0134]

number

[0135] In the fourth cycle, each real element of the 4-dimensional intermediate operation vector 1710 is input to its corresponding input line. Each real element of the 4-dimensional input vector input to the input line is multiplied by the weight element stored in the memory cell.

[0136] The calculation result 1721 output from the output line of the first column is given by formula (7) in R * YM ·I WNM This applies, and the negative sign may be applied by the corresponding sign selector. The calculation result 1722 output from the output line of the second column is given by equation (7) in R * YM ·J WNM This applies, and the negative sign may be applied by the corresponding sign selector. The calculation result 1723 output from the output line of the third column is given by equation (6) in R * YM ·K WNM This applies, and the negative sign may be applied by the corresponding sign selector.

[0137] In the fifth cycle, each imaginary element i of the 4-dimensional intermediate operation vector 1710 is input to its corresponding input line. The imaginary element i of each 4-dimensional input vector input to the input line is multiplied by the weight element stored in the memory cell.

[0138] The calculation result 1722 output from the second column's output line is given by equation (7) I * YM ·J WNM This applies, and the - sign may be applied by the corresponding sign selector. The calculation result 1723 output from the output line of the third column is given by equation (6) I * YM ·K WNM This applies, and the + sign may be applied by the corresponding sign selector.

[0139] In the sixth cycle, the imaginary element j of each of the four-dimensional intermediate operation vectors 1710 is input to the corresponding input line. The imaginary element j of each four-dimensional input vector input to the input line is multiplied by the weight element stored in the memory cell.

[0140] The calculation result 1721 output from the output line of the first column is given by formula (7) J * YM ·I WNM This applies, and the + sign may be applied by the corresponding sign selector. The calculation result 1723 output from the output line of the third column is J in formula (6). * YM ·K WNM This applies, and the negative sign may be applied by the corresponding sign selector.

[0141] In the seventh cycle, the imaginary element k of each of the four-dimensional intermediate operation vectors 1710 is input to the corresponding input line. The imaginary element k of each four-dimensional input vector input to the input line is multiplied by the weight element stored in the memory cell.

[0142] The calculation result 1721 output from the output line of the first column is given by equation (7) K * YM ·I WNM This applies, and the - sign may be applied by the corresponding sign selector. The calculation result 1722 output from the output line of the second column is K in formula (7). * YM ·J WNM This applies, and the + sign may be applied by the corresponding sign selector.

[0143] The values ​​accumulated in each adder between the 4th and 7th cycles may be included in the output vector as output elements. As can be seen from equation (7), the output vector may also be a three-dimensional vector containing three imaginary elements. The imaginary element i of the output vector is I YM The imaginary element j is J YMwhere the imaginary element k is K YM can be shown as

[0144] On the other hand, if complex number-based MAC operations and quaternion-based MAC operations are executed on the arithmetic unit described above, the number of memory cells for storing weight vectors (for example, the total number or a predetermined number) increases by 2 to 4 times compared to RVNN. Therefore, the area efficiency is considered not good. However, the CVNN-based neural network model reduces the number of nodes to 1 / 2 compared to the RVNN-based neural network model, and the number of weights accordingly decreases to 1 / 4. In other words, if the RVNN-based neural network model uses N one-dimensional weights, the CVNN-based neural network model uses N / 4 two-dimensional weights, and even if one two-dimensional weight is stored in two memory cells, the area of the crossbar array can be reduced to 1 / 2 compared to RVNN.

[0145] Also, the QNN-based neural network model reduces the number of nodes to 1 / 4 compared to the RVNN-based neural network model, and the number of weights accordingly decreases to 1 / 16. In other words, if the RVNN-based neural network model uses N one-dimensional weights, the QNN-based neural network model uses N / 16 four-dimensional weights, and even if one four-dimensional weight is stored in four memory cells, the crossbar array area can be reduced to 1 / 4 compared to RVNN.

[0146] FIG. 18 is a diagram showing an operation method of an arithmetic unit according to an embodiment.

[0147] In step S1810, the arithmetic unit inputs input elements included in each of a plurality of multi-dimensional input vectors to a plurality of input lines. The arithmetic unit can sequentially input a plurality of input elements included in each multi-dimensional input vector to the same reference input line during a cycle.

[0148] In step S1820, the arithmetic unit transmits the values ​​output from each of the multiple output lines that intersect the multiple input lines to one of the multiple adders via multiple selectors. The arithmetic unit can also, via a reference selector corresponding to a reference output line, divide the values ​​output from each of the reference output lines into output elements for each of the multidimensional output vectors and transmit them to different reference adders.

[0149] In step S1830, the arithmetic unit accumulates the values ​​transmitted from multiple selectors via multiple adders over a predetermined number of cycles. The number of cycles may be the same as the number of input elements contained in each of the multiple multidimensional input vectors.

[0150] The weight elements contained in each of the multiple multidimensional weight vectors applied to a multidimensional vector neural network are stored in a reference memory cell that connects a corresponding reference input line and two or more corresponding reference output lines. Any one of the weight elements contained in each of the multiple multidimensional weight vectors may be stored in a reference memory cell connected to a different reference output line than the other one.

[0151] Based on the values ​​accumulated in each reference adder corresponding to the reference output line, the output elements contained in each of the multiple multidimensional output vectors can be determined.

[0152] The number of output lines (e.g., the total number) can be determined based on the number of multidimensional weight vectors and the number of weight value elements contained in each multidimensional weight vector.

[0153] Since the aforementioned points apply directly to each step shown in Figure 18, referring to Figures 1 to 17, a more detailed explanation is omitted.

[0154] Figure 19 shows an electronic device according to one embodiment.

[0155] Referring to Figure 19, the electronic device 1900 includes a processor 1910 (e.g., one or more processors), a memory 1920 (e.g., one or more memories), an arithmetic unit 1930, a storage unit 1940, an input device 1950, an output device 1960, and a network interface 1970, which can communicate via a communication bus 1980. For example, the electronic device 1900 can be implemented as at least part of a mobile device such as a mobile phone, smartphone, PDA, netbook, tablet computer, or laptop computer; a wearable device such as a smartwatch, smart band, or smart glasses; a computing device such as a desktop or server; a home appliance such as a television, smart TV, or refrigerator; a security device such as a door lock; or a vehicle such as an autonomous vehicle or smart vehicle. The electronic device 1900 may perform one or more multidimensional vector-based MAC operations as described above via the arithmetic unit 1930.

[0156] The processor 1910 executes functions and instructions for execution within the electronic device 1900. For example, the processor 1910 can process instructions stored in the memory 1920 or the storage device 1940. The processor 1910 may perform one or more operations as described with reference to Figures 1 to 18. The memory 1920 may include a computer-readable storage medium or a computer-readable storage device. The memory 1920 can store instructions for execution by the processor 1910 and related information while the software and / or application is executed by the electronic device 1900.

[0157] The storage device 1940 can store a larger amount of information than the memory 1920 and can store information for a longer period of time. For example, the storage device 1940 may include a magnetic hard disk, an optical disk, flash memory, a floppy disk, or other forms of non-volatile memory known in the art.

[0158] The input device 1950 can receive input from the user via traditional input methods such as a keyboard and mouse, and newer input methods such as touch input, voice input, and image input. For example, the input device 1950 may include a keyboard, mouse, touchscreen, microphone, or any other device that can detect input from the user and transmit the detected input to the electronic device 1900. The output device 1960 can provide the user with the output of the electronic device 1900 via a visual, auditory, or tactile channel. The output device 1960 may include, for example, a display, touchscreen, speaker, vibration generator, or any other device that can provide output to the user. The network interface 1970 can communicate with external devices via a wired or wireless network.

[0159] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will understand that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device includes multiple processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

[0160] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, for interpretation by a processing unit or for providing instructions or data to a processing unit. Software can be distributed across a network of computer systems and stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0161] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.

[0162] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0163] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0164] Therefore, other embodiments, other embodiments, and claims equivalent to those described below also fall within the scope of the claims.

Claims

1. In a computing device for multidimensional vector neural networks, Multiple input lines into which multiple multidimensional input vectors are input, Multiple output lines intersecting the aforementioned multiple input lines, A plurality of memory cells are arranged at the intersections of the plurality of input lines and the plurality of output lines, and store weight elements included in a plurality of multidimensional weight vectors, Multiple selectors that transmit the values ​​output from each of the multiple output lines to one of the multiple adders, Multiple adders that accumulate the values ​​transmitted from the multiple selectors over a predetermined number of cycles, Includes, The weight elements contained in each of the aforementioned multiple multidimensional weight vectors are stored in a reference memory cell that connects a corresponding reference input line and two or more corresponding reference output lines, in an arithmetic unit.

2. The arithmetic device according to claim 1, wherein the input elements contained in each of the plurality of multidimensional input vectors are sequentially input to the same reference input line during the cycle.

3. The arithmetic device according to claim 1, wherein the number of cycles is the same as the number of input elements contained in each of the plurality of multidimensional input vectors.

4. The arithmetic device according to claim 1, wherein the output elements included in each of the multiple multidimensional output vectors are determined based on the values ​​accumulated in each of the reference adders corresponding to the reference output line.

5. The arithmetic device according to claim 4, wherein the reference selector among the plurality of selectors corresponding to the reference output line divides the values ​​output from each of the reference output lines into the output elements of each of the multidimensional output vectors and transmits them to the other reference adders.

6. The number of the plurality of output lines is determined based on the number of multidimensional weight vectors and the number of weight value elements contained in each of the multidimensional weight vectors, according to claim 1.

7. The arithmetic device according to claim 1, wherein one of the weight elements contained in each of the plurality of multidimensional weight vectors is stored in a reference memory cell connected to a different reference output line from the other one.

8. The arithmetic device according to claim 1, wherein the plurality of multidimensional input vectors, the plurality of multidimensional weight vectors, and the plurality of multidimensional output vectors determined based on the values ​​accumulated in each of the plurality of adders are complex vectors including real and imaginary elements.

9. The value output from the first reference output line, which is connected to the memory cell that stores the real number elements, is transmitted via the first reference selector to the second reference adder that accumulates the imaginary number elements while retaining their sign. The arithmetic apparatus according to claim 8, wherein the value output from the second reference output line connected to a memory cell that stores imaginary elements among the reference output lines is transmitted via a second reference selector to a first reference adder that accumulates real elements after its sign is inverted.

10. The arithmetic device according to claim 1, wherein the plurality of multidimensional input vectors, the plurality of multidimensional weight vectors, and the plurality of multidimensional output vectors determined based on the values ​​accumulated in each of the plurality of adders are quaternion vectors containing a plurality of imaginary elements or quaternion vectors containing a real element and a plurality of imaginary elements.

11. The arithmetic apparatus according to claim 10, further comprising a plurality of second selectors that determine the sign of a value output from each of the plurality of output lines based on the type of multidimensional weight vector applied to an operation to be performed in each cycle, the type of weight elements stored in the plurality of memory cells, and the type of input elements input to the plurality of input lines, and transmit the sign-determined value to a corresponding selector.

12. The arithmetic device according to claim 10, wherein the elements contained in each of the partial calculation result vectors accumulated by the plurality of adders during a portion of the cycle are sequentially input to the plurality of input lines during the remainder of the cycle.

13. The predetermined number of the cycles is two, in response to the case where the multiple multidimensional weight vectors are complex vectors. The arithmetic device according to any one of claims 1 to 12, wherein the number of multidimensional weight vectors is seven or eight in response to the case where the plurality of multidimensional weight vectors are quaternion vectors.

14. In the operation method of a computing device for a multidimensional vector neural network, The steps include inputting the input elements contained in each of the multiple multidimensional input vectors into multiple input lines, The steps include transmitting the values ​​output from each of the multiple output lines that intersect the multiple input lines via multiple selectors to one of the multiple adders, A step of accumulating values ​​transmitted from the multiple selectors via multiple adders over a predetermined number of cycles, Includes, A method of operation for an arithmetic unit, wherein the weight elements contained in each of the multiple multidimensional weight vectors applied to the multidimensional vector neural network are stored in a reference memory cell that connects a corresponding reference input line and two or more corresponding reference output lines.

15. The method of operating the arithmetic device according to claim 14, wherein the input step involves sequentially inputting a plurality of input elements contained in each multidimensional input vector to the same reference input line during the cycle.

16. The method for operating the arithmetic device according to claim 14, wherein the number of cycles is the same as the number of input elements contained in each of the plurality of multidimensional input vectors.

17. The method of operating the arithmetic device according to claim 14, wherein the output elements included in each of the multiple multidimensional output vectors are determined based on the values ​​accumulated in each of the reference adders corresponding to the reference output line.

18. The method of operating the arithmetic device according to claim 17, wherein the transmission step involves transmitting the values ​​output from each of the reference output lines to different reference adders via a reference selector among the plurality of selectors that corresponds to the reference output line, by dividing them into output elements of the multidimensional output vector.

19. The method for operating the arithmetic device according to claim 14, wherein the number of the plurality of output lines is determined based on the number of multidimensional weight vectors and the number of weight value elements contained in each of the multidimensional weight vectors.

20. The method of operating the arithmetic device according to any one of claims 14 to 19, wherein one of the weight elements contained in each of the plurality of multidimensional weight vectors is stored in a reference memory cell connected to a different reference output line from the other one.

Images