Apparatus and method for performing matrix multiplication operation of neural network

Abstract

A neural network device for performing matrix multiplication operations includes: a memory having at least one program stored therein; and a processor that performs one or more operations by executing the at least one program. The processor can determine whether to partition initial weights in one of the column and row directions based on whether reshaping and transpose operations are performed before or after the matrix multiplication operations, and generates partition weights by dividing the initial weights by a head count in the determined direction. Furthermore, the processor can generate intermediate feature maps by performing matrix multiplication operations between an input feature map and the partition weights, and generate a final feature map based on the intermediate feature maps.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2019-0161676, filed on December 6, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0003] The following description relates to apparatus and methods for performing matrix multiplication operations of neural networks. Background Technology

[0004] A neural network is a computational architecture used to model the biological brain. With the development of neural network technology, neural networks are used in various types of electronic systems to analyze input data and extract useful information.

[0005] Research has been actively conducted on hardware accelerators to enable efficient use of deep neural networks (DNNs) with low power. Neural network processing devices require extensive computation on complex input data.

[0006] In particular, in the implementation of devices with low power and low performance, there is a need for a technique that can effectively handle the operations on neural networks in order to extract desired information by analyzing large amounts of input data in real time using neural networks. Summary of the Invention

[0007] This summary is provided to describe in simplified form the selection of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help define the scope of the claimed subject matter.

[0008] Apparatus and methods for performing matrix multiplication operations of a neural network. A computer-readable recording medium storing a program that executes these methods when executed by a computer.

[0009] In one general aspect, a neural network device includes: a memory having at least one program stored therein; and a processor that performs one or more operations by executing the at least one program, wherein the processor retrieves an input feature map and initial weights from the memory; determines whether to partition the initial weights in the column direction or the row direction based on whether the reshaping and transpose operations are performed before or after matrix multiplication; generates partition weights by dividing the initial weights by a head count in the determined column direction or row direction; generates an intermediate feature map by performing a matrix multiplication operation between the input feature map and the partition weights; and generates a final feature map based on the intermediate feature map.

[0010] When the reshaping and transpose operations are performed after the matrix multiplication operation, the processor can generate partition weights by dividing the initial weights by the head count in the column direction of the initial weights, and generate the final feature map by concatenating the intermediate feature maps.

[0011] When the reshaping and transpose operations are performed before the matrix multiplication operations, the processor can generate partition weights by dividing the initial weights by the head count in the row direction of the initial weights, and generate the final feature map by summing the intermediate feature maps element by element.

[0012] The matrix multiplication operation between the input feature map and multiple partition weights can be either a one-dimensional convolution operation or a two-dimensional convolution operation.

[0013] The processor may include a weight partitioner, which may divide the initial weight by the head count in one of the column and row directions.

[0014] In another general aspect, one approach includes: retrieving an input feature map and initial weights from memory; determining whether to partition the initial weights in one of the column or row directions based on whether the reshaping and transpose operations are performed before or after matrix multiplication; generating partition weights by dividing the initial weights by a head count in the determined column or row direction; generating an intermediate feature map by performing matrix multiplication between the input feature map and the partition weights; and generating a final feature map based on the intermediate feature map.

[0015] In another general aspect, one approach includes: receiving an initial feature map and initial weights; dividing the initial weights into partition weights; performing matrix multiplication between the input feature map and each partition weight to generate an intermediate feature map; and manipulating the intermediate feature map to generate an output feature map.

[0016] The method may include: determining whether the input feature map has been reshaped and transposed.

[0017] If the input feature map has already undergone reshaping and transpose operations, the initial weights can be divided into partition weights based on the head count of the initial weights in the row direction.

[0018] The method may include generating an output feature map by summing the intermediate feature maps element by element.

[0019] Without performing reshaping and transpose operations on the input feature map, the initial weights can be divided into partition weights based on the head count of the initial weights in the column direction.

[0020] The method may include generating an output feature map by concatenating intermediate feature maps.

[0021] In another general aspect, a computer-readable recording medium stores a program that, when executed by a computer, performs one or more of the methods.

[0022] Other features and aspects will become clear from the following detailed description, drawings and claims. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the operations performed in a deep neural network (DNN) according to an example.

[0024] Figure 2 It is a schematic diagram used to describe the relationship between the input feature map and the output feature map in a neural network based on the example.

[0025] Figure 3A and Figure 3B This is a diagram used to illustrate the reshaping and transpose operations based on the example.

[0026] Figure 4A This is a schematic diagram used to describe self-attention based on the example.

[0027] Figure 4B This is a schematic diagram used to describe multi-head self-attention based on the example.

[0028] Figure 5A This is a diagram used to illustrate the number of memory accesses when performing reshape and transpose operations, based on an example.

[0029] Figure 5B This is a diagram used to illustrate the number of memory accesses without performing reshaping and transpose operations, based on an example.

[0030] Figure 6 This is a schematic diagram illustrating the process of performing matrix multiplication using a weighted partitioner, as shown in the example.

[0031] Figure 7 This is a schematic diagram showing the result of performing matrix multiplication when the weights are divided in the column direction, according to the example.

[0032] Figure 8 This is a schematic diagram showing the result of performing matrix multiplication when the weights are divided in the row direction, according to the example.

[0033] Figure 9 This is a block diagram illustrating the hardware configuration of a neural network device according to an example.

[0034] Figure 10 This is a flowchart illustrating a method for performing matrix multiplication in a neural network device, based on an example.

[0035] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0036] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, after understanding the disclosure of this application, various changes, modifications, and equivalents to the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders of operations set forth herein, but can be obviously changed after understanding the disclosure of this application, except for operations that must be performed in a certain order. Furthermore, for clarity and conciseness, descriptions of known features may be omitted after understanding the disclosure of this application.

[0037] The features described herein may be implemented in various forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become apparent upon understanding the disclosure of this application.

[0038] Throughout this specification, when a component is described as "connected to" or "coupled to" another component, it may be directly "connected to" or "coupled to" that other component, or there may be one or more other components in between. Conversely, when an element is described as "directly connected to" or "directly coupled to" another element, there may be no other elements in between. Similarly, similar expressions such as "between" and "directly between," and "adjacent to" and "immediately adjacent to" should be interpreted in the same way. As used herein, the term "and / or" includes any combination of any one and any two or more of the listed items.

[0039] While terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Therefore, without departing from the teachings of the examples described herein, the first component, assembly, region, layer, or part mentioned in the examples may also be referred to as the second component, assembly, region, layer, or part.

[0040] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the articles “a,” “an,” and “the” are also intended to include the plural form. The terms “comprising,” “including,” and “having” specify the presence of the described features, numbers, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, components, elements, and / or combinations thereof.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) as used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of this application. Terms such as those defined in common dictionaries should be interpreted as having the same meaning as in the context of the relevant art and the disclosure of this application, and should not be interpreted as having an ideal or overly formal meaning, unless expressly defined herein. In this document, the use of the term “may” to describe an example or embodiment (e.g., what an example or embodiment may include or implement) implies the existence of at least one example or embodiment that includes or implements such a feature, and not that all examples are limited thereto.

[0042] Examples can be represented by functional block components and various processing operations. Some or all of these functional blocks can be implemented by any number of hardware and / or software components performing a specific function. For example, a functional block can be implemented by one or more microprocessors, or by circuit components for a specific function. Furthermore, functional blocks can be implemented using various programming or scripting languages, for example. Functional blocks can be implemented by algorithms that execute in one or more processors. Terms such as “mechanism,” “element,” “unit,” and “configuration” can be used broadly and are not limited to mechanical and physical configurations.

[0043] Furthermore, the connecting members or lines between elements shown in the accompanying drawings are merely examples of functional and / or physical or logical connections. In actual devices, connections between elements can be represented by a variety of alternative or additional functional, physical, or logical connections.

[0044] The example will be described in detail below with reference to the accompanying drawings.

[0045] Figure 1 This is a schematic diagram illustrating the operations performed in a deep neural network (DNN) according to an example.

[0046] refer to Figure 1The DNN 100 can have a structure including an input layer, a hidden layer and an output layer, and can perform operations based on the received input data (e.g., I1 and I2) and generate output data (e.g., O1 and O2) based on the results of the operations.

[0047] For example, such as Figure 1 As shown, DNN 100 may include an input layer (layer 1), two hidden layers (layers 2 and 3), and an output layer (layer 4). Because DNN 100 may include more layers capable of processing effective information, it can handle more complex datasets compared to neural networks with a single layer. Although DNN 100 is shown as having four layers, this is just an example, and DNN 100 may include fewer or more layers or fewer or more channels. That is, DNN 100 may include... Figure 1 The layers shown have various structures with different configurations.

[0048] Each layer included in a DNN 100 can include multiple channels. Channels can correspond to multiple artificial nodes referred to as neurons, processing elements (PEs), units, or similar terms. For example, as... Figure 1 As shown, layer 1 may include two channels (nodes), and each of layers 2 and 3 may include three channels. However, this is just an example, and each layer included in DNN100 may include a variety of numbers of channels (nodes).

[0049] The channels included in each layer of the neural DNN 100 can be connected to each other to process data. For example, one channel can receive and manipulate data from other channels and output the results of the manipulation to other channels.

[0050] The input and output of each channel can be referred to as input activation and output activation, respectively. That is, activation can be the output of a channel and parameters corresponding to the inputs of channels included in the next layer. Furthermore, each channel can determine its own activation based on the activations and weights received from channels included in the previous layer. Weights can be parameters used to calculate the output activation in each channel, or they can be values ​​assigned to the connection relationships between channels.

[0051] Each channel can be processed by a computational unit or PE that receives input and activates the output, and the input-output of each channel can be mapped. For example, when σ is the activation function, It represents the weights from the k-th channel included in layer (i-1) to the j-th channel included in layer i. It is the bias of the j-th channel included in the i-th layer, and When the activation of the k-th channel in the (i-1)-th layer is known, the activation of the j-th channel in the i-th layer can be calculated using Equation 1 below.

[0052] [Equation 1]

[0053]

[0054] like Figure 1 As shown, the activation of the first channel (CH1) in the second layer (layer 2) can be represented as Moreover, according to Equation 1, It can have The value of . However, Equation 1 is merely an example used to describe the activations and weights used in DNN100 for processing data, and the example is not limited thereto. Activation can be a value obtained by applying the activation function to the sum of activations received from the previous layer via a rectified linear unit (ReLU).

[0055] In the example, DNN100 can determine the factors that define the relationship between descriptors and attributes by learning based on descriptors and attribute values. That is, in the first to fourth layers of DNN 100, the descriptor can be the first layer as the input layer, the attribute value can be the fourth layer as the output layer, and the factor can be at least one hidden layer (the second and / or the third layer).

[0056] DNN 100 can perform operations by using descriptors as input data in the input layer, and can generate attribute values ​​as output data based on the results of the operations.

[0057] Figure 2 It is a schematic diagram used to describe the relationship between the input feature map and the output feature map in a neural network based on the example.

[0058] refer to Figure 2 In layer 200 of the neural network, the first feature map FM1 can correspond to the input feature map, and the second feature map FM2 can correspond to the output feature map. A feature map can refer to a dataset representing various features of the input data. Feature maps FM1 and FM2 can have elements of a two-dimensional matrix or elements of a three-dimensional matrix, and pixel values ​​can be defined in each element. Feature maps FM1 and FM2 can have a width W (or, columns), a height H (or, rows), and a depth D. In this case, the depth D can correspond to the number of channels.

[0059] A convolution operation can be performed on a first feature map FM1 and weights WM, resulting in a second feature map FM2. The weights can be weights defined in each element and can be used to filter features of the first feature map FM1 by performing a convolution operation with it. The weights can be convolved with a window (or block) of the first feature map FM1 while moving the first feature map FM1 in a sliding window manner. During each movement, the weights included in each element can be multiplied and summed with each pixel value of the overlapping window in the first feature map FM1. When the first feature map FM1 and weights are convolved together, one channel of the second feature map FM2 can be generated. Although in Figure 2 The diagram shows one weight, but multiple weights can be convolved with the first feature map FM1 to generate a second feature map FM2 with multiple channels.

[0060] Furthermore, the second feature map FM2 can correspond to the input feature map of the next layer. For example, the second feature map FM2 can be the input feature map of a pooling (or subsampling) layer.

[0061] exist Figure 2 For ease of description, only a schematic architecture of the neural network is shown in this illustration. However, those skilled in the art will understand that, unlike the illustration, a neural network can be implemented with more or fewer layers, feature maps, weights, etc., and its size can be modified in various ways.

[0062] Figure 3A and Figure 3B This is an example diagram used to describe the reshaping and transpose operations based on the example.

[0063] Figure 3A The reshaping operation process is illustrated. A reshaping operation can be a process of changing the structure of specific data. When reshaping specific data, its data structure can change, but its data order may remain unchanged.

[0064] For example, suppose one-dimensional data 311 [1, 2, 3, 4, 5, 6, 7, 8] contains eight elements. When one-dimensional data 311 is reshaped into a (2, 4) data structure, the structure of one-dimensional data 311 can be changed into two-dimensional data 312 [[1, 2, 3, 4], [5, 6, 7, 8]].

[0065] Alternatively, when one-dimensional data 311 is reshaped into a (2, 2, 2) data structure, the structure of one-dimensional data 311 can be changed into three-dimensional data 313 [[[1, 2], [3, 4]], [[5, 6], [7, 8]]].

[0066] Figure 3BThe transpose operation is illustrated. A transpose operation is a process of obtaining a new matrix by exchanging row and column values. When transposing specific data, depending on the initial structure of the data, you can either change only the order of the data, or you can change both the order and structure of the data.

[0067] For example, when the two-dimensional data 321[[1,2],[3,4]] is transposed, it can be changed into the two-dimensional data 322[[1,3],[2,4]]. In this case, the order of the data can be changed, but the data structure can remain the same as before.

[0068] Alternatively, when the two-dimensional data 323[[1,2],[3,4],[5,6]] is transposed, it can be transformed into two-dimensional data 324[[1,3,5],[2,4,6]]. In this case, not only can its data order be changed, but its data structure can also be changed. That is, the two-dimensional data 323 with a (3,2) structure can be transformed into the two-dimensional data 324 with a (2,3) structure.

[0069] Figure 4A This is an example graph used to describe self-attention based on examples.

[0070] Self-attention can be used to measure the relationships between words in a sentence. In this case, a relationship value for each word with other words can be calculated. This value can be called an attention score. Attention scores between highly related words may be high. An attention score table can be called an attention graph.

[0071] In the transformer model, attention scores are obtained by performing dot product operations between word vectors. After obtaining the attention scores of a specific word relative to other words, a softmax function can be applied to the attention map that collects these scores. The result is that the attention map presents the relevance of a specific word to another word in the form of probabilities.

[0072] The probability values ​​of each existing word vector and attention map can be called a weighted sum. A weighted sum is an operation that multiplies each probability value by each word vector and then adds the results. The weighted sum can be used as the vector value for a specific word.

[0073] Figure 4B This is an example graph used to describe multi-head self-attention based on the example.

[0074] Multihead self-attention is a method that identifies attention about various feature values ​​by generating multiple attention maps. It can be used in transformer models, automatic speech recognition (ASR) models, and more, but is not limited to these.

[0075] Multi-head self-attention can take the form of overlapping, scaled-up dot-product attention structures. The input to dot-product attention can include a query, a key, and / or a value. For example, in the case of finding the meaning of a specific word in an English dictionary, that specific word can correspond to the query, the word registered in the dictionary can correspond to the key, and the meaning of the key can correspond to the value.

[0076] To obtain multi-head self-attention, each of the feature values ​​of value V, key K, and query Q can be divided by the head count h, and then concatenated through the first linear layer 410 and dot product attention. Finally, when the concatenated value is output through the second linear layer 420, multi-head self-attention is obtained.

[0077] In the example, matrix multiplication, reshaping, and transpose operations can be performed in the first linear layer 410 and the second linear layer 420. Specifically, in the first linear layer 410, reshaping and transpose operations can be performed after matrix multiplication, while in the second linear layer 420, reshaping and transpose operations can be performed before matrix multiplication.

[0078] The following describes a method for obtaining multi-head self-attention by using matrix multiplication without performing reshaping and transpose operations in the first linear layer 410 and the second linear layer 420.

[0079] Figure 5A This is a diagram used to illustrate the number of memory accesses when performing reshape and transpose operations, based on an example.

[0080] Figure 5A This can correspond to a part of the process of obtaining multi-head self-attention, and can correspond to in Figure 4B The process is executed in the first linear layer 410.

[0081] refer to Figure 5A The input feature map can have a (B, L, H) structure. In the (B, L, H) structure, B represents the batch size, L represents the rows of the input feature map, and H represents the columns of the input feature map.

[0082] Matrix multiplication can be performed between an input feature map of structure (B, L, H) and weights of structure (H, H). The weights can be a structure with H rows and H columns. As a result of the matrix multiplication, a first intermediate feature map of structure (B, L, H) can be generated. A reshaping operation can be performed on the first intermediate feature map to generate a second intermediate feature map of structure (B, L, S, H / S). A transpose operation can be performed on the second intermediate feature map to generate an output feature map of structure (B, S, L, H / S).

[0083] exist Figure 5A In the process of generating an output feature map of a (B, S, L, H / S) structure from an input feature map of a (B, L, H) structure, a total of four read / write operations may be required on the memory (e.g., SRAM or DRAM).

[0084] Figure 5B This is a diagram used to illustrate the number of memory accesses without performing reshaping and transpose operations, based on an example.

[0085] Figure 5B It can also correspond to a part of the process of obtaining multi-head self-attention, and can correspond to in Figure 4B The process is executed in the first linear layer 410.

[0086] refer to Figure 5B The input feature map can have a (B, L, H) structure. In the (B, L, H) structure, B represents the batch size, L represents the rows of the input feature map, and H represents the columns of the input feature map.

[0087] exist Figure 5B Instead of performing matrix multiplication between the input feature map of the (B, L, H) structure and the initial weights of the (H, H) structure, the initial weights of the (H, H) structure can be divided into S partition weights with the (H, H / S) structure.

[0088] Matrix multiplication can be performed between the input feature map of the (B, L, H) structure and the S partitioning weights of the (H, H / S) structure. As a result of the matrix multiplication, S intermediate feature maps of the (B, L, H / S) structure can be generated. These S intermediate feature maps of the (B, L, H / S) structure can be concatenated to finally generate the output feature map of the (B, S, L, H / S) structure.

[0089] exist Figure 5B In the process of generating the output feature map of the (B, L, H) structure from the input feature map of the (B, L, H) structure, a total of two read / write operations may be required on the memory (e.g., SRAM or DRAM).

[0090] Figure 5A The structures of the input and output feature maps can be similar to... Figure 5B The input and output feature maps have the same structure. As for... Figure 5A and Figure 5B The comparison between memory access counts, due to the comparison between memory access counts, is performed on the memory access counts. Figure 5B The transpose operation was not performed, so it is the same as... Figure 5A compared to, Figure 5B The memory access count can be reduced. Figure 5B In this configuration, the memory access count can be reduced by half compared to 5A, and correspondingly, the read / write data can be reduced by 2*BLH (=B*L*S*H / S). Furthermore, as the batch size increases, compared to... Figure 5A In comparison, based on Figure 5B The method of reducing memory access counts may be more effective.

[0091] Figure 6 This is an example diagram used to describe the process of performing matrix multiplication by using a weighted partitioner, based on an example.

[0092] refer to Figure 6 The initial weights and the input feature map can be stored in memory 610. In the following text, it is assumed that the initial weights are a (512, 512) structure with 512 rows and 512 columns, and the input feature map is a (T, 512) structure with T rows (T is a natural number) and 512 columns.

[0093] The initial weights of the (512, 512) structure stored in memory 610 can be input to the weight divider 620. The weight divider 620 can divide the initial weights into multiple partition weights. The weight divider 620 can partition the initial weights in either the column direction or the row direction.

[0094] Specifically, the weight divider 620 can determine whether to divide the initial weights of the (512, 512) structure in either the column or row direction, depending on whether the reshaping and transpose operations are performed after or before the matrix multiplication operation.

[0095] For example, when the reshaping and transpose operations are performed after the matrix multiplication operation, the weight partitioner 620 can partition the initial weights of the (512, 512) structure in the column direction to generate multiple partition weights. Alternatively, when the reshaping and transpose operations are performed before the matrix multiplication operation, the weight partitioner 620 can partition the initial weights of the (512, 512) structure in the row direction to generate multiple partition weights.

[0096] Furthermore, the weight divider 620 can divide the initial weights of the (512, 512) structure by the head count in the determined direction. For example, when the head count is 16, 16 partition weights with a (512, 32) structure (column direction partitioning) or 16 partition weights with a (32, 512) structure (row direction partitioning) can be generated depending on the partitioning direction of the initial weights.

[0097] Each partition weight generated by the weight partitioner 620 can be sent to the PE. The PE can perform matrix multiplication on the partition weights received from the weight partitioner 620 and the input feature map received from the memory 610. As a result of the matrix multiplication, an intermediate feature map can be output from the PE.

[0098] For example, when the weight partitioner 620 partitions the initial weights of the (512, 512) structure in the column direction, the weight partitioner 620 can send the first partition weights to the sixteenth partition weights with the (512, 32) structure to the first PE to the sixteenth PE 6301, 6302, ..., 6316, respectively. Moreover, the first PE 6301 to the sixteenth PE 6316 can receive the input feature map from the memory 610.

[0099] The first PE 6301 can perform matrix multiplication between the first partition weight of the (512, 32) structure and the input feature map of the (T, 512) structure, and output the first intermediate feature map of the (T, 32) structure. In the same manner, the second PE 6302 to the sixteenth PE 6316 can output the second to sixteenth intermediate feature maps of the (T, 32) structure, respectively.

[0100] although Figure 6 The illustration shows the case where the weight divider 620 divides the initial weights in the column direction, but the above description can also be applied to the case where the weight divider 620 divides the initial weights in the row direction.

[0101] When initial weights are assigned in the column direction, the first to sixteenth intermediate feature maps of the (T, 32) structure output from the first PE 6301 to the sixteenth PE 6316 can be concatenated to generate the final feature map. The final feature map can have a (16, T, 32) structure. In the example, the first to sixteenth intermediate feature maps can be stored sequentially at consecutive locations in memory 610, thus allowing for concatenation.

[0102] When initial weights are assigned in the row direction, the final feature map can be generated by summing the first to sixteenth intermediate feature maps output from the first PE6301 to the sixteenth PE6316 element-wise. Element-wise summation can be performed in the PE.

[0103] although Figure 6 The diagram shows the assumption that 16 head counts are used, resulting in 16 PEs. However, depending on the head count, the number of PEs used can be fewer or more. Furthermore, when performing matrix multiplication on an input feature map, at least one PE can be used several times.

[0104] Figure 7 This is an example diagram showing the result of performing matrix multiplication when the weights are divided in the column direction, based on the example.

[0105] Figure 7 This is a schematic diagram illustrating the situation where reshaping and transpose operations are performed after matrix multiplication.

[0106] refer to Figure 7 The input feature map 710 can have a (1, 4, 6) structure. That is, the input feature map 710 can have a batch size of 1, 4 rows, and 6 columns. The initial weights 720 can have a (1, 6, 6) structure. That is, the initial weights 720 can have a batch size of 1, 6 rows, and 6 columns.

[0107] Matrix multiplication can be performed on the input feature map 710 with a (1, 4, 6) structure and the initial weights 720 with a (1, 6, 6) structure. Then, reshaping and transposing operations can be performed on the result. The result is that a final feature map 740 with a (1, 3, 4, 2) structure can be generated. That is, the output feature map 740 can have a batch size of 1, 3 channels, 4 rows, and 2 columns.

[0108] Furthermore, initial weights of 720 can be assigned along the column direction to generate multiple partition weights. In this case, the number of partition weights generated can be determined based on the head count. Figure 7 An example of the following situation is: the head count is 3, and the initial weight 720 of the (1, 6, 6) structure can be divided into the first partition weight 731, the second partition weight 732, and the third partition weight 733 of the (1, 6, 2) structure.

[0109] Matrix multiplication can be performed between the input feature map 710 of the (1, 4, 6) structure and the first partition weights 731 to the third partition weights 733 of the (1, 6, 2) structure, and the results can be concatenated to generate the final feature map 740 of the (1, 3, 4, 2) structure.

[0110] In the example, when the reshaping and transpose operations are performed after the matrix multiplication operation, the initial weights 720 can be divided in the column direction to generate the first division weights 731 to the third division weights 733, and the matrix multiplication operation can be performed between the input feature map 710 and the first division weights 731 to the third division weights 733 to generate the same final feature map 740 as in the case where the reshaping and transpose operations have been performed.

[0111] Figure 8 This is an example diagram showing the result of performing matrix multiplication when the weights are divided in the row direction, according to the example.

[0112] refer to Figure 8 The input feature map 810 can have a (1, 3, 4, 2) structure. That is, the input feature map 810 can have a batch size of 1, 3 channels, 4 rows, and 2 columns. When the reshaping and transpose operations are performed on the input feature map 810, the input feature map 811 can have a (1, 4, 6) structure. That is, the input feature map 811 can have a batch size of 1, 4 rows, and 6 columns. The initial weights 820 can have a (1, 6, 6) structure. That is, the initial weights 820 can have a batch size of 1, 6 rows, and 6 columns.

[0113] When a matrix multiplication operation is performed on the input feature map 811 of the (1, 4, 6) structure and the initial weights 820 of the (1, 6, 6) structure, the final feature map 850 of the (1, 4, 6) structure can be generated as a result. That is, the final feature map 850 can have a batch size of 1, 4 rows and 6 columns.

[0114] Furthermore, initial weights of 820 can be assigned along the column direction to generate multiple partition weights. In this case, the number of partition weights generated can be determined based on the head count. Figure 8 An example of the following situation is: the head count is 3, and the initial weight 820 of the (1, 6, 6) structure can be divided into the first partition weight 831, the second partition weight 832, and the third partition weight 833 of the (1, 2, 6) structure.

[0115] Matrix multiplication can be performed between the input feature map 810 of the (1, 3, 4, 2) structure and the first to third partition weights 831 of the (1, 2, 6) structure, and as a result, a first intermediate feature map 841, a second intermediate feature map 842, and a third intermediate feature map 843 of the (1, 4, 6) structure can be generated. The final feature map 850 of the (1, 4, 6) structure can be generated by summing the first to third intermediate feature maps 841 and 843 element by element.

[0116] In the example, when the reshaping and transpose operations are performed before the matrix multiplication operation, the initial weights 820 can be divided in the row direction to generate the first division weights 831 to the third division weights 833, and the matrix multiplication operation can be performed between the input feature map 810 and the first division weights 831 to the third division weights 833 to generate the same final feature map 850 as in the case where the reshaping and transpose operations have been performed.

[0117] Figure 9 This is a block diagram illustrating the hardware configuration of a neural network device according to an example.

[0118] The neural network device 900 can be implemented by various types of devices, such as personal computers (PCs), server devices, mobile devices, or embedded devices. As a specific example, the neural network device 900 can correspond to, but is not limited to, smartphones, tablets, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous vehicles, robots, medical devices, etc., that perform speech recognition, image recognition, image classification, etc., using neural networks. Furthermore, the neural network device 900 can correspond to a dedicated hardware (HW) accelerator installed on the aforementioned devices, and the neural network device 900 can be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine (which is a dedicated module for driving neural networks), but is not limited to these.

[0119] refer to Figure 9 The neural network device 900 may include a processor 910 and a memory 920. Figure 9 In the neural network device 900 shown, only components relevant to each example are illustrated. Therefore, it will be clear to those skilled in the art that, in addition to... Figure 9 In addition to the components shown, the neural network device 900 may also include other general-purpose components.

[0120] The processor 910 can control the overall functions used to execute the neural network device 900. For example, the processor 910 can typically control the neural network device 900 by executing a program stored in the memory 920 within the neural network device 900. The processor 910 can be implemented by, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc., disposed within the neural network device 900.

[0121] The memory 920 can be hardware used to store various data processed in the neural network device 900. For example, the memory 920 can store data that has been processed or will be processed in the neural network device 900. Furthermore, the memory 920 can store applications, drivers, etc., that will be driven by the neural network device 900. The memory 920 can include random access memory (RAM) (such as dynamic random access memory (DRAM) or static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray, other optical disc storage, hard disk drive (HDD), solid-state drive (SSD), or flash memory.

[0122] Processor 910 can read / write neural network data, such as image data, feature map data, weight data, etc., from / to memory 920, and execute the neural network using the read / written data. When executing the neural network, processor 910 can repeatedly perform convolution operations between the input feature maps and weights to generate data about the output feature maps. In this case, the amount of convolution operation can be determined depending on various factors (e.g., the number of channels in the input feature map, the number of channels in the weights, the size of the input feature map, the size of the weights, and the precision of their values). Figure 1 Unlike the DNN 100 shown, the actual neural network driven in the neural network device 900 can be implemented with a more complex architecture. Therefore, the processor 910 may perform a very large number of operations (operation counts), ranging from hundreds of millions to tens of billions, and the frequency with which the processor 910 accesses memory 920 to perform operations may also surge. Due to this computational load, neural networks may not be able to be processed smoothly in mobile devices with relatively low processing performance (such as smartphones, tablets, wearable devices, embedded devices, etc.).

[0123] Processor 910 can perform matrix multiplication, reshaping, and transpose operations. In the example, processor 910 can perform matrix multiplication, reshaping, and transpose operations to obtain multi-head self-attention. During the process of obtaining multi-head self-attention, the reshaping and transpose operations can be performed after or before the matrix multiplication operations.

[0124] Processor 910 can even execute a portion of the process for obtaining multi-head self-attention without performing reshaping and transpose operations. Processor 910 can determine whether to partition initial weights in either the column or row direction based on whether the reshaping and transpose operations are performed before or after matrix multiplication, and generate multiple partition weights by dividing the initial weights by the head count in the determined direction. Processor 910 can generate multiple intermediate feature maps by performing matrix multiplication between the input feature map and the multiple partition weights, and generate a final feature map based on the multiple intermediate feature maps. In this manner, even without performing reshaping and transpose operations, processor 910 can reduce the number of memory accesses to 920 by executing a portion of the process for obtaining multi-head self-attention.

[0125] Figure 10 This is a flowchart illustrating a method for performing matrix multiplication in a neural network device, based on an example. Because... Figure 10 The method for performing matrix multiplication in a neural network device shown is related to the example described above with reference to the accompanying drawings. Therefore, even if omitted below, the description given above with reference to the accompanying drawings can be applied to [the example described above with reference to the accompanying drawings]. Figure 10 The method.

[0126] refer to Figure 10 In operation 1010, the neural network device can obtain the input feature map and initial weights from the memory.

[0127] In operation 1020, the neural network device can determine whether to assign initial weights in either the column or row direction based on whether the reshaping and transpose operations are performed after or before the matrix multiplication operation.

[0128] When the reshaping and transpose operations are performed after matrix multiplication, the neural network device can partition the initial weights along the column direction of the initial weights. Alternatively, when the reshaping and transpose operations are performed before matrix multiplication, the neural network device can partition the initial weights along the row direction of the initial weights.

[0129] In operation 1030, the neural network device can generate multiple partition weights by dividing the initial weights by the head count in the direction determined in operation 1020.

[0130] For example, when the initial weight structure is (512, 512) and the head count is 16, 16 partition weights with a (512, 32) structure (column direction partition) or a (32, 512) structure (row direction partition) can be generated depending on the partition direction of the initial weight.

[0131] In operation 1040, the neural network device can generate multiple intermediate feature maps by performing matrix multiplication between the input feature map and multiple partition weights.

[0132] The matrix multiplication operation between the input feature map and multiple partition weights can be either a one-dimensional convolution operation or a two-dimensional convolution operation.

[0133] In operation 1050, the neural network device can generate the final feature map based on multiple intermediate feature maps.

[0134] When initial weights are assigned in the column direction during operation 1020, the neural network device can generate the final feature map by concatenating multiple intermediate feature maps.

[0135] When initial weights are assigned in the row direction during operation 1020, the neural network device can generate the final feature map by summing the elements of multiple intermediate feature maps.

[0136] The examples can also be implemented in the form of computer-readable recording media, which include computer-executable instructions, such as computer-executable program modules. Computer-readable recording media can be any available medium accessible to a computer and can include all volatile or non-volatile media, as well as removable or non-removable media. Furthermore, computer-readable recording media can include all computer storage media and communication media. Computer storage media can include all volatile or non-volatile media, as well as removable or non-removable media, implemented by any method or technology to store information such as computer-readable instructions, data structures, program modules, or other data. For example, communication media can include any information transmission medium and can include other data such as computer-readable instructions, data structures, or program modules, through other transmission mechanisms or modulated data signals.

[0137] Furthermore, in this document, "unit" may include hardware components such as processors or circuits and / or software components executed by hardware components such as processors.

[0138] The foregoing is illustrative of various examples, and it will be readily understood by those skilled in the art that various modifications can be made therein without substantially departing from the spirit or characteristics of the examples. Therefore, it should be understood that the above examples should be considered in a descriptive sense only and not for limiting purposes. For example, elements described as being combined can also be implemented in a distributed manner, and elements described as being distributed can also be implemented in a combined manner.

[0139] The scope of each example is not limited by the specific embodiments described above, but by the appended claims, and all modifications or differences within the scope of the claims should be interpreted as being included in each example.

[0140] According to the various examples, the same result can be obtained by repeating matrix multiplication several times without performing reshaping and transpose operations, thus reducing memory access counts and consequently reducing memory power consumption.

[0141] Although this disclosure includes specific examples, it will be apparent upon understanding the disclosure of this application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. Descriptions of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not defined by the detailed description but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in this disclosure.

Claims

1. A neural network device, comprising: A memory, wherein at least one program is stored; as well as A processor is configured to perform one or more operations by executing the at least one program, wherein the processor is configured to: The input feature map and initial weights are obtained from the memory. The initial weights are determined whether they are assigned in the column or row direction based on whether the reshaping and transpose operations are performed before or after matrix multiplication. The partition weights are generated by dividing the initial weights by the head count in the determined column or row direction. An intermediate feature map is generated by performing matrix multiplication between the input feature map and the partitioning weights. The final feature map is generated based on the intermediate feature map. The processor is configured as follows: When the reshaping and transpose operations are performed after matrix multiplication, the initial weights are divided along the column direction of the initial weights; and When the reshaping and transpose operations are performed before the matrix multiplication operation, the initial weights are divided in the row direction of the initial weights.

2. The neural network device according to claim 1, wherein the processor is configured to: When the reshaping and transpose operations are performed after matrix multiplication, the partition weights are generated by dividing the initial weights by the head count in the column direction of the initial weights; and The final feature map is generated by concatenating the intermediate feature maps.

3. The neural network device according to claim 1, wherein the processor is configured to: When the reshaping and transpose operations are performed before the matrix multiplication operations, the partition weights are generated by dividing the initial weights by the head count in the row direction of the initial weights; and The final feature map is generated by summing the intermediate feature maps element by element.

4. The neural network apparatus according to claim 1, wherein the matrix multiplication operation between the input feature map and the partitioning weights is one of a one-dimensional convolution operation and a two-dimensional convolution operation.

5. The neural network apparatus of claim 1, wherein the processor includes a weight partitioner, and The weight divider is configured to divide the initial weight by the head count in both the column and row directions.

6. A method in a neural network device, the method comprising: Retrieve the input feature map and initial weights from memory; The initial weights are determined whether they are divided in the column or row direction based on whether the reshaping and transpose operations are performed before or after matrix multiplication. The partition weights are generated by dividing the initial weights by the head count in the determined column or row direction. An intermediate feature map is generated by performing matrix multiplication between the input feature map and the partitioning weights; as well as The final feature map is generated based on the intermediate feature map. The initial weighting includes: When the reshaping and transpose operations are performed after matrix multiplication, the initial weights are divided along the column direction of the initial weights; and When the reshaping and transpose operations are performed before the matrix multiplication operation, the initial weights are divided in the row direction of the initial weights.

7. The method of claim 6, wherein, When the reshaping and transpose operations are performed after matrix multiplication, generating the partition weights includes: generating the partition weights by dividing the initial weights by the head count in the column direction of the initial weights; and Generating the final feature map includes: generating the final feature map by concatenating the intermediate feature maps.

8. The method of claim 6, wherein, When the reshaping and transpose operations are performed before the matrix multiplication operations, generating the partition weights includes: generating the partition weights by dividing the initial weights by the head count in the row direction of the initial weights; and Generating the final feature map includes: generating the final feature map by summing the intermediate feature maps element by element.

9. The method according to claim 6, wherein the matrix multiplication operation between the input feature map and the partitioning weights is one of a one-dimensional convolution operation and a two-dimensional convolution operation.

10. The method of claim 6, wherein generating the partitioning weights comprises: The initial weights are input into the weight divider; as well as The partitioning weight is obtained from the weight partitioner by dividing the initial weight by the head count in one of the column and row directions of the initial weight.

11. A non-transitory computer-readable recording medium storing a program that, when executed by a computer, performs the method of claim 6.

Images