Artificial intelligence neural network accelerator and method for transformer neural networks

Abstract

To provide a highly energy-efficient mobile large language model accelerator.SOLUTION: An AI neural network accelerator 100 includes a computing unit configured to predict output tokens in units of input tokens and including a plurality of transformer operation cores operating based on a transformer model using n weights, and a controller configured to control operation of each of the transformer operation cores after determining a size of the transformer model based on the number of weights. The controller performs control to perform primary prediction in which each of the plurality of transformer operation cores predicts output tokens using a reduced transformer model using m weights (m<n) in response to acquisition of the input tokens, and further perform secondary prediction of predicting output tokens for the input tokens using a transformer model before reduction only when prediction accuracy of a result of the prediction is less than or equal to a preset threshold value.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an artificial intelligence neural network accelerator and its method, and particularly to an artificial intelligence neural network accelerator and its method for a Transformer neural network that is difficult to reuse weight values, requires a large amount of weight values, has a large amount of external memory access, and consequently has a high power consumption.

Background Art

[0002] A Transformer neural network is a neural network that tracks relationships within sequential data such as words in a sentence and learns context and meaning, replacing a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN).

[0003] Such a Transformer neural network can construct a large labeled learning dataset without having to mathematically find patterns between each element, is suitable for parallel processing, and due to its fast execution speed, is widely used in image classification and large language models, and is also expected to be used in mobile systems that provide real-time responses.

[0004] However, such a Transformer neural network has the problem that it is difficult to reuse weight values, requires a large amount of weight values, has a large amount of external memory access, and as a result, power consumption increases.

[0005] Therefore, in order to solve the above problems, various techniques have been proposed to increase hardware utilization and reduce power consumption (References 1 to 3 (Non-Patent Documents 1-3)), but the system power consumption and response time of these transformer processors are still not suitable for mobile devices. For example, large language models such as GPT-2 have many weight values ​​(400M to 700M), and external memory access consumes 68% of the total power.

[0006] Furthermore, a transformer accelerator with pruning applied (Reference 4 (Non-Patent Literature 4)) has been proposed to mitigate the bottleneck phenomenon of external memory access. However, while this can increase the rarity of weighted values, it can only be applied to simple tasks such as predicting the next word (e.g., language modeling), and has the problem that it cannot achieve high rarity in advanced tasks such as language translation, question answering, and summarization.

[0007] Therefore, in order to reduce external memory access for accelerating large-scale language models for mobile devices with high energy efficiency, there is a need for a new method that can compress weights. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Reference 1: Y. Wang et al., "A 28nm 27.5TOPS / W Approximate-Computing-Based Transformer Processor with Asymptotic Sparsity Speculating and Out-of-Order Computing", 2022 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, 2022, pp. 1-3. [Non-Patent Document 2] Reference 2: F. Tu et al., "A 28nm 15.59μJ / Token Full-Digital Bitline-Transpose CIM-Based Sparse Transformer Accelerator with Pipeline / Parallel Reconfigurable Modes", 2022 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, 2022, pp. 466-468. [Non-Patent Document 3] Reference 3: F. Tu et al., "16.1 MuITCIM:A 28nm 2.24μJ / Token Attention-Token-Bit Hybrid Sparse Digital CIM-Based Accelerator for Multimodal Transformers", 2023 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, 2023, pp. 248-250. [Non-Patent Document 4] Reference 4: S. Liu et al., “16.2 A 28nm 53.8TOPS / W8b Sparse Transformer Accelerator with In-Memory Butterfly Zero Skipper for Unstructured-Pruned NN and CIM-Based Local-Attention-Reusable Engine”, 2023 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, 2023, pp. 250-252. [Overview of the project] [Problems that the invention aims to solve]

[0009] The present invention has been made to solve the above-mentioned problems and aims to provide an artificial intelligence neural network accelerator and method that can accelerate large-scale language models for mobile devices with high energy efficiency by reducing the amount of external memory access required by reducing the amount of weighting required for accelerating the artificial intelligence neural network, and consequently reducing power consumption.

[0010] Furthermore, the present invention aims to provide an artificial intelligence neural network accelerator and method that first accelerates a reduced model obtained by reducing the required amount of basic weights necessary to accelerate the basic model by a predetermined ratio, and then further accelerates the basic model in a secondary acceleration stage only in special cases where the prediction accuracy for the result is below a predetermined critical value, thereby reducing the required amount of weights for accelerating the artificial intelligence neural network.

[0011] Furthermore, the present invention provides an artificial intelligence neural network accelerator and method that includes a weight value embedding logic that is pre-generated as a result of learning each weight value of a transformer neural network matched to the kernel position information of the transformer neural network, and that reduces the amount of external memory access required to receive the weight values ​​by generating implicit weight values ​​for accelerating the artificial intelligence neural network while receiving only kernel position information from external memory, and then supplying the implicit weight values ​​using an on-chip network.

[0012] Furthermore, the present invention aims to provide an artificial intelligence neural network accelerator and method that can shorten the time required to receive kernel position information by receiving code-compressed kernel position information from external memory and decompressing it, thereby shortening the external memory access time. [Means for solving the problem]

[0013] To solve the above problems, the artificial intelligence neural network accelerator provided by the present invention is an artificial intelligence neural network accelerator that accelerates a Transformer neural network. It includes an arithmetic unit containing a number of Transformer arithmetic cores that operate based on a Transformer model that predicts output tokens in input token units and uses n (where n is a natural number) weight values; and a controller that determines the size of the Transformer model based on the number of the weight values and then controls the operations of each of the Transformer arithmetic cores. The controller controls each of the number of Transformer arithmetic cores to perform a primary prediction of predicting output tokens using a reduced Transformer model (hereinafter referred to as the "reduced model") that uses m (where m is a natural number and m < n) weight values in response to the acquisition of the input tokens, and only when the prediction accuracy of the primary prediction result is below a preset threshold value, further perform a secondary prediction of predicting output tokens for the input tokens using the Transformer model before being reduced (hereinafter referred to as the "basic model").

[0014] Preferably, the artificial intelligence neural network accelerator includes weight value embedding logic that is pre-generated as a result of learning by matching each weight value of the Transformer neural network with the a x b kernel position information of the Transformer neural network, and further includes a weight value generator that generates implicit weight values based on the position information of the kernel input from an external memory. The controller can provide the implicit weight values via an on-chip network in response to requests from the number of Transformer arithmetic cores.

[0015] Preferably, the weighted value generator may include an implicit weighted value generation unit that decompresses kernel position information input from the external memory in a sign-compressed state, and a weighted value embedding logic that applies the kernel position information decompressed by the sign-compression decompression unit to the weighted value embedding logic to generate implicit weighted values ​​corresponding to the decompressed kernel positions.

[0016] Preferably, the implicit weight generation unit may include a two-dimensional MAC array that performs multiplication and accumulation operations to generate the implicit weight using the decompressed kernel position information; and a weight embedding logic that selects a weight embedding corresponding to the kernel position information and transmits it to the two-dimensional MAC array.

[0017] Preferably, the weighting value generator may further include a transformer weighting value memory for storing the implicit weighting values; and an on-chip network switch for transmitting the implicit weighting values ​​via an on-chip network in response to a request from at least one of the plurality of transformer arithmetic cores.

[0018] On the one hand, the artificial intelligence neural network acceleration method provided by the present invention includes a number of Transformer operation cores operating based on a Transformer model using n (where n is a natural number) weight values. In the artificial intelligence neural network acceleration method using an artificial intelligence neural network accelerator for accelerating a Transformer neural network, the artificial intelligence neural network accelerator performs a primary prediction step of predicting an output token using a reduced Transformer model (hereinafter referred to as "reduced model") using m (where m is a natural number and m < n) weight values in response to obtaining an input token; a prediction accuracy calculation step of calculating the prediction accuracy for the prediction result of the primary prediction step; and a secondary prediction step in which the artificial intelligence neural network accelerator further performs a secondary prediction of predicting an output token for the input token using the Transformer model before reduction (hereinafter referred to as "basic model") when the prediction accuracy is below a preset threshold value.

[0019] Preferably, the method includes an implicit weight value generation step in which the artificial intelligence neural network accelerator has weight value embedding logic generated in advance as a result of learning by matching each weight value of an existing Transformer neural network with the axb kernel position information of the Transformer neural network, and generates an implicit weight value based on the position information of the kernel input from an external memory and the weight value embedding logic; an implicit weight value storage step in which the artificial intelligence neural network accelerator stores the implicit weight value; and an implicit weight value transmission step in which the artificial intelligence neural network accelerator transmits the implicit weight value via an on-chip network. Each of the primary prediction step and the secondary prediction step can predict an output token for the input token using the implicit weight value.

[0020] Preferably, the implicit weight value generation stage further includes a decompression stage for decompressing the position information of the kernel input in a compression-coded state from the external memory, and applying the decompressed position information of the kernel to the weight value embedding logic to generate an implicit weight value corresponding to the decompressed position of the kernel.

Advantages of the Invention

[0021] The artificial intelligence neural network accelerator and method of the present invention as described above can accelerate a high energy efficiency large language model for mobile by reducing the amount of external memory access by reducing the required amount of weight values for accelerating the artificial intelligence neural network, and as a result, reducing power consumption.

[0022] In addition, the present invention first accelerates a reduced model in which the required amount of basic weight values required to accelerate the basic model is reduced at a predetermined ratio, and further performs a secondary acceleration stage of accelerating the basic model only in a special case where the prediction accuracy for the result is below a preset critical value. As a result, it has the effect of reducing the required amount of weight values for accelerating the artificial intelligence neural network.

[0023] In addition, the present invention includes weight value embedding logic generated in advance as a result of learning each weight value of the Transformer neural network matched to the kernel position information of the Transformer neural network. After generating an implicit weight value for accelerating the artificial intelligence neural network while receiving only the kernel position information from the external memory, by supplying the implicit weight value using an on-chip network, it has the effect of reducing the amount of external memory access for receiving the weight value.

Brief Description of the Drawings

[0024] [Figure 1]This is a diagram illustrating the structure of an artificial intelligence neural network accelerator according to one embodiment of the present invention. [Figure 2] This is a diagram illustrating the structure of an artificial intelligence neural network accelerator according to one embodiment of the present invention. [Figure 3] This figure illustrates the acceleration process of an artificial intelligence neural network accelerator using a mixed structure according to one embodiment of the present invention. [Figure 4] This figure illustrates the process by which an artificial intelligence neural network accelerator according to one embodiment of the present invention generates implicit weights. [Figure 5] This figure illustrates the process of training a neural network to generate implicit weights according to one embodiment of the present invention. [Figure 6] This figure illustrates compression and decompression for the use of implicit weighting values ​​according to one embodiment of the present invention. [Figure 7] This is a processing flowchart illustrating an artificial intelligence neural network acceleration method according to one embodiment of the present invention. [Figure 8] This is a processing flowchart illustrating an artificial intelligence neural network acceleration method according to one embodiment of the present invention. [Figure 9] This graph illustrates the effects of applying an artificial intelligence neural network accelerator and its method according to one embodiment of the present invention to various transformer models. [Modes for carrying out the invention]

[0025] The following describes embodiments of the present invention with reference to the attached drawings, providing a detailed explanation so that a person with ordinary skill in the art to which the present invention pertains can easily implement the invention. However, the present invention can be embodied in various different forms and is not limited to the embodiments described herein. On the other hand, in the drawings, parts unrelated to the description have been omitted in order to clearly illustrate the present invention, and similar parts throughout the specification have been denoted by similar reference numerals. Furthermore, even when detailed explanations are omitted, explanations of parts that can be easily understood by a person skilled in the art have been omitted.

[0026] In the entirety of the specification and claims, when a part is described as containing one component, this means, unless otherwise stated, that it may contain other components rather than excluding them.

[0027] Figures 1 and 2 are diagrams illustrating the structure of an artificial intelligence neural network accelerator according to one embodiment of the present invention. Figure 1 shows a schematic block of the artificial intelligence neural network accelerator of the present invention, and Figure 2 shows in more detail the computing unit and weighted value generator included in the artificial intelligence neural network accelerator of the present invention.

[0028] Referring to Figures 1 and 2, an artificial intelligence neural network accelerator 100 according to one embodiment of the present invention includes a plurality of gateways 10, 20, a token input unit 110, a processing unit 120, a controller 130, and a weighted value generator 200. In particular, the example in Figure 2 illustrates an artificial intelligence neural network accelerator 100 that includes 48 transformer processing cores 121, 2 weighted value generators 200, a top-level controller, ID SIMD, and an on-chip network.

[0029] In this configuration, the transformer arithmetic core 121 may include eight multipliers and accumulators (for 8-bit inputs and 8-bit weighted values), an input loader, a controller, an on-chip network switch, and input / weighted / output memory to perform the necessary matrix multiplication for each decoder block of the transformer and generate output tokens each time.

[0030] Each gateway 10, 20 can connect an external memory (not shown) to the artificial intelligence neural network accelerator 100. Each gateway 10, 20 can be used to transmit each weight value stored in the external memory (not shown) to the artificial intelligence neural network accelerator 100 and to transmit each processing result generated by the artificial intelligence neural network accelerator 100 to the external memory (not shown).

[0031] The token input unit 110 receives tokens. That is, the token input unit 110 receives one token each time from an input sentence composed of multiple tokens and transmits it to the arithmetic unit 120.

[0032] The arithmetic unit 120 predicts output tokens in units of input tokens input via the token input unit 110. For this purpose, the arithmetic unit 120 may include a number of transformer arithmetic cores 121, each of which includes a transformer model.

[0033] In this case, the size of the transformer model is determined by the number of weight values ​​required, and the larger the size of the transformer model, the greater the power consumption. That is, the larger the required number of weight values, the larger the size of the transformer model. Due to this characteristic, when the required number of weight values ​​for an arbitrary transformer model using n (where n is a natural number) weight values ​​(hereinafter referred to as the "basic model") can be reduced, the size of the basic model can be reduced, and as a result, power consumption can be greatly reduced.

[0034] Therefore, the present invention uses such features to reduce a large number of tokens among the entire input tokens to m (where m is a natural number satisfying m < n) in terms of the required amount of the weighted values, and predicts using a transformer model (hereinafter referred to as a "reduced model") with its size reduced. As a result, the required amount of the weighted values for accelerating the artificial intelligence neural network is reduced, thereby reducing the external memory access amount and consequently reducing the power consumption.

[0035] For this purpose, the arithmetic unit 120 receives the control of the controller 130 described later.

[0036] After determining the size of the transformer model according to the number of the weighted values, the controller 130 can control the operations of each of the transformer arithmetic cores 121. In particular, the controller 130 can control each of the multiple transformer arithmetic cores 121 such that after each of the multiple transformer arithmetic cores 121 makes a primary prediction on the input tokens using the reduced model, the prediction accuracy for the result of the primary prediction is calculated, and a secondary prediction using the basic model is further performed only when the prediction accuracy is less than or equal to a preset threshold value (for example, a prediction accuracy of 60%).

[0037] At this time, the prediction accuracy can be calculated using various known techniques. For example, in order to calculate the prediction accuracy, the controller 130 can manage to preferentially calculate a small model and calculate a large model as appropriate.

[0038] For example, the controller 130 has each transformer computing core 121 respond to the acquisition of the input token by first predicting the output token using a reduced model in which the required number of weights of the basic model has been reduced to n / 10 (where n is a natural number). If the accuracy of the prediction exceeds the critical value, the controller 130 reduces the required number of weights by omitting the second-order prediction process. This processing process of the controller 130 is illustrated in Figure 3.

[0039] Figure 3 is a diagram illustrating the acceleration process of an artificial intelligence neural network accelerator with a mixed structure according to one embodiment of the present invention. It illustrates the process in which the controller 130 preferentially predicts output tokens using a reduced model with a low weight requirement for each input token acquired by the controller, and then additionally predicts output tokens using a basic model with a high weight requirement only for a predetermined number of input tokens that have low prediction accuracy (i.e., are difficult to distinguish).

[0040] Referring to Figure 3, there are four input tokens ( <bos>The diagram illustrates the token probabilities of each output token (the / man / walks / runs) predicted using the reduced model (i.e., a transformer model with a weight requirement of "N / 10") 30 for each of the / the / man / walks). That is, since the token probabilities (i.e., prediction accuracy) (P) of each of the remaining output tokens (the / man / walks), excluding the rightmost output token (runs), exceed the critical value (Th), the output token is additionally predicted using the basic model (i.e., a transformer model with a weight requirement of "N") 40 only for the rightmost output token (runs), and as a result, the output token (across) is predicted in which the token probability (P) exceeds the critical value (Th).

[0041] Thus, the present invention has the characteristic of significantly reducing the weight requirement compared to conventional techniques that predict output tokens for all input tokens using the basic model, by using a reduced model with a low weight requirement to predict output tokens for all input tokens, and then using a basic model with a high weight requirement to predict additional output tokens only for a small number of input tokens with low prediction accuracy.

[0042] Thus, when applying the present invention's method, which first performs calculations only on small models and skips calculations on larger models when the predicted probability of a particular token exceeds a predefined critical value, to language modeling using GPT-2, it has the effect of reducing external memory access by 39%.

[0043] Referring again to Figures 1 and 2, the weighted value generator 200 generates weighted values ​​to be provided to each of the transformer arithmetic cores 121.

[0044] The weighting generator 200 generates implicit weights using an "artificial intelligence neural network" trained to implicitly store the weights of a transformer neural network. It can receive only the axb kernel position information of the transformer neural network as input and output an implicit weight corresponding to that position. In this case, the "artificial intelligence neural network" is a neural network different from the transformer neural network, and can be trained and used as a multi-layer perceptron, which is commonly used to generate weights of a transformer neural network. The process of training the artificial intelligence neural network and moving data will be explained later with reference to Figures 5 and 6.

[0045] To generate the implicit weighted values, the weighted value generator 200 may include a decoded decompression unit 210, an implicit weighted value generation unit 220, a transformer weighted value memory 230, and an on-chip network switch 240.

[0046] The decoded decompression unit 210 decompresses the kernel position information input from external memory in a code-compressed state. To this end, the decoded decompression unit 210 includes an index memory (IDX_MEM) 211, an MSB memory (MSB_MEM) 212, a sign memory (Sign_MEM) 213, an LSB memory (LSB_MEM) 214, and a router 215 as a configuration for decompressing normally code-compressed data.

[0047] The index memory (IDX_MEM) 211 stores the address of data whose MSB is not filled with consecutive sign bits; the MSB memory (MSB_MEM) 212 stores the MSB partial value of each data (i.e., uncompressed data) whose MSB side data is not filled with sign extension bits; the sign memory (Sign_MEM) 213 stores the sign bit compressed by 1 bit; the LSB memory 214 stores the LSB data; and the router 215 decompresses the data that has been sign-compressed by the respective data information in the index memory (IDX_MEM) 211, MSB memory (MSB_MEM) 212, sign memory (Sign_MEM) 213, and LSB memory 214, and transmits this to an 8-bit queue.

[0048] The implicit weight generation unit 220 includes a two-dimensional MAC array 221 that performs multiplication and accumulation operations to generate implicit weights using the kernel position information decompressed by the code decompression unit 210, and a weight embedding logic 222 that is pre-generated as a result of learning by matching each weight of the transformer neural network with the axb kernel position information of the transformer neural network.

[0049] When the implicit weighting generation unit 220 receives the kernel position information decompressed by the code decompression unit 210, it can detect the weighting embedding corresponding to the position information from the weighting embedding logic 222 and generate an implicit weighting value.

[0050] To this end, the weighted value embedding logic 222 generates implicit weights by selecting weighted value embeddings corresponding to the kernel position information decompressed by the sign decompression unit 210 and transmitting them to the 2D MAC array 221.

[0051] The transformer weight memory 230 stores the implicit weights generated by the implicit weight generation unit 220.

[0052] The on-chip network switch 240, in response to a request from at least one of the numerous transformer arithmetic cores 121, transmits an implicit weight value stored in the transformer weight value memory 230 to that transformer arithmetic core 121 via the on-chip network. For this purpose, the on-chip network switch 240 can be controlled by the controller 130. That is, the controller 130 can control the operation of the on-chip network switch 240 to provide the implicit weight value via the on-chip network in response to requests from the numerous transformer arithmetic cores 121.

[0053] The processing steps of this weighted value generator 200 are illustrated in Figure 4.

[0054] Figure 4 is a diagram illustrating the process by which an artificial intelligence neural network accelerator according to one embodiment of the present invention generates implicit weights. Referring to Figure 4, when a sign-compressed weight with Start IDX = 0 and End IDX = 7 is input to the decode unit 210 to generate the first eight weights of the neural network, the decode unit 210 receives address values ​​from IDX MEM 211 and checks whether there are any weights between weights 0 to 7 whose MSB is filled with consecutive sign bits. In the example in Figure 4, the addresses of data whose MSB is not filled with consecutive sign bits are 1 and 6, so IDX MEM 211 stores 1 and 6, and the decode unit 210 reads the address values ​​of 1 and 6. In this case, the decoded decompression unit 210 determines that the MSBs of weights 0, 2, 3, 5, and 7 are all filled with code extension bits, but the weights 1 and 6 are not. The decoded unit 210 then fills the MSB data for weights 0, 2, 3, 5, and 7 by retrieving the code bits compressed into 1 bit from the Sign MEM 213, and fills the MSB data for weights 1 and 6 by retrieving the data from the MSB MEM 212. This is to supply the MSB data completely to the implicit weight generation unit 220. At this time, the MSB data router 215 performs the process of retrieving the two data necessary for weights 1 and 6 from the MSB MEM 212. On the other hand, whether the MSB portion is filled by retrieving data from the MSB MEM 212 for each weighted value, or by retrieving data from the LSB MEM 214, depends on the situation, and this is determined by the multiplexer logic connected to the rear end of the MSB data router 215.

[0055] The data decompressed through this process (i.e., the decompressed kernel position information) is stored in an 8-bit queue and then transmitted to a 2D MAC array 221, where an implicit weight is generated by detecting a weight embedding corresponding to the position information from the weight embedding logic 222.

[0056] The implicit weights thus generated are used in decoder block operations (not shown) that make up the transformer network, and specifically, they can be used in attention or feed-forward operations.

[0057] Figure 5 is a diagram illustrating the process of training a neural network to generate implicit weights according to one embodiment of the present invention, and Figure 6 is a diagram illustrating compression and decompression for the use of implicit weights according to one embodiment of the present invention.

[0058] First, referring to Figure 5, in order to train the artificial intelligence neural network, the embedding value 310, which is based on the position information of a specific axb kernel among the weight values ​​of the transformer neural network, is first transmitted to the artificial intelligence neural network 320, which is composed of a multilayer perceptron. Then, through the calculations of the artificial intelligence neural network 320, a specific M x N kernel is generated, and by collecting all of these kernels, a new transformer neural network 330 can be generated. In this learning process, the most important thing is how similar the generated new transformer neural network 330 can be to the existing transformer neural network 340. Therefore, for this purpose, the difference (loss) between the results generated by the new transformer neural network 330 and the existing transformer neural network 340 is calculated and substituted into the loss function. After that, learning is performed on the artificial intelligence neural network 320 that generates the transformer network through backpropagation. The artificial intelligence neural network 320 generated through this learning process can then be used as the weighting value embedding logic 222 for the implicit weighting value generation unit 220 to reduce external memory access.

[0059] Referring to Figure 6, as a data movement for the use of implicit weights, during external compression of the chip, an embedding is generated while the M x N transformer neural network weights at specific locations are input to the artificial intelligence neural network 320. Then, instead of the M x N transformer neural network weights, only a small-sized embedding is stored in DRAM, and the chip reduces external memory access by loading only this embedding. During internal decompression of the chip, the embedding with location information of the kernel to be processed is input to the artificial intelligence neural network 320, and the artificial intelligence neural network 320 generates implicit weights of the transformer neural network through computation. The implicit weights of the transformer neural network generated at this time are used for language model work.

[0060] Figures 7 and 8 are processing flowcharts illustrating an artificial intelligence neural network acceleration method according to one embodiment of the present invention. The artificial intelligence neural network acceleration method according to one embodiment of the present invention will be described below with reference to Figures 1 to 8.

[0061] First, in step S110, the artificial intelligence neural network accelerator 100 of the present invention generates implicit weights. Specifically, in step S110, the artificial intelligence neural network accelerator 100 is equipped with a weight embedding logic that has been pre-generated as a result of learning by matching each weight of an existing transformer neural network with the axb kernel position information of the transformer neural network, and generates implicit weights based on the kernel position information input from external memory and the weight embedding logic.

[0062] For this purpose, in step S111, the implicit weight value generation unit 220 stores weight value embedding logic, and in steps S112 and S113, the compression of the position information of the kernel input in a code-compressed state from the external memory is released.

[0063] In step S114, the implicit weight value generation unit 220 applies the position information of the kernel decompressed in step S113 to the weight value embedding logic to generate an implicit weight value corresponding to the position of the decompressed kernel. That is, in step S114, the implicit weight value generation unit 220 generates an implicit weight value based on the decompressed kernel position and the corresponding weight value embedding logic.

[0064] In step S115, the transformer weight memory 230 stores the implicit weight value.

[0065] In steps S120 and S130, the transformer operation core 121 performs a primary prediction of predicting an output token using a reduced transformer model (hereinafter referred to as "reduced model") that uses m (where m is a natural number such that m < n) weight values in response to the acquisition of input tokens. For this purpose, in step S130, the transformer operation core 121 is under the control of the controller 130.

[0066] In step S140, the controller 130 calculates the prediction accuracy for the primary prediction result in step S130.

[0067] Furthermore, in stages S150 and S160, the controller 130 compares the prediction accuracy with a preset critical value, and if the prediction accuracy is less than or equal to the preset critical value, controls the transformer computing core 121 to perform a second-order prediction using the transformer model before reduction (hereinafter referred to as the "basic model") to predict the output token for the input token. That is, in stage S160, the transformer computing core 121 performs the second-order prediction under the control of the controller 130.

[0068] To this end, in steps S130 and S160, the transformer arithmetic core 121 may further include an implicit weighting transfer step in which the transformer weighting memory 230 requests an implicit weighting, and in response, the on-chip network switch 240 transmits the implicit weighting via the on-chip network.

[0069] Furthermore, in steps S130 and S160, the transformer arithmetic core 121 predicts the output token for the input token using the implicit weighting value.

[0070] Thus, the present invention has the effect of reducing external memory access by performing some calculations internally in the accelerator instead of calling all weight values ​​for accelerating the transformer neural network from an external source. In other words, the present invention can increase energy efficiency while maintaining the accuracy of transformer inference through a mixed structure of large and small artificial intelligence neural networks and implicit weight value generation.

[0071] As one example, in the case of language modeling using GPT-2, the present invention's method, which first calculates only small models and skips the calculation of larger models when the predicted probability of a particular token exceeds a predefined critical value, can reduce external memory access by 39%. Furthermore, by generating implicit weights using only kernel positions as input, external memory access can be reduced by 60%, and by applying code compression techniques to compress the artificial intelligence neural network used for generating implicit weights, external memory access can be reduced by up to 74%.

[0072] As another example, in the case of language translation using mT5, external memory access can be reduced by 42% through a mixed network structure of large and small sizes, by 67% through implicit weighting generation, and by up to 78% through code compression techniques.

[0073] Furthermore, in the case of summarization using T5, external memory access can be reduced by 59% through a mixed network structure of large and small sizes, by 71% through implicit weighting generation, and by up to 78% through code compression techniques.

[0074] Finally, in the case of language translation using FSMT, external memory access can be reduced by 48% through a mixed network structure of large and small networks, by 72% through implicit weighting generation, and by up to 81% through code compression techniques.

[0075] The effects of the present invention are illustrated in Figure 9.

[0076] Figure 9 is a graph illustrating the effects of applying an artificial intelligence neural network accelerator and method according to one embodiment of the present invention to various transformer models (e.g., GPT-2, mT5, T5, FSMT), where Figure 9(a) shows the reduction in weighted loading energy for each of the aforementioned models when the present invention is applied, Figure 9(b) shows the difference in accuracy compared to the prior art when the present invention is applied to each of the aforementioned models, and Figure 9(c) shows the difference in weighted value compared to the prior art when the present invention is applied to each of the aforementioned models.

[0077] Referring to Figure 9(a), it can be seen that when the present invention is applied to GPT-2, the weighted loading energy is reduced by 71%, when the present invention is applied to mT5, the weighted loading energy is reduced by 73%, when the present invention is applied to T5, the weighted loading energy is reduced by 76%, and when the present invention is applied to FSMT, the weighted loading energy is reduced by 76%.

[0078] Furthermore, referring to Figure 9(b), it can be seen that when the present invention is applied to GPT-2, the accuracy decreases by 1.26; when the present invention is applied to mT5, the accuracy decreases by 0.59; when the present invention is applied to T5, the accuracy decreases by 0.96; and when the present invention is applied to FSMT, the accuracy decreases by 1.29.

[0079] On the other hand, referring to Figure 9(c), it can be seen that when the present invention is applied to GPT-2, mT5, T5, and FSMT, the weighted value is significantly reduced.

[0080] Thus, compared to conventional technologies, the present invention exhibits an accuracy of -0.52 to -1.29, a weighted value compression ratio of 74% to 81%, and a weighted value loading energy reduction ratio of 71% to 76%.

[0081] Thus, the artificial intelligence neural network accelerator and method of the present invention have the characteristic of being able to accelerate large-scale language models for mobile devices with high energy efficiency by reducing the amount of external memory access required by reducing the amount of weight values ​​required for accelerating the artificial intelligence neural network, and as a result reducing power consumption.

[0082] Furthermore, the present invention has the characteristic that it can reduce the amount of weights required to accelerate the artificial intelligence neural network. This is achieved by first accelerating a reduced model, which has a predetermined ratio of the amount of basic weights required to accelerate the basic model, and then performing a second acceleration step in which the basic model is accelerated only in special cases where the prediction accuracy of the result is below a predetermined critical value.

[0083] Furthermore, the present invention includes a weight value embedding logic that is pre-generated as a result of learning each weight value of a transformer neural network matched to the kernel position information of the transformer neural network. It has the characteristic of reducing the amount of external memory access required to receive the weight values ​​by generating implicit weight values ​​for accelerating the artificial intelligence neural network while receiving only kernel position information from external memory, and then supplying the implicit weight values ​​using an on-chip network.

[0084] While embodiments of the present invention have been described above, the scope of the present invention is not limited thereto, and the present invention includes all changes and modifications that are readily apparent to a person skilled in the art to which the present invention pertains and are considered equivalent to the embodiments. [Explanation of Symbols]

[0085] 100 Artificial Intelligence Neural Network Accelerators 110 Token input section 120 Arithmetic section 121 Transformer Computing Cores 130 Controllers 200 Weighted Value Generator 210 Code decompression unit 211 IDX_MEM 212 MSB_MEM 213 Sign_MEM 214 LSB_MEM 215 Lauter 220 Implicit Weighting Value Generation Unit 221 2D MAC Array 222 Weighted Value Embedding Logic 230 Transformer Weighted Value Memory 240 On-Chip Network Switches< / bos>

Claims

1. In an artificial intelligence neural network accelerator that accelerates transformer neural networks, An arithmetic unit including a number of transformer arithmetic cores that predict output tokens on an input token basis and operate based on a transformer model using n (where n is a natural number) weighted values; and A controller that controls the operation of each of the transformer computing cores after determining the size of the transformer model by the number of weight values; The aforementioned controller, Each of the numerous transformer computing cores is controlled to perform a first-order prediction in response to acquiring an input token, using a reduced transformer model (hereinafter referred to as the "reduced model") with m (where m is a natural number such that m < n) weighted values ​​to predict the output token, and only if the prediction accuracy of the first-order prediction is below a predetermined critical value, to further perform a second-order prediction using the original transformer model (hereinafter referred to as the "basic model") to predict the output token for the input token. Furthermore, it includes a weight value embedding logic that is pre-generated as a result of learning by matching each weight value of the transformer neural network with the a x b kernel position information of the transformer neural network, and further includes a weight value generator that generates implicit weight values ​​based on kernel position information input from external memory. The aforementioned controller, In response to the requests of the numerous transformer computing cores, the implicit weighting values ​​are provided via an on-chip network. An artificial intelligence neural network accelerator characterized by the following features.

2. The aforementioned weighted value generator is A decompression unit for decompressing kernel position information input from the external memory in a code-compressed state; and, An implicit weight generation unit comprising the weight value embedding logic, which applies the kernel position information decompressed by the decompression unit to the weight value embedding logic to generate an implicit weight value corresponding to the position of the decompressed kernel; The artificial intelligence neural network accelerator according to claim 1.

3. The implicit weighting value generation unit is, A two-dimensional MAC array that performs multiplication and accumulation operations to generate the implicit weighted value using the position information of the decompressed kernel; and Includes a weighted embedding logic that selects a weighted embedding corresponding to the kernel's position information and transmits it to the two-dimensional MAC array; The artificial intelligence neural network accelerator according to claim 2.

4. The aforementioned weighted value generator is A transformer weighting memory for storing the implicit weighting values; and An on-chip network switch that transmits the implicit weighting value via an on-chip network in response to a request from at least one of the numerous transformer computing cores; further comprising The artificial intelligence neural network accelerator according to claim 2.

5. An artificial intelligence neural network acceleration method using an artificial intelligence neural network accelerator that has a large number of transformer computing cores that operate based on a transformer model using n (where n is a natural number) weight values, In the first prediction stage, the artificial intelligence neural network accelerator performs a first prediction in response to the acquisition of an input token, using a reduced transformer model (hereinafter referred to as the "reduced model") that uses m (where m is a natural number such that m < n) weight values ​​to predict the output token; The artificial intelligence neural network accelerator performs a prediction accuracy calculation step in which it calculates the prediction accuracy for the prediction result of the first prediction step; and, The artificial intelligence neural network accelerator includes a second prediction step in which, if the prediction accuracy is below a predetermined critical value, a second prediction is performed using the transformer model before reduction (hereinafter referred to as the "basic model") to predict the output token for the input token; Furthermore, the artificial intelligence neural network accelerator includes a weight value embedding logic that is pre-generated as a result of learning by matching each weight value of an existing transformer neural network with the a x b kernel position information of the transformer neural network, and an implicit weight value generation step that generates implicit weight values ​​based on the kernel position information input from external memory and the weight value embedding logic; An implicit weighting value storage step in which the artificial intelligence neural network accelerator stores the implicit weighting values; and, The artificial intelligence neural network accelerator further includes an implicit weighting transfer step in which the implicit weighting is transmitted via an on-chip network; Each of the above-mentioned primary prediction stage and secondary prediction stage is: Using the implicit weighting value, predict the output token for the input token. A method for accelerating artificial intelligence neural networks, characterized by the following:

6. The aforementioned implicit weighting value generation step is The process further includes a decompression step for decompressing the kernel position information input from the external memory in a code-compressed state, The kernel position information decompressed in the decompression step is applied to the weighted value embedding logic to generate an implicit weighted value corresponding to the decompressed kernel position. The method for accelerating an artificial intelligence neural network according to claim 5.

Images