Accelerator configured to perform artificial intelligence calculation, method of operating accelerator, and artificial intelligence system including accelerator

Abstract

Disclosed is an accelerator for performing artificial intelligence (AI) calculations, comprising a processing element for generating first result data by performing a first calculation on first activation data and first weight data loaded from a memory, and a quantizer for generating first output data by performing quantization on the first result data, and the first activation data, the first weight data, and the first output data are of a low precision type, the first result data are of a high precision type, and the first output data are stored in a memory.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10 - 2024 - 0010718, filed on January 24, 2024, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Embodiments of the present disclosure described herein relate to an artificial intelligence system, and more particularly, to an accelerator configured to perform artificial intelligence calculations, a method of operating the accelerator, and an artificial intelligence system including the accelerator.

[0004] Recently, artificial intelligence (AI) is a field of computer science widely used in various fields such as natural language understanding, natural language translation, robotics, artificial vision, problem - solving, learning, knowledge acquisition, and cognitive science.

[0005] Artificial intelligence is implemented based on various algorithms. As an example, a neural network consists of a complex network in which nodes and synapses are repeatedly connected. In the process of moving data from a current node to a next node, various signal processes may occur according to corresponding synapses, and this signal process is called a "layer". Specifically, a neural network may include various layers that are complexly connected to each other. Since the various layers included in the neural network require a large amount of calculations, various methods for optimizing calculations are being studied. In other words, due to the large amount of calculations even in a single layer, even a small improvement in calculation optimization may have a significant impact on the speed, efficiency, power consumption, etc. of a multi - layer network. Summary of the Invention

[0006] Embodiments of the present disclosure provide an accelerator configured to perform artificial intelligence calculations at reduced cost and improved performance, a method of operating the accelerator, and an artificial intelligence system including the accelerator.

[0007] According to at least one embodiment of the present disclosure, an accelerator for performing artificial intelligence (AI) operations includes: a processing element configured to generate first result data by performing a first operation on first activation data and first weight data loaded from a memory, and a quantizer configured to generate first output data by performing quantization on the first result data, wherein the first activation data, the first weight data, and the first output data are of a low - precision type, the first result data is of a high - precision type, and wherein the accelerator is configured to store the first output data in the memory and / or enable the first output data to be stored in the memory.

[0008] According to at least one embodiment of the present disclosure, a method of operating an accelerator configured to perform artificial intelligence (AI) operations includes: loading first activation data and first weight data from a memory, generating first result data by performing a first operation based on the first activation data and the first weight data, performing quantization on the first result data to generate first output data, and storing the first output data in the memory, and the first activation data, the first weight data, and the first output data are of a low-precision type, and the first result data is of a high-precision type.

[0009] According to at least one embodiment of the present disclosure, an artificial intelligence system includes: a memory configured to store first activation data and first weight data; an accelerator configured to load the first activation data and the first weight data from the memory, perform a first calculation on the first activation data and the first weight data to generate first result data, and generate first output data by performing quantization on the first result data, and a CPU (Central Processing Unit) that controls the memory and the accelerator, and the first activation data, the first weight data, and the first output data are of a low-precision type, the first result data is of a high-precision type, and the first output data is stored in the memory. Brief Description of the Drawings

[0010] By describing embodiments of the present disclosure in detail with reference to the accompanying drawings, the above and other objects and features of the present disclosure will become apparent.

[0011] Figure 1 is a block diagram showing a comparison system for processing an artificial intelligence model.

[0012] Figure 2 is a block diagram showing a system according to at least one embodiment of the present disclosure.

[0013] Figure 3 is for describing Figure 2 a conceptual diagram of a deep learning layer executed by an accelerator.

[0014] Figure 4 is for describing Figure 2 the concept of MAC calculation executed in an accelerator.

[0015] Figure 5 is for describing Figure 2 the quantization operation executed by a quantizer in an accelerator.

[0016] Figure 6 is showing Figure 2 a block diagram of an accelerator.

[0017] Figure 7 is showing Figure 6 a block diagram of a quantizer.

[0018] Figure 8 is a block diagram of one quantization kernel among multiple quantization kernels that shows Figure 7 .

[0019] Figure 9 is a block diagram of a conversion circuit that shows Figure 8 .

[0020] Figure 10 is a flowchart of the operation of an accelerator that shows Figure 2 .

[0021] Figures 11A to 11C is a diagram for describing the quantization operation performed by a quantizer that is Figure 7 .

[0022] Figure 12 is a block diagram of an accelerator that shows Figure 2 .

[0023] Figure 13 is a diagram showing the structure of an accelerator according to at least one embodiment of the present disclosure.

[0024] Figure 14 is a diagram showing the structure of an accelerator according to at least one embodiment of the present disclosure.

[0025] Figure 15 is a block diagram showing the structure of an accelerator according to at least one embodiment of the present disclosure.

[0026] Figure 16 is a block diagram of a system according to at least one embodiment of the present disclosure.

[0027] Figure 17 is a block diagram of a system according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure can be described in detail and clearly so that those of ordinary skill in the art can easily implement the present disclosure.

[0029] The terms "unit", "module", "-er / machine", etc. used in the specification or the drawings in the specification, or the functional blocks shown in the drawings, may be implemented in the form of a processing circuit, which includes software, hardware, or a combination thereof configured to perform specific functions. As an example, a "computing module" may be a hardware circuit configured to perform the corresponding functions or computations described herein. Additionally, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc., and / or may include active and / or passive electrical components, such as transistors, resistors, capacitors, etc., and / or an electronic circuit including one or more of the aforementioned components.

[0030] Figure 1 is a block diagram showing a comparison system for processing an artificial intelligence model. Referring Figure 1 , system 10 may include a central processing unit (CPU) 11, a memory 12, and an accelerator 13. The CPU 11 may be configured to control the overall operation of the system. The memory 12 may be configured to store information, such as various information for system operation. The accelerator 13 may be configured to perform various learning and / or inference using the artificial intelligence model stored in the memory 12 under the control of the CPU 11.

[0031] The accelerator 13 may be configured to perform learning and / or inference on the artificial intelligence model using the artificial intelligence model (e.g., weights) stored in the memory 12. For example, the accelerator 13 may be configured to perform repeated multiplication and addition calculations on the inputs (e.g., activations and weights) to perform a learning operation or an inference operation based on the artificial intelligence model. In this case, the inputs (e.g., activations and weights), intermediate calculation values, and / or calculation results of the accelerator 13 are stored in the memory 12, and the accelerator 13 repeatedly accesses the memory 12 to perform learning or inference. In this case, when the size of the data stored in the memory 12 is large, a large bandwidth and a large capacity are required in the memory 12. Additionally, due to the limitation of the memory bandwidth, when the size of the data transferred between the accelerator 13 and the memory 12 is large, the calculation speed of the accelerator 13 may decrease and / or the power consumed in the memory 12 may increase.

[0032] The artificial intelligence model (or weights) stored in the memory 12 can be quantized by the CPU 11 after learning is completed. Quantization can refer to an operation of converting relatively high-precision data into relatively low-precision data. For example, the first data can have a value represented as floating-point-32 (FP32). In this case, quantization is performed on the first data such that the first data can be converted into a value represented as Int-8 (Integer-8). Reference will be made to Figure 5 describe quantization in more detail. When the artificial intelligence model (or weights) is quantized, the capacity of the data related to the artificial intelligence model can be reduced.

[0033] As described above, the artificial intelligence model (or weights) stored in the memory 12 can be quantized by the CPU 11 such that the data related to the artificial intelligence model can occupy relatively less capacity. Therefore, the speed at which the accelerator 13 accesses the artificial intelligence model (or weights) can be increased, or the power consumption for accessing the memory 12 can be reduced. However, when the accelerator 13 performs inference on the artificial intelligence model, the calculation result and / or the intermediate calculation value output from the accelerator 13 can be data with relatively high precision. The calculation result and / or the intermediate calculation value can be used as an input for subsequent layers of the accelerator. Specifically, since the accelerator 13 repeatedly accesses data with relatively high precision and / or large capacity from the memory 12, even when the data is quantized, the memory 12 still requires high bandwidth and large capacity, and thus the power consumption used in the memory 12 increases. Therefore, the cost of implementing the system 10 increases and / or the performance of the system 10 deteriorates.

[0034] Figure 2 is a block diagram showing a system according to at least one embodiment of the present disclosure. Figure 3 is for describing the Figure 2 conceptual diagram of the deep learning layer executed by the accelerator. Reference Figure 2 and Figure 3 , the system 100 may include a memory 101, a controller 102, and an accelerator 1000. In at least one embodiment, the system 100 can be dedicated hardware configured to process an artificial intelligence model. For example, the system 100 can be (or be included in) a graphics processing unit (GPU), a neural processing unit (NPU), and / or separate dedicated hardware. In at least one embodiment, the system 100 can be included in an application processor (AP), a data server, a mobile device, etc.

[0035] In at least one embodiment, the artificial intelligence model driven by the system 100 can be created and / or updated through machine learning. Machine learning can include various learning methods, such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, but the scope of the present disclosure is not limited thereto.

[0036] In at least one embodiment, an artificial intelligence model can be generated or trained by one or a combination of at least two of various neural networks such as a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, etc. The artificial intelligence model can include multiple neural network layers, and each of the multiple neural network layers can be configured to perform artificial intelligence computations based on a learned model or weights. In at least some embodiments, for example, the system 100 can be applied to a smart phone, a tablet device, a smart TV, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, a robot, a medical device, a drone, an advanced driver assistance system (ADAS), an image display device, a data processing server, a measurement device, etc. that perform speech recognition, image recognition, image classification, and image processing by using a neural network, and / or can be installed in one of various electronic devices.

[0037] Hereinafter, for ease of description of embodiments of the present disclosure, the term "artificial intelligence computation" is used. "Artificial intelligence computation" can collectively refer to various computations performed within the system 100 so that the system 100 trains an artificial intelligence model and / or infers a certain result. As an example, artificial intelligence computation can include multiplication and accumulation (MAC) computations performed in each layer of the artificial intelligence model.

[0038] For example, as Figure 3 shown, the system 100 can operate based on a deep neural network. The deep neural network can include an input layer IL, a hidden layer HL, and an output layer OL. The input layer IL can be configured to receive input data or input features F1 to F4. The input features F1 to F4 received through the input layer IL can be calculated (e.g., weight multiplication) based on corresponding weights, and the result can be transmitted to a subsequent layer (e.g., the hidden layer HL). The hidden layer HL can include one or more layers and can be configured to perform weight multiplication on various inputs and accumulate the results. The result of the hidden layer HL can be provided to the output layer OL. The output layer OL can perform multiplication or accumulation of the result of the hidden layer HL and can output inference results IFR1 and IFR2. As described above, the deep neural network can repeatedly perform multiplication and accumulation (MAC) computations on input data and weights. However, the scope of the present disclosure is not limited thereto, and the system 100 can be configured to perform various computations.

[0039] The memory 101 may be configured to store various data, weights, parameters, etc. used in artificial intelligence calculations performed by the system 100. For example, the memory 101 may store an artificial intelligence model for the artificial intelligence calculations of the system 100. The artificial intelligence model may include various weight information. In at least one embodiment, the memory 101 may be a dynamic random access memory (DRAM). However, the scope of the present disclosure is not limited thereto, and the memory 101 may include various types of memories, such as static RAM (SRAM), phase change RAM (PRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), flash memory, etc.

[0040] The accelerator 1000 may be configured to perform artificial intelligence calculations using the data, weights, and / or parameters stored in the memory 101. In at least one embodiment, the accelerator 1000 may include a plurality of processing elements PEs configured to perform artificial intelligence calculations. For example, each of the plurality of processing elements may be configured to perform multiply-accumulate (MAC) calculations on the data, weights, or parameters stored in the memory 101. The artificial intelligence model may be trained, or a specific result may be inferred based on the calculation results of the plurality of processing elements.

[0041] The controller 102 may be configured to control the memory 101 and the accelerator 1000. In at least one embodiment, the controller 102 may be a central processing unit (CPU) configured to control the overall operation of the system 100.

[0042] In at least one embodiment, as referred to Figure 1 above, the artificial intelligence model (or weights) stored in the memory 101 may be quantized by the CPU 11. Conversely, the accelerator 1000 may perform calculations based on the data type (with relatively high accuracy) to improve the accuracy of artificial intelligence inference. In this case, the calculation results and / or intermediate calculation values determined by the accelerator 1000 may be data types with relatively high precision. In this case, similar to that described in reference Figure 1 above, the calculation results or intermediate calculation values of the accelerator 1000 are stored in the memory 101, so the memory access speed may be reduced and / or the power consumption may increase.

[0043] Additionally, the accelerator 1000 according to at least one embodiment of the present disclosure may include a quantizer 1100. The quantizer 1100 may be configured to perform quantization on the intermediate calculation values and / or calculation results of the accelerator 1000. Accordingly, the accelerator 1000 may be configured to perform quantization on the intermediate calculation values and / or calculation results generated during the execution of artificial intelligence calculations and / or during the inference process. In this case, since the intermediate calculation values and / or calculation results are quantized, the capacity of the intermediate calculation values and / or calculation results stored in the memory 101 is reduced. Therefore, the bandwidth and capacity required for the memory 101 may also be reduced. Additionally, since the intermediate calculation values and / or calculation results are quantized, the power consumption used in the memory 101 may be reduced. Therefore, the implementation cost of the system 100 may be reduced or the performance of the system 100 may be improved.

[0044] As an example, in the inference of an artificial intelligence system regarding recent large language models (such as Chat-GPT), a large amount of data is loaded or stored from / to the memory. In this case, when the memory resources are limited, bottlenecks associated with memory access may occur. This results in a reduction in the overall performance of the artificial intelligence system. In contrast, according to the present disclosure, when quantization is performed on the calculation result data and / or subsequent activation data within the accelerator (e.g., by a quantizer included in the accelerator), the total capacity of the data stored in and / or loaded from the memory is reduced. Therefore, the impact of potential bottlenecks in the limited memory resources can be reduced and / or avoided, and an accelerator and / or an artificial intelligence system with improved performance and reduced cost are provided.

[0045] Figure 4 is a diagram for describing the concept of MAC calculations performed in the Figure 2 accelerator. In at least one embodiment, the MAC calculation may include one multiplication calculation and one accumulation calculation on two pieces of data.

[0046] For example, as Figure 4 shown, a multiplication may be performed on an N-bit weight and an N-bit activation, where N is an integer. In this case, the multiplication result has a size of 2N bits. Thereafter, when the multiplication results are accumulated, the accumulated result may have a size of (2N + M) bits, where M is an integer. When the size of the accumulated result increases, the memory capacity for storing the accumulated result increases. Therefore, it is beneficial to reduce the size of the accumulated result. Accordingly, the accumulated result may be normalized to N-bit output data. The normalized output data may be stored in the memory 101 again. The output data stored in the memory 101 may be used to activate subsequent MAC calculations.

[0047] In at least one embodiment, the data in the MAC calculation can have various data types. For example, the data in the MAC calculation can be of integer type. Alternatively, the data for the MAC calculation can be of floating-point type. The floating-point type expresses data in the form of a sign, a fraction, and an exponent. In the floating-point type, there are methods such as 32-bit single-precision and 64-bit double-precision. Depending on the type of data, the size of the data, etc., the accuracy of the calculation result, the area of the hardware structure, and the power consumption can change. Therefore, the type and size of the data can be determined in various ways according to the purpose of system 100.

[0048] In at least one embodiment, the MAC calculation is performed based on the floating-point type. In this case, the input data (e.g., weights and activations) and the output data (e.g., the MAC calculation result) have the floating-point type. As mentioned above, the floating-point type has relatively high precision, but requires a relatively large number of bits to represent one piece of information. In this case, when loading the input data from the memory 101 or storing the output data in the memory 101, the power consumption increases and / or a relatively long time is required. However, the exemplary embodiments are not limited thereto.

[0049] According to at least one embodiment of the present disclosure, the accelerator 1000 includes a quantizer 1100, and the quantizer 1100 is configured to perform quantization on the output data to represent the output data with a relatively small number of bits. The output data with a relatively small number of bits is stored in the memory 101, and the output data stored in the memory 101 can be loaded into the accelerator 1000 as an activation for a subsequent MAC calculation. Specifically, the quantizer 1100 of the accelerator 1000 performs quantization on the intermediate calculation values and / or calculation results generated during the artificial intelligence calculation process of the accelerator 1000, thereby reducing the size of the information stored in the memory 101. Therefore, the overall performance of the system 100 can be improved and / or the power consumption can be reduced.

[0050] Figure 5 is a diagram for describing the quantization operation performed by the quantizer in the Figure 2 accelerator. Referring to Figure 5 , data can be expressed in various ways. For example, the data can be expressed as a high-precision type HP. In at least one embodiment, the high-precision type can refer to the floating-point type, which represents data in the form of a sign, a fraction, and an exponent. Alternatively, the data can be represented as a low-precision type LP. In at least one embodiment, the low-precision type LP can refer to an integer type that represents data in integer form. In at least one embodiment, compared with the high-precision type HP, the low-precision type LP can represent one piece of data with a relatively small number of bits, but may have relatively low precision.

[0051] In at least one embodiment, the high-precision type HP can be expressed with a relatively large number of bits, such as the brain floating-point format BF16 type, the half-precision IEEE floating-point format FP16 type, the single-precision floating-point format FP32 type, and the double-precision floating-point format FP64 type, and can include data types with relatively high precision. The low-precision type LP can be expressed with a relatively small number of bits (such as INT4, INT8, INT16, etc.) and can include data types with relatively low precision. In at least one embodiment, the low-precision type LP can be implemented as a combination of integer data types (such as an integer data type with a width of 4 (INT4 type), an integer type with a width of 8 (INT8 type), an integer type with a width of 16 (INT16 type), etc.) and floating-point data types (such as BF16 type, FP16 type, FP32 type, etc.), but the low-precision type LP can be expressed with a relatively small number of bits compared to the high-precision type HP.

[0052] The quantizer 1100 can convert (or reduce) high-precision type data to low-precision type data. As an example, the quantizer 1100 can convert high-precision type HP data to low-precision type LP data. In this case, the number of bits required to represent the data can be reduced. In this case, since the capacity of the data stored in and / or loaded from the memory 101 is reduced, the requirements for memory bandwidth and memory capacity can be lowered.

[0053] Figure 6 is a block diagram of Figure 2 the accelerator shown. Referring to Figure 2 and Figure 6 the accelerator 1000 can include a quantizer 1100 and a processing element 1200.

[0054] The processing element 1200 can be configured to load the activation data ACT and the weight data WT stored in the memory 101 and perform artificial intelligence calculations on the loaded activation data ACT and the loaded weight data WT. For example, the processing element 1200 can include a processing circuit configured to repeatedly perform MAC calculations on the activation data ACT and the weight data WT and can output calculation result data RST.

[0055] In at least one embodiment, the processing element 1200 can perform artificial intelligence calculations based on the high-precision type HP. For example, the processing element 1200 can perform MAC calculations on the activation data ACT and the weight data WT based on the FP16 type, which is a type of the high-precision type HP. In this case, the calculation result data RST output by the processing element 1200 will have the high-precision type HP.

[0056] Quantizer 1100 can be configured to perform quantization on the computation result data RST of processing element 1200 and generate output data OUT. For example, as described above, the computation result data RST can be of high-precision type HP. Quantizer 1100 can perform quantization on the computation result data RST of high-precision type HP and can generate output data OUT of low-precision type LP. The output data OUT of low-precision type LP can be stored in memory 101.

[0057] In at least one embodiment, as referred to Figure 4 above, accelerator 1000 repeatedly performs artificial intelligence computations through multiple layers. The output data OUT stored in memory 101 can be used as an input (e.g., activation data ACT) for the artificial intelligence computations of subsequent layers of accelerator 1000. In this case, since the output data OUT stored in memory 101 is of low-precision type LP, the activation data ACT input to processing element 1200 of accelerator 1000 for the artificial intelligence computations of subsequent layers will also be of low-precision type LP.

[0058] As described above, quantizer 1100 of accelerator 1000 can perform quantization on the computation result data RST of processing element 1200 to generate output data OUT. The output data OUT is stored in memory 101, and the output data OUT stored in memory 101 can be used as an input (e.g., activation data ACT) for subsequent layers of accelerator 1000. In this case, the data stored in memory 101 and / or loaded from memory 101 is of low-precision type LP with a relatively small capacity, so the bandwidth and capacity required for memory 101 can be reduced.

[0059] Figure 7 is a block diagram of the quantizer showing Figure 6 . Referring to Figure 6 and Figure 7 , quantizer 1100 can include a polling switch 1110, a plurality of quantization cores 1120 to 112n, and a control logic circuit 1130.

[0060] The polling switch 1110 can be configured to receive the computation result data RST as an input INPUT from the processing element 1200. In at least one embodiment, the computation result data RST (e.g., the input INPUT) can be of a high-precision type HP. The polling switch 1110 can sequentially provide the computation result data RST to the plurality of quantization kernels 1120 to 112n based on a polling method. For example, each of the plurality of quantization kernels 1120 to 112n can perform quantization on a predetermined number (e.g., “k”) of data. In this case, the polling switch 1110 can provide k computation results RST to each of the plurality of quantization kernels 1120 to 112n in a polling manner.

[0061] Each of the plurality of quantization kernels 1120 to 112n can perform a quantization operation on the input data. For example, each of the plurality of quantization kernels 1120 to 112n can include various computing modules and / or computing units for performing a quantization operation on the input data. Each of the plurality of quantization kernels 1120 to 112n can use the computing modules and / or computing units to perform a quantization operation on the input data to generate output data OUT. The output data OUT can be of a low-precision type LP. The generated output data OUT can be provided to the polling switch 1110. The polling switch 1110 can output the output data OUT to the memory 101.

[0062] The control logic circuit 1130 can be configured to control each of the plurality of quantization kernels 1120 to 112n. For example, each of the plurality of quantization kernels 1120 to 112n can operate in parallel and / or independently. The control logic circuit 1130 can control the operation timing of each of the plurality of quantization kernels 1120 to 112n.

[0063] Alternatively, each of the plurality of quantization kernels 1120 to 112n can be configured to perform quantization based on various algorithms. According to the quantization algorithm executed in each of the plurality of quantization kernels 1120 to 112n, the computing modules and the computing order executed in each of the plurality of quantization kernels 1120 to 112n can vary. The control logic circuit 1130 can be configured to individually control the computing modules of each of the plurality of quantization kernels 1120 to 112n according to the quantization algorithm executed in each of the plurality of quantization kernels 1120 to 112n.

[0064] Figure 8 is a block diagram of one of the plurality of quantization kernels shown Figure 7 For ease of description, the zeroth quantization kernel 1120 is described with reference to Figure 8 However, the scope of the present disclosure is not limited thereto, and the other quantization kernels 1121 to 112n can also have a structure similar to that of the zeroth quantization kernel 1120.

[0065] Reference Figure 8 , the quantization core 1120 may include an input re-formatter 1120a, an output re-formatter 1120b, and a conversion circuit 1120c. The input re-formatter 1120a may be configured to receive input data INPUT (e.g., from the polling switch 1110) and change the format of the received input data INPUT. In at least one embodiment, the input data INPUT may be the computation result data RST generated by the processing element 1200. The input data INPUT may include multiple computation result data RST. The number of multiple computation result data RST included in one input data INPUT may be the number of data that can be quantized by one quantization in the quantization core 1120.

[0066] The input re-formatter 1120a may include a first-in first-out unit FIFO, a transpose TRSP, a scalar-vector replication unit REPC, and a first register RGST1. The first-in first-out unit FIFO may be configured to perform first-in first-out on the input data INPUT received from the polling switch 1110. The transpose TRSP may be configured to compute the transpose of the input data INPUT. For example, when the input data INPUT is in the form of a vector consisting of 1 row and 4 columns, the transpose TRSP may perform a transpose on the input data INPUT to generate transposed data consisting of 4 rows and 1 column. The scalar-vector replication unit REPC may replicate the input data INPUT as a scalar value and convert it into a vector value. The first register RGST1 may be configured to store the values, data, and / or vectors generated by the input re-formatter 1120a and / or the intermediate values generated by the conversion circuit 1120c. Each of the first-in first-out unit FIFO, the transpose TRSP, the scalar-vector replication unit REPC, and / or the first register RGST1 may communicate with any or all of the other elements described with reference to the input re-formatter 1120a. For example, each of the first-in first-out unit FIFO, the transpose TRSP, the scalar-vector replication unit REPC, and / or the first register RGST1 may communicate with each other unidirectionally and / or bidirectionally and / or by broadcast to transmit and / or exchange and / or receive information, such as but not limited to data and / or commands. The information may be in various encoded formats, such as analog format and / or digital format.

[0067] The output re-formatter 1120b can be configured to store the results (e.g., quantized result data) converted by the conversion circuit 1120c, and output the conversion results (e.g., output data OUT) to the polling switch 1110. In at least one embodiment, the output data OUT may include multiple quantized data. Each of the multiple quantized data may be data in which multiple calculation result data RST are quantized.

[0068] The output re-formatter 1120b may include a second register RGST2 and an address selector ADDR. The second register RGST2 can be configured to store the output data OUT. The address selector ADDR can select and / or control the address of the second register RGST2 such that the output data OUT stored in the second register RGST2 is output to the polling switch 1110. In this way, the second register RGST2 can communicate with the address selector ADDR. For example, the second register RGST2 and the address selector ADDR can communicate unidirectionally and / or bidirectionally and / or by broadcast with each other to transmit and / or exchange and / or receive information such as, but not limited to, data and / or commands.

[0069] The conversion circuit 1120c can receive the input data INPUT and / or the transformed input data INPUT from the input re-formatter 1120a, and can perform various calculations on the transformed input data INPUT. In this case, the various calculations may include quantization calculations on the input data INPUT and / or the transformed input data INPUT. The intermediate data generated during the calculation of the conversion circuit 1120c can be stored in the first register RGST1 of the input re-formatter 1120a. When the conversion circuit 1120c completes quantization, the conversion circuit 1120c can provide the output data OUT to the output re-formatter 1120b.

[0070] In at least one embodiment, the conversion circuit 1120c may include various calculation modules to support various quantization algorithms. The conversion circuit 1120c can execute various calculation modules under the control of the control logic circuit 1130.

[0071] Figure 9 is a block diagram showing Figure 8 the conversion circuit. Referring to Figure 2 、 Figure 8 and Figure 9 , the conversion circuit 1120c may include various calculation modules to support various quantization algorithms. For example, the conversion circuit 1120c may include a symbol processing module 1120c-1, a scalar calculation module 1120c-2, a vector-scalar calculation module 1120c-3, and a vector-vector calculation module 1120c-4.

[0072] The symbol processing module 1120c-1 can be configured to manage the signs of the input data INPUT (or the data received from the input reformatter 1120a) and / or perform calculations related to signs. The symbol processing module 1120c-1 can include a symbol extractor SIGN-EXT, a sign inverter unit SIGN-INV, and an absolute value unit ABS.

[0073] The symbol extractor SIGN-EXT can be configured to extract the signs of the input data INPUT. For example, the input data INPUT can include multiple calculation result data RST. The symbol extractor SIGN-EXT can extract the signs of each of the multiple calculation result data RST included in the input data INPUT and can generate symbol data corresponding to the extracted signs. In at least one embodiment, the symbol data can have the same form as the input data INPUT (e.g., vector and / or scalar). For example, when the input data INPUT is in the vector form of [2.66 1.05 -0.07 0.65], the symbol data can have the vector form of [1 1 -1 1]. The above symbol data can each be represented by 1 bit (e.g., 1 when the corresponding data is positive and 0 when the corresponding data is negative).

[0074] The sign inverter unit SIGN-INV can be configured to invert the signs of the input data INPUT. For example, the sign inverter unit SIGN-INV can invert the signs of each of the multiple calculation result data RST included in the input data INPUT to generate inverted data. As an example, if the input data INPUT is in the vector form of [2.66 1.05 -0.07 0.65], then the inverted data can have the vector form of [-2.66 -1.05 0.07 -0.65].

[0075] The absolute value unit ABS can be configured to extract the absolute values of the input data INPUT. For example, the absolute value unit ABS can generate absolute value data by extracting the absolute values of each of the multiple calculation result data RST included in the input data INPUT. As an example, when the input data INPUT is in the vector form of [2.66 1.05 -0.07 0.65], the absolute value data can have the vector form of [2.66 1.05 0.07 0.65].

[0076] The scalar calculation module 1120c-2 can be configured to perform a scalar calculation on the input data INPUT. For example, the input data INPUT may have a vector form. In this case, the scalar calculation module 1120c-2 can be configured to perform a scalar calculation on one piece of data included in the input data INPUT. The scalar calculation module 1120c-2 may include a reciprocal unit RCP and a precision control unit PRC.

[0077] The reciprocal unit RCP can be configured to calculate the reciprocal of one piece of data included in the input data INPUT. The precision control unit PRC can change the precision of one piece of data included in the input data INPUT.

[0078] The vector-scalar calculation module 1120c-3 can be configured to perform a vector-scalar calculation on the input data INPUT. For example, the input data INPUT may have a vector form. In this case, the vector-scalar calculation module 1120c-3 can perform a vector-scalar calculation on the input data INPUT to generate data in scalar form.

[0079] The vector-scalar calculation module 1120c-3 may include an adder tree ADD1, a minimum finder MIN, and a maximum finder MAX. The adder tree ADD1 can be configured to perform addition on multiple pieces of data included in the input data INPUT to output addition data. For example, when the input data INPUT is [2.66 1.05 -0.07 0.65], the addition data may be 4.29. The minimum finder MIN can be configured to search for the minimum value among multiple pieces of data included in the input data INPUT to output minimum value data. For example, when the input data INPUT is [2.66 1.05 -0.07 0.65], the minimum value data may be -0.07. The maximum finder MAX can search for the maximum value among multiple pieces of data included in the input data INPUT to output maximum value data. For example, when the input data INPUT is [2.66 1.05 -0.07 0.65], the maximum value data may be 2.66.

[0080] The vector-vector calculation module 1120c-4 can be configured to perform a vector-vector calculation on the input data INPUT. For example, the input data INPUT may have a vector form. The vector-vector calculation module 1120c-4 can generate data in vector form by performing a vector-vector calculation on the input data INPUT and other data in vector form.

[0081] The vector-vector calculation module 1120c-4 may include an adder ADD2, a multiplier MUL, and a shifter SFT. The adder ADD2 may be configured to perform an addition calculation on two vector data to output addition data. For example, when the two vector data are [2.66 1.05 -0.07 0.65] and [1.11 1.11 -1.11 1.11], the addition data may be [3.77 2.16 -1.18 1.76].

[0082] The multiplier MUL may be configured to perform a multiplication calculation on two vector data. In this case, the multiplier MUL may perform an inner product and / or outer product calculation on the two vector data. Alternatively, the multiplier MUL may perform a scalar multiplication calculation on one vector data. As an example, when one vector data is [1 1 -1 1] and is multiplied by 1.11, the multiplication data may be [1.11 1.11 -1.11 1.11]. The shifter SFT may be configured to perform a shift calculation on the vector data.

[0083] As described above, the conversion circuit 1120c may perform various calculations to quantify the input data INPUT. Each element provided in the sign processing module 1120c-1, the scalar calculation module 1120c-2, the vector-scalar calculation module 1120c-3, and the vector-vector calculation module 1120c-4 may communicate with at least one of the other elements described with reference to the sign processing module 1120c-1, the scalar calculation module 1120c-2, the vector-scalar calculation module 1120c-3, and the vector-vector calculation module 1120c-4, respectively. For example, the sign extractor SIGN-EXT, the sign inverter unit SIGN-INV, and / or the absolute value unit ABS may perform one-way and / or two-way and / or broadcast communication with each other to transmit and / or exchange and / or receive information, and / or may perform one-way and / or two-way and / or broadcast communication with the elements included in the scalar calculation module 1120c-2, the vector-scalar calculation module 1120c-3, and / or the vector-vector calculation module 1120c-4. In at least one embodiment, the quantization kernel 1120 may perform quantization based on various quantization algorithms, and depending on the quantization algorithm being executed, the type and / or order of the calculation modules executed in the quantization kernel 1120 may vary. The control logic circuit 1130 may control each calculation module of the quantization kernel 1120 to adapt to the quantization algorithm being executed.

[0084] Figure 10 is a flowchart showing Figure 2 the operation of the accelerator. For ease of description, reference is made to Figure 10Describes an example of performing an artificial intelligence calculation once in accelerator 1000. However, the embodiments are not limited thereto. For example, accelerator 1000 can repeatedly and / or parallelly execute Figure 10 the operations of the flowchart for inference associated with specific data, and the result data and / or output data generated by the operations of Figure 10 the flowchart can be used as inputs in subsequent calculation processes (e.g., activation data ACT).

[0085] Refer to Figure 2 、 Figure 6 and Figure 10 , in operation S110, accelerator 1000 loads activation data ACT and weight data WT. For example, accelerator 1000 can load activation data ACT and weight data WT from memory 101.

[0086] In operation S120, accelerator 1000 performs a dequantization operation on activation data ACT and weight data WT. For example, the activation data ACT and weight data WT loaded from memory 101 can be of low-precision type LP. In contrast, the processing element 1200 of accelerator 1000 can perform MAC calculations based on high-precision type HP. Therefore, in at least some of these cases, the activation data ACT and weight data WT loaded from memory 101 can be converted to high-precision type HP. In at least one embodiment, the high-precision type HP can be BF16 type, FP16 type, and / or FP32 type.

[0087] However, in at least one embodiment, operation S120 can be omitted according to the calculation algorithm of processing element 1200. For example, as described below, when quantizer 1100 performs quantization based on the BCQ (binary-coded quantization) algorithm and processing element 1200 performs calculations based on BiQGEMM (non-generic matrix-to-matrix multiplication for binary-coded quantization neural networks), dequantization can be omitted. Therefore, calculations can be performed without separate conversion and / or dequantization of activation data ACT and weight data WT.

[0088] In operation S130, accelerator 1000 performs calculations on activation data ACT and weight data WT. For example, the processing element 1200 of accelerator 1000 can perform MAC calculations on activation data ACT and weight data WT to generate calculation result data RST. Operation S130 can also be referred to as performing calculations based on activation data ACT and weight data WT. In at least one embodiment, processing element 1200 performs MAC calculations based on high-precision type HP, so the calculation result data RST generated by processing element 1200 can be of high-precision type HP.

[0089] In operation S140, the accelerator 1000 may perform quantization on the computation result data RST. For example, the processing element 1200 may perform computations based on the high-precision type HP. In this case, the computation result data RST computed by the processing element 1200 may be of the high-precision type HP. The quantizer 1100 of the accelerator 1000 may perform quantization on the computation result data RST of the high-precision type HP and may convert the quantized data into output data OUT of the low-precision type LP. In at least one embodiment, the computation result data RST may be of the BF16 type, the FP16 type, and / or the FP32 type, and the output data OUT may be a combination of INT8 and / or INT8 and FP16. In this case, the total size and / or capacity of the output data OUT may be smaller than the total size and / or capacity of the computation result data RST.

[0090] In operation S150, the accelerator 1000 may store the quantized output data OUT in the memory 101. In at least one embodiment, the output data OUT stored in the memory 101 may be used as an input (e.g., activation data ACT) for subsequent computation operations of the accelerator 1000.

[0091] As described above, the accelerator 1000 according to at least one embodiment of the present disclosure may perform quantization on data (e.g., activation data ACT, weight data WT, and / or computation result data RST) generated during learning and / or inference processing. In this case, since the size and / or capacity of the data stored in and / or loaded from the memory 101 is reduced, the required bandwidth and / or required capacity regarding the memory 101 may be reduced, and the power consumption used in the memory 101 may be reduced. Therefore, an accelerator 1000 with reduced cost and improved performance is provided.

[0092] Figures 11A to 11C is a diagram for describing the quantization operation performed by the Figure 7 quantizer. For ease of description, the quantizer 1100 performs non-uniform quantization on 16 FP16-type data; however, as described above, the embodiments are not limited thereto. More specifically, the quantization technique described with reference to Figures 11A to 11C is only an example, and the scope of the present disclosure is not limited thereto. The quantizer 1100 of the accelerator 1000 according to the present disclosure may perform quantization based on various quantization algorithms.

[0093] First, with reference to Figure 2 、 Figure 7 、 Figure 8 、 Figure 9 、 Figure 11A 、 Figure 11B and Figure 11C, sixteen (16) result data can be generated by the computing operation of the processing element 1200. Each of the 16 result data can be of the FP16 type. The 16 result data can be input to the quantizer 1100. The polling switch 1110 of the quantizer 1100 can provide the 16 result data as the zero-th input data INPUT0 to the zero-th quantization kernel 1120.

[0094] First, referring to Figure 11A , the zero-th input data INPUT0 including 16 result data can be aligned in a vector form as shown in Figure 11A . For example, the scalar-vector replication unit REPC of the input re-formatter 1120a can align the 16 result data in a vector form like the zero-th input data INPUT0 shown in Figure 11A . The zero-th input data INPUT0 can be temporarily stored in the first register RGST1 of the input re-formatter 1120a.

[0095] After that, the conversion circuit 1120c can perform a quantization calculation on the zero-th input data INPUT0. As an example, the conversion circuit 1120c can perform an absolute value calculation and an average calculation on the zero-th input data INPUT0 in row units to generate the zero-th average data a0. For example, the absolute value unit ABS of the sign processing module 1120c-1 of the conversion circuit 1120c can perform an absolute value calculation on the zero-th input data INPUT0 to generate absolute value data. After that, the adder tree ADD1 of the vector-scalar calculation module 1120c-3 of the conversion circuit 1120c can perform an addition calculation on the absolute value data in row units to generate addition data. After that, the multiplier MUL of the vector-vector calculation module 1120c-4 of the conversion circuit 1120c can perform a division calculation on the addition data by the number of elements in each row of the zero-th input data INPUT0 to generate the average value of each row. The average value of each row can be stored in the first register RGST1 of the input re-formatter 1120a. The scalar-vector replication unit REPC of the input re-formatter 1120a can replicate the average value of each row to generate the zero-th average data a0 in vector form. The zero-th average data a0 can be stored in the first register RGST1 of the input re-formatter 1120a.

[0096] In at least one embodiment, the zero-th average data a0 can include four pieces of data. In this case, each of the four pieces of data included in the zero-th average data a0 can be temporarily stored in the first register RGST1 of the input re-formatter 1120a in the form of the FP16 type. Specifically, the zero-th average data a0 can have a size of 16 * 4 = 64 bits.

[0097] The conversion circuit 1120c can perform sign extraction on the zero-th input data INPUT0 to generate zero-th symbol data b0. For example, the sign extractor SIGN-EXT of the symbol processing module 1120c-1 of the conversion circuit 1120c can perform sign extraction on the zero-th input data INPUT0 to generate zero-th symbol data b0 in vector form.

[0098] In at least one embodiment, the zero-th symbol data b0 may include 16 pieces of data. In this case, since the 16 pieces of data of the zero-th symbol data b0 represent positive numbers and / or negative numbers, each of the 16 pieces of data may be 1 bit (e.g., 1 and / or 0), and may be temporarily stored in the first register RGST1 of the input re-formatter 1120a. Specifically, the zero-th symbol data b0 may have a size of 16 * 1 = 16 bits.

[0099] The conversion circuit 1120c can generate the a-th intermediate data INTa by performing multiplication on the zero-th average data a0 and the zero-th symbol data b0. For example, the multiplier MUL of the vector-vector calculation module 1120c-4 of the conversion circuit 1120c can perform scalar multiplication on the elements of the zero-th average data a0 and the rows of the zero-th symbol data b0 to generate the a-th intermediate data INTa. Specifically, the a-th intermediate data INTa is represented as the product of the zero-th average data a0 and the zero-th symbol data b0, so when the zero-th average data a0 and the zero-th symbol data b0 are stored in the first register RGST1 of the input re-formatter 1120a, the a-th intermediate data INTa can be generated.

[0100] Next, as Figure 11B shown, the conversion circuit 1120c can perform a subtraction operation on the zero-th input data INPUT0 and the a-th intermediate data INTa to generate the first intermediate data INT1. For example, the sign inverter SIGN-INV of the symbol processing module 1120c-1 of the conversion circuit 1120c can perform sign inversion on the a-th intermediate data INTa to generate inverted data. Then, the adder ADD2 of the vector-vector calculation module 1120c-4 of the conversion circuit 1120c can perform addition on the zero-th input data INPUT0 and the inverted data to generate the first intermediate data INT1.

[0101] After that, the conversion circuit 1120c can perform row-by-row absolute value calculation and average calculation on the first intermediate data INT1 to generate the first average data a1, and can generate the first symbol data b1 by performing sign extraction on the first intermediate data INT1.

[0102] As referenced Figure 11AAs described above, the per-line absolute value calculation and the average calculation can be performed by the absolute value unit ABS of the symbol processing module 1120c-1, the adder tree ADD1 of the vector-scalar calculation module 1120c-3, and the multiplier MUL of the vector-vector calculation module 1120c-4, and the symbol extraction operation can be performed by the symbol extractor SIGN-EXT of the symbol processing module 1120c-1. Its detailed description will be omitted to avoid redundancy.

[0103] Almost as in the reference Figure 11A As described above, the first average data a1 can include four pieces of data, and each of the four pieces of data can be temporarily stored in the first register RGST1 of the input reformatting unit 1120a in the form of FP16 type. Specifically, the first average data a1 can have a size of 16 * 4 = 64 bits.

[0104] Almost as in the reference Figure 11A As described above, the first symbol data b1 can include 16 pieces of data, and the 16 pieces of data represent positive numbers and / or negative numbers. Therefore, each of the 16 pieces of data can be 1 bit (e.g., 1 or 0) and can be temporarily stored in the first register RGST1 of the input reformatting unit 1120a. Specifically, the first symbol data b1 can have a size of 16 * 1 = 16 bits.

[0105] The conversion circuit 1120c can calculate the sum of the multiplication result of the zero-th average data a0 and the zero-th symbol data b0 (e.g., a0 * b0) and the multiplication result of the first average data a1 and the first symbol data b1 (e.g., a1 * b1) to generate the b-th intermediate data INTb. As in the reference Figure 11A As described above, the multiplication of each piece of data can be performed by the multiplier MUL of the vector-vector calculation module 1120c-4. The addition operation on the multiplication results (e.g., a0 * b0 + a1 * b1) can be performed by the adder ADD2 of the vector-vector calculation module 1120c-4.

[0106] As Figure 11B As shown, the b-th intermediate data INTb can be expressed as a0 * b0 + a1 * b1. Therefore, when the zero-th average data a0, the zero-th symbol data b0, the first average data a1, and the first symbol data b1 are stored in the first register RGST1 of the input reformatting unit 1120a, the b-th intermediate data INTb can be generated. Specifically, 16 * 4 + 1 * 16 + 16 * 4 + 1 * 16 = 160 bits can be used to represent the b-th intermediate data INTb.

[0107] Next, as Figure 11BAs shown, the conversion circuit 1120c can perform a subtraction operation on the zero - input data INPUT0 and the b - th intermediate data INTb to generate the second intermediate data INT2. After that, the conversion circuit 1120c can perform a per - row absolute - value calculation and an average calculation on the second intermediate data INT2 to generate the second average data a2, and can generate the second sign data b2 by performing sign extraction on the second intermediate data INT2.

[0108] As in the reference Figure 11A and 11B described, the subtraction operation can be performed by the sign - inversion unit SIGN - INV of the sign - processing module 1120c - 1 and the adder ADD2 of the vector - vector calculation module 1120c - 4. The per - row absolute - value calculation and the average calculation can be performed by the absolute - value unit ABS of the sign - processing module 1120c - 1, the adder tree ADD1 of the vector - scalar calculation module 1120c - 3, and the multiplier MUL of the vector - vector calculation module 1120c - 4. The sign - extraction operation can be performed by the sign - extractor SIGN - EXT of the sign - processing module 1120c - 1, and its additional description is omitted to avoid redundancy.

[0109] The conversion circuit 1120c can generate the output data OUT by performing an addition (e.g., a0*b0 + a1*b1 + a2*b2) on the multiplication of the zero - average data a0 and the zero - sign data b0 (e.g., a0*b0), the multiplication of the first average data a1 and the first sign data b1 (e.g., a1*b1), and the multiplication of the second average data a2 and the second sign data b2 (e.g., a2*b2).

[0110] As Figure 11C shown, the output data OUT is represented as a0*b0 + a1*b1 + a2*b2. Specifically, the output data OUT is represented by the zero - average data a0, the zero - sign data b0, the first average data a1, the first sign data b1, the second average data a2, and the second sign data b2. For this purpose, a total capacity of 3*(16*4 + 1*16)=240 bits is required.

[0111] In at least one embodiment, due to the repeated execution of the above calculations, the error compared to the zero - input data INPUT0 can be reduced. For example, the mean - square error MSEa of the zero - input data of the a - th intermediate data INTa can be greater than the mean - square error MSEb of the zero - input data of the b - th intermediate data INTb, and the mean - square error MSEb of the zero - input data of the b - th intermediate data INTb can be greater than the mean - square error MSEc of the zero - input data of the output data OUT.

[0112] In at least one embodiment, through the above quantization, compared with the zero-th input data INPUT0 (e.g., raw data), some errors occur in the output data OUT, but the total data capacity is reduced. For example, the zero-th input data INPUT0 includes 16 pieces of FP16 data. Specifically, the zero-th input data INPUT0 has a capacity of 16 * 16 = 256 bits. In contrast, when the above quantization operation is performed, the output data OUT has a capacity of 3 * (16 * 4 + 1 * 16) = 240 bits. Therefore, when the calculation results and / or activation data ACT generated during the inference process of the accelerator 1000 are quantized, the bandwidth and capacity required by the memory 101 can be reduced.

[0113] In at least one embodiment, Figures 11A to 11C The quantization method shown may be binary-coded quantization (BCQ). In this case, the output data OUT can be represented by the multiplication and addition of average data and sign data, and later, when the output data OUT is used as the input for subsequent artificial intelligence calculations, it can be simply calculated by BiQGEMM (binary-coded quantization neural network's non-general matrix-to-matrix multiplication). In at least one embodiment, the calculation result of BiQGEMM can have a high-precision type HP (e.g., BF16, FP16, FP32, etc.), and the quantizer 1100 can perform quantization in the same manner as described above for the calculation result.

[0114] Figure 12 is a block diagram showing Figure 2 of the accelerator. Referring to Figure 2 and Figure 12 , the accelerator 1000 may include a quantizer 1100, a unified buffer unit 1300, a plurality of processing elements PE11 to PE44, and an accumulator 1400.

[0115] The unified buffer unit 1300 may be configured to store various data, weights, and / or parameters for artificial intelligence calculations to be performed in the accelerator 1000. In at least one embodiment, the various data, weights, and / or parameters stored in the unified buffer unit 1300 may be provided from Figure 2 the memory 101 and / or may be obtained from the calculation results of the plurality of processing elements PE11 to PE44 of the accelerator 1000. In at least one embodiment, the information stored in the unified buffer unit 1300 may have a low-precision type LP.

[0116] Multiple processing elements PE11 to PE44 may perform artificial intelligence calculations and / or MAC calculations based on data provided by the unified buffer unit 1300. For example, each of the multiple processing elements PE11 to PE44 may receive activation data ACT and weight data WT from the unified buffer unit 1300, and may output partial sum data PSUM by performing artificial intelligence calculations and / or MAC calculations on the received activation data ACT and weight data WT.

[0117] The accumulator 1400 may be configured to accumulate the partial sum data PSUM of each of the multiple processing elements PE11 to PE44. The output of the accumulator 1400 may be provided as calculation result data RST to the quantizer 1100.

[0118] The quantizer 1100 may perform quantization on the calculation result data RST received from the accumulator 1400 to generate output data OUT. The output data OUT may be stored in the unified buffer unit 1300. In at least one embodiment, the output data OUT stored in the unified buffer unit 1300 may be reused as activation data ACT regarding the multiple processing elements PE11 to PE44.

[0119] In at least one embodiment, the multiple processing elements PE11 to PE44 and the accumulator 1400 may perform calculations based on a high-precision type HP (e.g., BF16, FP16, FP32, etc.). In this case, the partial sum data PSUM output from the multiple processing elements PE11 to PE44 and the calculation result data RST output from the accumulator 1400 will have the high-precision type HP. In contrast, the quantizer 1100 performs quantization on the calculation result data RST to generate output data OUT, and the generated output data OUT is stored in the unified buffer unit 1300. Specifically, the quantizer 1100 performs quantization on the intermediate data generated during the inference operation of the accelerator 1000. In this case, since a relatively small amount of data is stored in and / or output from the unified buffer unit 1300, the required capacity and required bandwidth of the unified buffer unit 1300 can be reduced, and the power consumption used by the unified buffer unit 1300 can be reduced.

[0120] Figure 13 is a diagram showing the structure of an accelerator according to at least one embodiment of the present disclosure. For the sake of brevity of the drawings and convenience of description, only some components related to at least one embodiment of the present disclosure are shown in Figure 13 Only some components related to at least one embodiment of the present disclosure are shown.

[0121] Refer to Figure 13, the accelerator 2000 may include a memory 2001, a quantizer 2100, and a processing element 2200. The memory 2001 may be configured to store various data, weights, parameters, etc. required for artificial intelligence computing.

[0122] The quantizer 2100 may be located in the path between the memory 2001 and the processing element 2200. The quantizer 2100 may perform dequantization on the data received from the memory 2001 and / or quantization on the data received from the processing element 2200. For example, the processing element 2200 may perform artificial intelligence computing based on the high-precision type HP. Specifically, the processing element 2200 may perform MAC calculations on the high-precision type HP data and output the result data of the high-precision type HP. In contrast, the data stored in the memory 2001 may have a low-precision type LP. Therefore, the quantizer 2100 may quantize the high-precision type HP data received from the processing element 2200 into low-precision type LP data. Alternatively, the quantizer 2100 may dequantize the low-precision type LP data received from the memory 2001 into high-precision type HP data.

[0123] In at least one embodiment, the quantizer 2100 may have the same and / or substantially similar structure as the quantizer 1100 described with reference Figures 7 to 9 and may be controlled in various ways according to various quantization and / or dequantization algorithms.

[0124] Figure 14 is a diagram showing the structure of an accelerator according to at least one embodiment of the present disclosure. For the sake of simplicity of the drawings and convenience of description, only some components related to at least one embodiment of the present disclosure are shown in Figure 14 .

[0125] Referring to Figure 14 , the accelerator 3000 may include a memory 3001, a quantizer 3100, a processing element 3200, and a dequantizer 3300. Since the memory 3001 and the processing element 3200 are the same as and / or substantially similar to those described with reference Figure 13 , additional description thereof is omitted to avoid redundancy.

[0126] The quantizer 3100 may quantize the high-precision type HP calculation result data RST generated by the processing element 3200 to generate the low-precision type LP output data OUT. The output data OUT may be stored in the memory 3001. The dequantizer 3300 may dequantize the low-precision type LP data received from the memory 3001 to generate high-precision type HP data (e.g., activation data and / or weights, etc.). The generated high-precision type data may be provided to the processing element 3200.

[0127] Referring toFigure 13 and 14 describe some structures of accelerators 2000 and 3000, but the scope of the present disclosure is not limited thereto. In at least one embodiment, data of low-precision type LP can be generated by a quantizer of the accelerator. In this case, a processing element of the accelerator can perform calculations on the low-precision type data to generate high-precision type result data. For example, the quantizer can generate low-precision type data based on BCQ, and the processing element of the accelerator can perform calculations on the low-precision type data based on BiQGEMM (non-general matrix-to-matrix multiplication for binary-coded quantized neural networks). In this case, separate dequantization may not be performed inside the accelerator.

[0128] Figure 15 is a block diagram showing the structure of an accelerator according to at least one embodiment of the present disclosure. Refer to Figure 15 , accelerator 4000 may include a memory 4001, a quantizer 4100, and a processing element 4200. Since the memory 4001, the quantizer 4100, and the processing element 4200 are the same as and / or substantially similar to those described with reference to Figure 2 , additional descriptions thereof are omitted to avoid redundancy.

[0129] In at least one embodiment, accelerator 4000 can perform calculations on multiple layers to perform learning and / or inference on an artificial intelligence model. In this case, different calculation methods can be applied according to the operation characteristics and / or reliability of each of the multiple layers. For example, for some of the multiple layers, the amount of calculation may be relatively large, and for some other layers, the amount of calculation may be relatively small. Alternatively, for some of the multiple layers, calculation accuracy may be important, and for some other layers, calculation speed may be important. In this case, according to the characteristics of the subsequent layer, accelerator 4000 can quantize the calculation result data RST for storage in memory 4001, and / or can omit quantization and can store the calculation result data RST in memory 4001.

[0130] For example, when the subsequent layer that accelerator 4000 is to calculate requires a large amount of calculation and a fast calculation speed, quantizer 4100 can quantize the calculation result data RST for storage in memory 4001. In this case, since the size of the data stored in memory 4001 is reduced, the memory can be accessed quickly. On the contrary, when the subsequent layer that accelerator 4000 is to calculate requires a small amount of calculation and accurate calculation, accelerator 4000 can omit quantization and can store the calculation result data RST (e.g., high-precision type HP) in memory 4001. In this case, since the data stored in memory 4001 has high precision, relatively accurate calculations can be performed.

[0131] Figure 16 is a block diagram showing a system according to at least one embodiment of the present disclosure. Referring to Figure 16 , system 5000 may include a first accelerator 5110, a second accelerator 5120, a memory 5200, and a controller 5300. Since the memory 5200 and the controller 5300 are the same as and / or substantially the same as the memory and the controller described in reference Figure 2 , their additional descriptions will be omitted to avoid redundancy.

[0132] System 5000 may be dedicated hardware configured to perform processing of an artificial intelligence model. In at least one embodiment, the first accelerator 5110 and the second accelerator 5120 of system 5000 may perform computations in parallel to process a large artificial intelligence model. For example, the first accelerator 5110 and the second accelerator 5120 may process a large artificial intelligence model in parallel and / or independently through a data parallel mechanism, a model parallel mechanism, and / or a tensor parallel mechanism. In at least one embodiment, the first accelerator 5110 and the second accelerator 5120 may operate based on the operation method and / or the computation method described in reference Figures 1 to 15 . For example, the first accelerator 5110 may include a first quantizer 5111, and the second accelerator 5210 may include a second quantizer 5121. Each of the first quantizer 5111 and the second quantizer 5121 may be the quantizer described in reference Figures 1 to 15 and / or may operate based on the method described in reference Figures 1 to 15 . For example, the first quantizer 5111 may quantize the result data generated by the artificial intelligence computation of the first accelerator 5110 to generate first output data, and the first output data may be stored in the memory 5200. The second quantizer 5121 may generate second output data by quantizing the result data generated by the artificial intelligence computation of the second accelerator 5120, and the second output data may be stored in the memory 5200. Each of the first and second output data stored in the memory 5200 may be provided to the first accelerator 5110 and / or the second accelerator 5120 for subsequent artificial intelligence computations.

[0133] Figure 17 is a block diagram showing a system according to at least one embodiment of the present disclosure. Referring to Figure 17 , system 6000 may include a first accelerator 6110, a second accelerator 6120, a memory 6200, a controller 6300, and a quantizer 6400. Since the memory 6200 and the controller 6300 are the same as and / or substantially the same as the memory and the controller described in reference Figure 2 , their additional descriptions will be omitted to avoid redundancy. Since reference Figure 16The first accelerator 6110, the second accelerator 6120, the memory 6200, and the controller 6300 are described, and thus additional descriptions thereof will be omitted to avoid redundancy.

[0134] In at least one embodiment, the system 6000 may include a quantizer 6400. The quantizer 6400 may perform quantization on the computation result data RST generated from the first accelerator 6110 and the second accelerator 6120 to generate output data. The generated output data may be stored in the memory 6200. In at least one embodiment, since the configuration, structure, and operation method of the quantizer 6400 are similar to the configuration, structure, and operation method described in the reference Figures 1 to 16 except that the quantizer 6400 is present outside the accelerators 6110 and 6120, additional descriptions thereof will be omitted to avoid redundancy.

[0135] As described above, according to the present disclosure, the accelerator may include a quantizer configured to quantize result data. In this case, the quantizer may perform quantization on the computation result data and / or computation intermediate data generated during artificial intelligence computation, learning, inference, etc. of the accelerator, thereby reducing the capacity of the data accessed from the memory. Therefore, the bandwidth and / or power consumption required for the memory can be reduced.

[0136] According to at least one embodiment of the present disclosure, the accelerator may perform computation on an artificial intelligence model. In this case, the accelerator may include a quantizer configured to perform quantization on various data generated during learning and / or inference of the accelerator. Therefore, since the capacity of various data (e.g., activation data, weight data, etc.) generated during learning and / or inference of the accelerator is reduced, the required bandwidth and required capacity of the memory configured to store and / or load various data can be reduced. Accordingly, an accelerator configured to perform artificial intelligence computation at a reduced cost and with improved performance, a method of operating the accelerator, and an artificial intelligence system including the accelerator are provided.

[0137] The above description is for implementing detailed embodiments of the present disclosure. Embodiments that simply change the design and / or can be easily changed in design, as well as the above at least one embodiment, may be included in the present disclosure. Additionally, technologies that can be easily changed and implemented by using the above embodiments may be included in the present disclosure. Therefore, the scope of the present disclosure should not be limited to the above embodiments, and should be defined not only by the claims described later, but also by those equivalent to the claims of the present disclosure.

Claims

1. An accelerator configured to perform artificial intelligence (AI) operations, the accelerator comprising: A processing element configured to generate first result data by performing a first operation based on first activation data and first weight data loaded from a memory; And A quantizer configured to generate first output data by performing quantization on the first result data, and Wherein the first activation data, the first weight data, and the first output data are of low-precision type, and the first result data is of high-precision type, and Wherein the accelerator is configured such that the first output data can be stored in the memory.

2. The accelerator according to claim 1, wherein, The size of the first output data is smaller than the size of the first result data.

3. The accelerator according to claim 1, wherein, The high-precision type includes at least one of brain floating-point format BF16 type, half-precision IEEE floating-point format FP16 type, single-precision floating-point format FP32, or double-precision floating-point format FP64 type, and Wherein the low-precision type includes at least one of an integer type with width 4 (INT4 type), an integer type with width 8 (INT8 type), or an integer type with width 16 (INT16 type).

4. The accelerator according to claim 1, wherein, The first operation includes a multiply-accumulate (MAC) operation on the first activation data and the first weight data.

5. The accelerator according to claim 1, wherein, The quantizer includes: A polling switch configured to receive the first result data; A plurality of quantization cores configured to generate the first output data by performing the quantization on the first result data received from the polling switch; and Control logic circuitry configured to control each quantization core among the plurality of quantization cores, and Wherein the polling switch is further configured to send the first output data generated by the plurality of quantization cores to the memory.

6. The accelerator according to claim 5, wherein The plurality of quantization cores are configured to perform the quantization in parallel.

7. The accelerator according to claim 5, wherein, Each quantization core among the plurality of quantization cores includes: An input re-formatter configured to change the format of the first result data received from the polling switch and store an intermediate result; A conversion circuit configured to generate the first result data by performing calculations on input data received from the input re-formatter; and An output re-formatter configured to store the first result data generated from the conversion circuit and output the first result data to the polling switch.

8. The accelerator according to claim 7, wherein, The conversion circuit includes a processing circuit configured to: Manage the sign of the input data; Perform scalar calculations on the input data; Perform vector-scalar calculations on the input data; and Perform vector-vector calculations on the input data.

9. The accelerator according to claim 8, wherein, The control logic circuitry is configured to sequentially control the input re-formatter, the output re-formatter, and the conversion circuit of each quantization core among the plurality of quantization cores based on a quantization algorithm to be performed in each quantization core among the plurality of quantization cores.

10. The accelerator according to claim 1, wherein, The quantizer is configured to perform the quantization according to binary-coded quantization (BCQ).

11. The accelerator according to claim 1, wherein, The processing element is further configured to generate second result data by performing a second calculation on the first output data and second weight data loaded from the memory.

12. The accelerator according to claim 11, wherein, The quantizer is further configured to perform the quantization on the second result data to generate second output data, and wherein the accelerator is configured to enable the second output data to be stored in the memory.

13. The accelerator according to claim 11, wherein, The second result data is stored in the memory without performing the quantization on the second result data.

14. A method of operating an accelerator configured to perform artificial intelligence (AI) operations, the method comprising: Loading first activation data and first weight data from a memory; Generating first result data by performing a first operation based on the first activation data and the first weight data; Performing quantization on the first result data to generate first output data; And Storing the first output data in the memory, and wherein the first activation data, the first weight data, and the first output data are of a low-precision type, and the first result data is of a high-precision type.

15. The method according to claim 14, wherein The size of the first output data is smaller than the size of the first result data.

16. The method according to claim 14, wherein, The high-precision type includes at least one of a brain floating-point format BF16 type, a half-precision IEEE floating-point format FP16 type, a single-precision floating-point format FP32, or a double-precision floating-point format FP64 type, and wherein the low-precision type includes at least one of an integer type with a width of 4 (INT4 type), an integer type with a width of 8 (INT8 type), or an integer type with a width of 16 (INT16 type).

17. The method according to claim 14, further comprising: Loading the first output data and second weight data from the memory; Generating second result data by performing a second operation based on the first output data and the second weight data; Generating second output data by performing the quantization on the second result data; And Storing the second output data in the memory, wherein the second output data is of the low-precision type, and the second result data is of the high-precision type.

18. An artificial intelligence system, comprising: A memory configured to store first activation data and first weight data; An accelerator configured to load the first activation data and the first weight data from the memory to generate first result data by performing a first operation based on the first activation data and the first weight data, and to generate first output data by performing quantization on the first result data; And A central processing unit CPU configured to control the memory and the accelerator, and wherein the first activation data, the first weight data, and the first output data are of a low-precision type, and the first result data is of a high-precision type, and wherein the first output data is stored in the memory.

19. The artificial intelligence system according to claim 18, wherein, The size of the first output data is smaller than the size of the first result data.

20. The artificial intelligence system according to claim 18, wherein, The CPU is further configured to quantize the first weight data.

Images