Low-power, low-memory multiply and accumulate (MAC) unit

Abstract

The present application relates to technologies related to a low-power, low-memory multiply and accumulate (MAC) unit. In an example, the MAC unit performs a MAC operation, which represents the multiplication of numbers. The bit representation of at least one number is compressed based on quantization and clustering of quantized values, thereby using index bits instead of the actual bit representation. The index bits are loaded into an index buffer, and the bit representation of another number is loaded into an input buffer. The index bits are used for lookups to determine whether to apply the corresponding bit representation and shift operation to the input buffer based on the bit representation, and then perform an accumulation operation.

Description

Technical Field

[0001] The present application relates to technology related to low-power, low-memory multiply and accumulate (MAC) units. Background Art

[0002] At the hardware level of the processor, the value of a variable can be quantized and represented by a bit sequence. Further, a multiply and accumulate (MAC) operation can be used based on the bit representation.

[0003] In deep neural network (DNN) applications (e.g., image processing, video recognition), quantization can be used to represent the values ​​used by the DNN. Typically, the most computationally intensive operation in DNN training and inference is the MAC operation. In computing, especially at the digital signal processor (DSP) level, a MAC operation can calculate the product of two numbers and add the product to an accumulator. The hardware unit that performs this operation is called a MAC unit. The MAC operation modifies the accumulator a: a←a+(b*c). When floating-point numbers are used in the MAC operation, rounding may be performed twice (common in many DSPs) or once. DNNs can use millions to billions of floating-point weights, and therefore floating-point MAC operations consume a lot of power. This becomes a critical issue when DNNs are deployed to edge devices or Internet of Things (IoT) devices with limited memory footprint and power budget, such as mobile phones, solid-state drives (SSDs), or cameras. Summary of the Invention

[0004] Techniques and systems are described that involve low-power, low-memory MAC units that perform MAC operations. The MAC operations can be used for, but are not limited to, DNN training and / or DNN inference.

[0005] In one example, a system implements a method. The method includes determining that a first number is to be multiplied by a second number, and performing a MAC operation to multiply the first number by the second number. Performing the MAC operation includes determining an index of a quantization level associated with the first number, wherein the index is represented by "m" bits, wherein the quantization level is represented by "l" bits, and wherein "l" is greater than "m". Performing the MAC operation also includes determining "r" groups of "k" bits (i.e., "r" groups, each group having "k" bits) by using at least the index in a lookup table, wherein "r*k" equals "l", and wherein the "l" bits include the "r" groups of "k" bits. Performing the MAC operation also includes using the "r" groups of "k" bits in a shift and accumulate operation applied to an input buffer that stores a bit representation of the second number.

[0006] This illustrative example is not described to limit or define the present disclosure, but rather to provide an example to aid understanding. Additional embodiments and examples are discussed in the detailed description, and further description is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] An understanding of the nature and advantages of various embodiments may be achieved by reference to the following drawings.

[0008] Figure 1 An example of a processor implementing an artificial intelligence (AI) inference engine is shown, according to certain embodiments of the present disclosure.

[0009] Figure 2 An example of an AI model that can be used as an AI reasoning engine during training according to certain embodiments of the present disclosure is shown.

[0010] Figure 3 An example of a process for performing a compression-based MAC operation according to certain embodiments of the present disclosure is shown.

[0011] Figure 4 Examples of quantization values ​​according to certain embodiments of the present disclosure are shown.

[0012] Figure 5 An example of target weight distribution according to certain embodiments of the present disclosure is shown.

[0013] Figure 6 An example of clustering of quantized values ​​based on target weight distribution according to certain embodiments of the present disclosure is shown.

[0014] Figure 7 An example of a MAC unit according to certain embodiments of the present disclosure is shown.

[0015] Figure 8 An exemplary process of performing a MAC operation by a MAC unit according to a specific embodiment of the present disclosure is shown.

[0016] Figure 9 Another exemplary process of performing a MAC operation by a MAC unit according to a specific embodiment of the present disclosure is shown.

[0017] Figure 10 Another exemplary process of compressing quantization levels according to certain embodiments of the present disclosure is shown.

[0018] Figure 11 An example of a computing system based on compressed MAC operations according to certain embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] Techniques related to a low-power, low-memory MAC unit are described. In an example, a device determines that a multiplication of two numbers should be performed. At the hardware level, a MAC operation is performed by the MAC unit based on bit representations of the two numbers. At least a first one of the numbers may be quantized to generate a first bit representation corresponding to a quantization level of the first number. The first bit representation is associated with an index of the quantization level, where the index has a relatively small number of bits. In other words, the index represents compression of the first bit representation. Quantization may be performed using flexible power-of-two (FPoT) non-uniform quantization to allow for a target distribution of quantization levels. Instead of storing the first bit representation, the index is stored as the corresponding quantization of the first number. During the MAC operation, a first register is used, whereby the size of the first register corresponds to the number of bits of the index. A second register is used for a second bit representation of the second number. Decompression is performed by determining the first bit representation based on the index. A portion of the first bit representation is input to the first register to perform a shift operation, followed by an accumulation operation on the second register. The shift operation and accumulation operation are repeated for each remaining portion of the first bit representation. Hereinafter, the result of the accumulation operation at least partially represents the multiplication of the two numbers. When using flexible PoT non-uniform quantization, the target distribution of quantization levels can be achieved to substantially cover the target dynamic range. In addition, because indexes are used instead of full bit representations, memory storage space can be saved. Furthermore, because the MAC operation uses registers with a smaller number of bits (e.g., corresponding to the index), the MAC unit can be more computationally efficient. Therefore, in summary, the MAC unit can not only have low power (due to computational efficiency) and low memory (due to memory storage efficiency), but also substantially cover the target dynamic range.

[0020] To illustrate, consider the example of a DNN inference engine implemented on a device for image processing. This engine uses nonlinear transformations and multiplies variables by weights, each of which corresponds to a node in a DNN layer. Depending on the structure of the DNN, the weights can number in the millions or even billions. The weights are quantized using a 16-bit floating-point representation. Furthermore, the 16-bit representation is compressed into a 3-bit index that represents the quantization level of the weight from one of eight quantization levels. A table is generated that associates the 3-bit index for each of the eight quantization levels with the corresponding 16-bit floating-point representation. Thus, instead of storing 16-bit floating-point representations of millions (or billions) of weights, millions (or billions) of 3-bit indices and tables are stored. To multiply a variable by a weight, the input buffer of the MAC unit stores the bit representation of the variable, and the weight buffer of the MAC unit stores the three-bit index. The three bits in the weight buffer are used to look up the table to determine the corresponding 16-bit floating-point representation. Based on portions of the 16-bit floating-point representation, shift and accumulate operations are applied to the input buffer, the result of which at least partially represents the multiplication of the variable by the weight.

[0021] For clarity of explanation, the embodiments of the present disclosure are described in conjunction with DNNs. However, the embodiments are not limited thereto and are similarly applicable to any other type of artificial intelligence (AI) model or any multiplication of numbers that can be implemented, at least in part, in hardware using MAC units.

[0022] Figure 1 An example of a processor 110 implementing an AI inference engine 120 according to a specific embodiment of the present disclosure is shown. The AI ​​inference engine can be an AI model that has been trained to perform a specific task. For example, the AI ​​model is a DNN trained for an image processing task, a video task, a speech processing task, or any other task. In this case, the AI ​​inference engine 120 is a DNN inference engine. The application 130 of the processor 110 can input data to the AI ​​inference engine 120 for the task by calling the AI ​​inference engine 120, and can receive output data from the AI ​​inference engine 120 indicating the results of performing the task. Performing a task via the AI ​​inference engine 120 may involve a substantial number of transformations, each of which may include the multiplication of one or more numbers. For example, in the case of a DNN inference engine, the transformations include nonlinear transformations between nodes in different layers of the DNN, each of which multiplies a variable by the weight of a connection or edge between two nodes. Some or all of the multiplications can be performed using MAC operations 140. For example, the AI ​​inference engine 120 includes one or more MAC units at the hardware level (e.g., DSP level) to perform MAC operations.

[0023] In an example, the processor 110 is implemented as a system on a chip (SoC) of a device. The device may be a mobile device such as a mobile phone or a wearable device, an IoT device, an edge device, or any other device suitable for hosting the AI ​​inference engine 120.

[0024] The AI ​​reasoning engine 120 can be trained offline on different devices, in which case the two devices together represent a system. Additionally or alternatively, the AI ​​reasoning engine 120 can be trained on the same device, or trained offline and further trained on that device. Once trained, the AI ​​model can be represented by a hardware configuration on the processor 110. For example, in the case of a DNN, the hardware configuration can represent the weights of the edges between nodes. The application 130 can represent program code implemented in software and / or firmware and can rely on the reasoning of the AI ​​reasoning engine 120 to process various data (e.g., images, video, audio).

[0025] Figure 2 An example of an AI model that can be used as an AI reasoning engine, such as AI reasoning engine 120, when trained according to certain embodiments of the present disclosure is shown. Figure 2 In the illustrative example of , the AI ​​model 200 is a neural network 200 such as a DNN, but other types of AI models are possible. A feature map 202 representing input data of an application such as application 130 is input to the neural network 200. In turn, the neural network 200 outputs a result 204 of performing a task (e.g., image classification, object detection, probability indication). Although a single result corresponding to a single task is shown, the output may include multiple results depending on the structure of the neural network. Further, although the feature map 202 is shown as an input, the input data can be directly input to the input layer of the neural network 200, where the input layer can generate the feature map 202.

[0026] As shown, neural network 200 includes multiple layers. Features from feature map 202 are connected to input nodes in input layer 210 of neural network 200. Results 204 are generated from output nodes of output layer 230. One or more hidden layers 220 of neural network 200 are present between input layer 210 and output layer 230. Neural network 200 is pre-trained to process features from feature map 202 through different layers 210, 220, and 230 to output results 204.

[0027] In some embodiments, neural network 200 is a multi-layer neural network, representing a network of interconnected nodes, such as an artificial intelligence (DNN), in which knowledge about the nodes (e.g., information about the specific features represented by the nodes) is shared between layers, while also retaining knowledge specific to each layer. Each node represents a piece of information. Knowledge can be exchanged between nodes through node-to-node interconnections. Input to neural network 200 activates a group of nodes. In turn, this group of nodes activates other nodes, thereby propagating knowledge about the input. This activation process is repeated for other nodes until a node in the output layer is selected and activated.

[0028] As shown, neural network 200 includes a hierarchy of layers, which represent a hierarchy of nodes interconnected in a feedforward manner. Input layer 210 exists at the lowest level and includes a set of nodes referred to herein as input nodes. When feature map 202 is input to neural network 200, each of the input nodes of input layer 210 is connected to each feature of the feature map. Each of the connections has a weight. These weights are a set of parameters obtained from training neural network 200. Input nodes transform features by applying an activation function to these features. The information obtained from this transformation is passed to nodes at higher levels in the hierarchy.

[0029] The output layer 230 exists at the highest hierarchical level and includes one or more output nodes. Each output node provides a specific result 204. The hidden layer 220 exists between the input layer 210 and the output layer 230. The hidden layer 220 includes "N" hidden layers, where "N" is an integer greater than or equal to 1. In turn, each of the hidden layers also includes a set of nodes referred to herein as hidden nodes. Exemplary hidden layers include upsampling, convolution, fully connected layers, and data transformation layers.

[0030] At the lowest level of hidden layer 220, the hidden nodes of that layer are interconnected with the input nodes. At the highest level of hidden layer 220, the hidden nodes of that level are interconnected with the output nodes. Input nodes are not directly interconnected with output nodes. If there are multiple hidden layers, the input nodes are interconnected with the hidden nodes of the lowest hidden layer. These hidden nodes, in turn, are interconnected with the hidden nodes of the next hidden layer, and so on.

[0031] An interconnection represents a piece of information learned about two interconnected nodes. The interconnections have numerical weights 240 that can be adjusted (e.g., based on a training data set) to enable the neural network 200 to adapt to the input and enable learning.

[0032] Typically, the hidden layer 220 allows knowledge about the input nodes of the input layer 210 to be shared among the output nodes of the output layer 230. To this end, a transformation f is applied to the input node by the hidden layer 220, thereby transforming the input X into the output Y. Typically, the transformation includes multiplying the variable represented by the node by the weight of the connection between the node and the node at the next level. The transformation can also be nonlinear. For example, at the first level, x∈X is the input. The first node of the level is connected to the second node of the next level, where the connection has a weight w1. x is transformed at the first node, where y1=w1×x. At the next level, a nonlinear transformation is used at the second node, where y2=f(y1+b1), and b1 is a bias term. Furthermore, the second node is connected to the third node at the next level. The transformation between the two layers is set to y3=w2×y2, where w2 is the weight between the two nodes.

[0033] In the example, the input signal x is usually in 32-bit or 64-bit floating point format (FLP-32 or FLP-64). The edge weight matrices W1 and W2 and the bias terms b1 and b2 are also in FLP-32 or FLP-64 format. The floating point multiplications of w1×x and w2×y2 consume most of the power. f(.) is an activation function such as ReLu function. Figure 5 As shown, the weights in W1 and W2 typically follow a highly non-uniform probability distribution.

[0034] During training, the neural network 200 also uses a loss function l (also known as a cost function c) to find an optimal solution. An optimal solution is one where no solution has a loss less than the optimal solution's loss. In this example, the loss function comprises a mean squared error function, which minimizes the average squared error between the output f(x) and the target value y for all example pairs (x, y). A backpropagation algorithm that uses gradient descent to minimize the loss function is used to train the neural network 200.

[0035] In this way, the hidden layer 220 retains knowledge about the input nodes. Based on the interconnection of the hidden nodes with the input nodes, a set of knowledge is shared between the output nodes.

[0036] Figure 3 An example of a process for performing a compression-based MAC operation according to a specific embodiment of the present disclosure is shown. The process may be performed by one or more processors (such as Figure 1 The process is also combined with the weights of the AI ​​reasoning engine (such as Figure 2 200). However, the process is not limited thereto and is similarly applicable to MAC operations on bit representations of numbers to be multiplied. As described below, the operations of the process can be performed by the system. Some or all of the instructions for performing the operations can be implemented as hardware circuits and / or stored as computer-readable instructions on a non-transitory computer-readable medium of the system. When implemented, the instructions represent a module including circuits or code that can be executed by the processor of the system. The use of such instructions configures the system to perform the specific operations described herein. Each circuit or code combined with an associated processor represents a means for performing the corresponding operation. Although the operations are shown in a particular order, it should be understood that no particular order is required and that one or more operations can be omitted, skipped, performed in parallel, and / or reordered.

[0037] As shown, the process begins at operation 302, where the system quantizes the weights of the AI ​​inference engine. Weights are examples of numbers used in multiplication. There may be different quantization schemes. In a first example, uniform quantization is used and is represented as Where α is the scaling factor and b is the bit resolution (e.g., the target number of bits). In the second example, PoT quantization is used and expressed as Where α is the scaling factor and b is the bit resolution. In the third example, adaptive power-of-two (APoT) quantization is used and expressed as in Where γ is a scaling factor, and k and n are integers. In a fourth example, FPoT quantization is used and the FPoT quantization is expressed as in β is a scaling factor, k and r are integers, and k×r is the bit resolution. The output of any of these quantization examples is a quantized value for each weight, where the quantized value can be represented by a bit representation such as a floating point representation or a fixed point representation. Compared to average quantization, when PoT quantization is implemented in hardware, better power and memory performance can be provided because it can be implemented by shifting in registers. Compared to PoT, APoT can provide a higher resolution of quantized values. Compared to APoT, as further described in the next figure, because FPoT allows for a better distribution of quantized values ​​within a target dynamic range and therefore allows for greater flexibility in clustering quantized values ​​into clusters according to a target weight distribution, power and memory performance can be further improved.

[0038] In operation 304, the system compresses the quantized weights. Instead of using a bit representation of the quantized value (floating point or fixed point), an index is used, where the index has a smaller number of bits and is associated with the bit representation of the quantization level in a table (e.g., a compression table or codebook). Figure 10 An example of this compression process is further described in . In short, the quantized values ​​are clustered into "Q" groups, where "Q" is an integer and where each group represents a quantization level and is associated with a bit representation of that quantization level. The clustering can take into account the performance of the AI ​​model (e.g., minor adjustments can be made so that performance is not significantly affected). When using FPoT quantization, the clustering can also take into account the target weight distribution (e.g., clusters can be defined to achieve that distribution). The number of bits used for indexing is a function of "Q" (e.g., number of bits = 2 Q ). Thus, instead of storing a bit representation of each of the quantized values, an index of the quantization level grouping the quantized values ​​is stored, wherein the index has a smaller number of bits than the bit representation and is associated therewith in a table.

[0039] At operation 306, the system performs a MAC operation based on the compressed weights. Referring to the AI ​​inference engine, a MAC operation can be performed to multiply the variable by the weight as part of the transform function. The weight index is used, and the bits of the index are loaded into the weight buffer. These bits are then looked up to determine the corresponding bit representation of the quantization level. The bits of the variable are loaded into the input buffer and shifted and accumulated based on the bit representation. Figure 9 Examples of MAC operations are further described in .

[0040] Figure 4An example of a quantized value 400 according to a specific embodiment of the present disclosure is shown. Here, FPoT quantization is used to generate the quantized value. The horizontal axis represents the floating point input. The vertical axis represents the floating point output of each FPoT quantization. Figure 4 16-bit resolution is shown in FIG, where K = 4 and r = 4. Each point in the curve represents a quantization value. As shown in the figure, the quantization value 400 has a fairly good distribution over the entire dynamic range. In contrast, if APoT or PoT quantization is used, the distribution of APoT or PoT quantization is sparser at the ends and denser in the center. Therefore, as in Figure 5 and Figure 6 As further described in , FPoT quantization allows for improved clustering of quantized values ​​to achieve a target distribution.

[0041] Figure 5 An example of target weight distribution according to a specific embodiment of the present disclosure is shown. Figure 2 The AI ​​model of the neural network model 200 corresponds to the first weight distribution 510 of the weights before clipping. On the left, a first weight distribution 510 of the weights before clipping is shown. On the right, a second weight distribution 520 of the weights after clipping is shown. The horizontal axis represents the value of the weight. The vertical axis represents the count of the weight. Each of the weights can be quantized into a quantized value. Therefore, the first weight distribution 510 and the second weight distribution 520 can similarly represent the distribution of quantized values ​​before and after clipping. Figure 5 As shown in the example of , the first weight distribution 510 and the second weight distribution 520 are highly non-uniform probability distributions. When FPoT quantization is used, clustering of the quantization levels can be performed to generate clusters having a target distribution that is set to be the same as or close to the first weight distribution 510 (before clipping) or the second weight distribution 520 (after clipping).

[0042] Figure 6 An example of clustering 600 of quantized values ​​based on target weight distribution according to certain embodiments of the present disclosure is shown. Here, the quantized values ​​are Figure 4 Cluster 600 includes eight groups (eg, Q=8), each of which is indexed with an index between zero and seven (eg, Q i , where i = {0, 1, 2, ..., 7}). Numbers other than eight are possible and are provided herein for illustrative purposes. Each group Q i Indicates the quantization level. Each group Q i The size of group Q i The number of quantized values ​​included in the group Q is set to achieve the target weight distribution (or, similarly, the target weight quantized value distribution). i The sizes between them can be different, so that each group Q iThe total count of quantized values ​​in can follow the target weight distribution (e.g., as Figure 5 As shown in , it follows a non-uniform probability distribution before or after clipping). In the example, as Figure 10 As further described in , k-means clustering iteratively generates groups Q i , and the number of these groups can be varied to meet the target performance.

[0043] In addition, each cluster can be composed of statistical points (such as Figure 6 The statistical point may be defined in different ways, including, for example, the center or the mean. The statistical point corresponds to a quantized value represented by a bit. Therefore, the group Q i All quantitative values ​​of the clusters are grouped by Q i The index "i" of each group and the bit representation of the statistical point (and / or similarly, the quantized value of the statistical point) are stored in a table, thereby associating the index "i" with the bit representation (and / or quantized value). During the MAC operation, the bit of the index can be used in a table lookup to then determine the corresponding group Q. i and bit representation.

[0044] Figure 7 An example of a MAC unit 700 according to certain embodiments of the present disclosure is shown. In the example, the MAC unit 700 includes an index buffer 702, an input buffer 704, a table 706, an accumulator 708, and a scaling factor 710. When used in conjunction with an AI inference engine, the index buffer 702 may also be referred to herein as a "weight buffer."

[0045] In an example, the index buffer 702 is used to load bits of a first number to be multiplied by a second number (e.g., an input number). The bits may correspond to a compressed representation of the first number and are referred to herein as index bits. For example, and referring back to the AI ​​inference engine, the first number is a weight, and the index bits are an index of a quantization level (e.g., a quantization group Q i The index buffer 702 may also store a bit representing the sign of the first number (in Figure 7 , where “S” can be set to “1” for a positive number and “0” for a negative number, and vice versa).

[0046] The input buffer 704 is used to load the bits of the second number. The second number can be a floating point representation or a fixed point representation. The bits are referred to herein as input bits. Typically, the input buffer 704 is larger than the index buffer 702. The input buffer 704 can also store bits representing the sign of the second number (in Figure 7 , where “S” can be set to “1” for a positive number and “0” for a negative number, and vice versa).

[0047] Table 706 may be a compressed table or codebook that associates the bits of an index with the bit representation of a quantization level. For example, each index in table 706 is a quantization group Q i The index "i" of the quantization group Q i is the quantization level. Table 706 associates the index “i” with the quantization group Q i The bit representation of the statistical point (and / or the actual quantized value of the statistical point) is associated with the bit representation of the statistical point.

[0048] The accumulator 708 may include an accumulator buffer for storing intermediate results. The intermediate result may be a shift of the input bits in the input buffer 704 based on the index bits in the index buffer 702. The intermediate result may also take into account the signs of the first number and the second number.

[0049] Scaling factor 710 may be a factor by which the output of accumulator 708 is scaled. This factor may depend on the type of quantization. For example, for uniform or PoT quantization, scaling factor 710 is α. For APoT quantization, scaling factor 710 is γ. For FPoT quantization, scaling factor 710 is β.

[0050] In operation, an index bit and a sign bit are loaded into an index buffer 702. An input bit and a sign bit are loaded into an input buffer 704. The index bit is used to look up a table 706 and determine a corresponding bit representation. Based on the bit representation, a set of shifts (e.g., left shifts) of the input bit are performed, with an accumulator 708 performing an accumulation operation after each shift. An XOR operation is performed by the accumulator 708 using the sign bits from the index buffer 702 and the input buffer 704. Hereinafter, the output of the accumulator 708 is scaled by a scaling factor 710. The scaled output corresponds to the bit representation of the multiplication of the first number and the second number.

[0051] Figure 3 An example of a process for performing operations based on compression of a bit representation of a quantized value is shown. The process may be performed by a processor including one or more processors such as Figure 1 Also combined with the weights of the AI ​​reasoning engine (such as Figure 2The process is illustrated using the weights of the neural network 200 of FIG. However, the process is not limited thereto and is similarly applicable to MAC operations on bit representations of numbers to be multiplied. As described below, the operations of the process can be performed by a system. Some or all of the instructions for performing the operations can be implemented as hardware circuits and / or stored as computer-readable instructions on a non-transitory computer-readable medium of the system. When implemented, the instructions represent a module including circuits or code that can be executed by a processor of the system. The use of such instructions configures the system to perform the specific operations described herein. Each circuit or code combined with an associated processor represents a means for performing the corresponding operation. Although the operations are shown in a particular order, it should be understood that no particular order is required and that one or more operations can be omitted, skipped, performed in parallel, and / or reordered.

[0052] Figure 8 An exemplary process for performing a MAC operation by a MAC unit according to certain embodiments of the present disclosure is shown. As shown, the process may begin at operation 802, where the system determines that a first number is to be multiplied by a second number. For example, the system includes an AI inference engine. In this example, the first number is a weight of an AI model implemented by the AI ​​inference engine. The second number is a variable of a transformation function of the AI ​​model, where the transformation function includes multiplying the variable by the weight.

[0053] At operation 804, the system performs a MAC operation to multiply the first number by the second number. The MAC operation can be performed using a bit representation and can include a set of shift operations, each followed by an accumulation operation. In an example, at least the bit representation of the first number is a compressed representation. An index is used as input to the index buffer instead of a bit representation. The index is associated with a set of q quantization levels (e.g., a "set Q"). i The index bits are then used in a table lookup to determine the bit representation of the quantization level, where a shift operation is performed based on the bit representation.

[0054] In an example, the quantization of the first number is FPoT quantization, which obtains a quantized value of the first number and an associated bit representation. Figure 8 As shown, performing a MAC operation 804 may include a plurality of sub-operations 810 through 830 .

[0055] At a first sub-operation 810, the system determines an index n of a quantization level associated with a first number, wherein the index is represented by "m" bits, wherein the quantization level is represented by "l" bits, and wherein "l" is greater than "m". For example, "m" bits are used to represent a group Q for clustering the quantized values ​​of the weights of the AI ​​model. i Although it can be a fixed-point format, "l" can also be 32 or 64 and can correspond to the FLP-32 or FLP-64 format.

[0056] In a second sub-operation 820, the system determines "r" groups of "k" bits by using at least the index in a table lookup, where "r*k" equals "1", and where "1" bits comprise "r" groups of "k" bits. "r" and "k" can be such that Figure 3 The parameters of the FPoT quantization described in . The "1" bit representation of the first number can be divided into "r" groups, each of which is "k" bits long.

[0057] In a third sub-operation 830, the system uses "r" groups of "k" bits in a shift and accumulate operation applied to an input buffer that stores a bit representation of the second number. The bit representation can be a floating point representation or a fixed point representation, and the bit representation can be uncompressed (although it can be compressed, in which case another table can be used to determine the bit representation and load it into the input buffer of the MAC unit). Each shift operation shifts the input bits to the left by a group "r" of "k" bits (for example, if "r" equals four and "k" equals three, then four shift operations will be performed, each shifting the input bits to the left by three bits). The shift operation can be followed by an accumulate operation and an XOR operation to update the sign of the result bits. The bits can be scaled by the scaling factor β of the FPoT quantization to obtain the output bit representation of the multiplication.

[0058] Figure 9 Another example process of performing MAC operation by the MAC unit according to a specific embodiment of the present disclosure is shown. This process can be used as Figure 8 804. Here, although the process can be similarly applied to other types of numbers and / or number multiplications, the process is specifically described in conjunction with the weights of the AI ​​inference engine. Although the process can be similarly applied to other types of quantization, the process is also described in conjunction with FPoT quantization, where once the decompression of the weight index occurs, the number of shift operations and the shift amount of each shift operation can depend on the specific quantization type (e.g., for uniform quantization, the shift is 1; for PoT quantization, the shift is "b"; for APoT, the shift is "k"). For clarity of explanation, FPoT quantization is shown as each of "k" and "r" equal to 4. Further, ceil(log2(Q)) = 3 or eight fully non-uniform quantization levels are used (e.g., "m" is equal to 3, so that the index is represented by three bits in addition to its sign bit). Each of these eight values ​​can be represented by the set It is represented by the sum of the four values ​​in .

[0059] As shown, the process can begin at operation 902, where the system loads weights into a weight buffer and loads input values ​​into an input buffer. Loading weights can include loading a 3-bit index of the weight in addition to loading a sign bit. The input value is the value of the variable to be multiplied by the weight. In addition to storing the sign bit, the input buffer can also store six bits of the bit representation of the input value. As an example, a high-precision fixed-point format is used for the input value, but other formats are also possible.

[0060] At operation 904, the system uses the weight value to lookup "k*r" bits. "k*r" corresponds to "l" bits of the quantization level. The lookup can use the index bits of the weight, which are loaded into the index buffer each time operation 902 is performed. For example, a 3-bit index is used to lookup "k*r" bits. The lookup identifies (s1, s2, ... s r ) set (for example, four sets) "s i ”, where each set has “k” bits (e.g., four bits) from the table.

[0061] At operation 906, for i=1:r (eg, for 1 to 4), the system shifts the input buffer left by s. i The shifted input buffer is shifted to the left by s1 bits and accumulated into the accumulator buffer. For example, four shift operations are performed. After the first operation, the input buffer is shifted to the left by s1 bits. After the second operation, the input buffer is shifted to the left by s2 bits. After the third operation, the input buffer is shifted to the left by s3 bits. And after the fourth operation, the input buffer is shifted to the left by s4 bits. An accumulation operation is performed after each of the first, second, and third shift operations.

[0062] At operation 908, the system uses the XOR of the sign bits from the weight buffer and the input buffer as the sign bit of the accumulator. For example, an XOR operation is performed on the sign bit and the result sign is set to the sign of the accumulation bit.

[0063] At operation 910, the system applies a scaling factor to the output of the accumulator. For example, the output is multiplied by the FPoT-quantized β. This multiplication can be done in full-precision fixed-point or floating-point format, or in FPoT format, which requires selecting the β scaling factor from another set of PoT sums that are allowed to have values ​​greater than 1.

[0064] Figure 10 Another example process for compressing quantization levels according to certain embodiments of the present disclosure is shown. Figure 8 and Figure 9 This process is performed before the process of generating quantization levels and related compression. Figure 8 and Figure 9Once the process is executed, the results can be used to implement the AI ​​inference engine, Figure 8 and Figure 9 The process can be applied to the AI ​​inference engine. Although the process can be similarly applied to other types of quantization, it is illustrated here in conjunction with FPoT quantization, where once the quantized values ​​are generated, they can be clustered and compression can be defined based on the clustering. Nevertheless, and as combined Figures 4 to 6 As described, FPoT can provide additional flexibility in clustering to meet the target weight distribution.

[0065] According to FPoT quantization, for all values ​​that can be represented by floating-point numbers, there are some values ​​that can cause simple multiplication operations. For example, if the floating-point number x (multiplicand) is multiplied by 1 / 8 (multiplier), the exponent (including the sign) can simply be subtracted by three. If the fixed-point number x (multiplicand) is multiplied by 1 / 8 (multiplier), the fixed-point part (mantissa) of the multiplicand can be shifted by three bits (padding to the left with zeros). All values ​​that retain this property form a set "S". The idea behind FPoT is to ensure that all quantized weights belong to "S" so that multiplication can be easily completed.

[0066] Combined with transforming floating-point DNN into FPoT format to describe Figure 10 When using FPoT quantization, the weights in the DNN are quantized to specific quantization values, which can be described in the following equation: in Where β is the scaling factor, and where k and r are integers, and k×r is the bit resolution. This equation essentially states that each quantized weight value belongs to a set "S," which is a subset of "C," which contains the sum of "r" candidate quantization values, each of which belongs to the set of PoT fractions. β is the full-precision scaling factor.

[0067] As shown in the figure, the process may start at operation 1002, where the system generates a set "C" of candidate quantization values ​​based on PoT quantization. For example, the set "C" is in

[0068] At operation 1004, the system finds a non-uniform quantization scheme for the weights such that the loss in prediction accuracy is acceptable: w' = g(w). In an example, a k-means clustering algorithm is used to find "Q" sets of weights, where "Q" is the number of quantization levels for non-uniform quantization, and the center of each cluster (or some other statistical point thereof) is a quantization level. According to this operation, the candidate quantization values ​​are clustered into "Q" clusters, where each of the "Q" clusters includes a subset of the candidate quantization values ​​and is represented by a statistical point based on the subset. Figures 5 and 6 As shown, clustering can be performed based on the target distribution of quantized values.

[0069] At operation 1006, for each weight in w', the system replaces it with the closest number in the set "C." For example, for the first of the "Q" clusters, the system replaces the statistical point with a value from the set "C." This value can be the closest number. This replacement can be repeated and performed for each of the remaining "Q" clusters.

[0070] At operation 1008, the system uses the weights from w' to determine the performance of the AI ​​inference engine. For example, real data (e.g., known data for which the AI ​​inference output is known) is input to the AI ​​inference engine to measure performance. The performance can be a function of the known output and the actual output generated by the AI ​​inference engine.

[0071] At operation 1010, the system determines whether the performance is acceptable. For example, the performance may be compared to a performance threshold. If the performance is greater than the performance threshold, the performance is acceptable, and operation 1020 is performed after operation 1010. Otherwise, the performance is unacceptable, and operation 1030 is performed after operation 1010. The performance threshold may depend on the application using the AI ​​inference engine.

[0072] At operation 1020, the performance is acceptable, and the system labels "Q" quantization levels from the set {0, 1, ..., Q-1}. Each label is an index. The labels form a set "S". A table "T" is generated to associate the quantization level index (0, 1, ..., Q-1) with its corresponding "r*k" bit representation, rather than using "r*k" bits to represent each weight. Each weight can be represented by ceil(log2(Q)) = the number of bits (e.g., the number of index bits "m" is ceil(log2(Q))).

[0073] At operation 1030, the performance is unacceptable, and the system changes any one of "k," "r," and / or "Q." For example, any or each of these parameters may be increased by one or some other integer increment. If either "k" or "r" is changed, the process loops back to operation 1002 to repeat operations 1002 through 1010. If "Q" is changed, the process loops back to operation 1004 to repeat operations 1004 through 1010.

[0074] In the example, operations 1008 to 1020 (including any loop through operation 1030) represent operations for generating a table "T". The table "T" includes a first index corresponding to a first cluster and associates the first index with a "1"-bit representation (or equivalently, an "r*k"-bit representation) of the value of a statistic point of the first cluster. The table includes similar associations between the index and the "1"-bit representation of the remaining statistic points of the remaining clusters.

[0075] In operation 1022, the system again determines whether the performance of the AI ​​inference engine is acceptable. Here, the weights of the AI ​​model are initialized using the quantization level bit representation in the table. The AI ​​model is retrained based on the initialized weights. For example, the system initializes the floating-point AI model using the floating-point representation of the weights obtained from operation 1020, and retrains the AI ​​model using the floating-point weights. The performance of the AI ​​model is then determined and compared with a performance threshold that is the same as or different from the performance threshold in operation 1010. If the performance is acceptable, operation 1040 is performed after operation 1022. Otherwise, operation 1050 is performed after operation 1022.

[0076] At operation 1040, the performance is acceptable and table "T" is output. This table can then be used for compression and decompression (adjusted if any according to operation 1030) of a bit representation using the FPoT quantization set with parameters "k" and "r" from operation 1002.

[0077] At operation 1050 , the performance is unacceptable, and similar to operation 1022 , the system is reinitialized and the AI ​​model is retrained, and the process loops back to operation 1022 .

[0078] Figure 11 Specific embodiments according to the present disclosure are shown such as by including Figure 1 An example of a computing system based on compressed MAC operations of the processor 110. Figure 11This is merely an illustration of an embodiment of the present disclosure and does not limit the scope of the present disclosure as described in the claims. In one embodiment, the system is a computer system 1100, which generally includes a monitor 1110, a computer 1120, a user output device 1130, a user input device 1140, a communication interface 1150, etc. Figure 1 The error correction system 100 implements some or all of the components of the computer system 1100 .

[0079] like Figure 11 As shown, computer 1120 may include a processor 1160 that communicates with a number of peripheral devices via a bus subsystem. These peripheral devices may include a user output device 1130, a user input device 1140, a communication interface 1150, and a storage subsystem such as random access memory (RAM) 1170 and a disk drive 1180.

[0080] User input devices 1140 include all possible types of devices and mechanisms for inputting information into computer system 1120. These may include a keyboard, a keypad, a touch screen integrated into a display, audio input devices such as a voice recognition system, a microphone, and other types of input devices. In various embodiments, user input devices 1140 are typically implemented as a computer mouse, a trackball, a touchpad, a joystick, a wireless remote control, a drawing tablet, a voice command system, an eye tracking system, etc. User input devices 1140 typically allow a user to select objects, icons, text, etc. that appear on monitor 1110 via commands such as clicking a button.

[0081] User output devices 1130 include all possible types of devices and mechanisms for outputting information from the computer system 1120. These may include a display (eg, monitor 1110), non-visual displays such as audio output devices, and the like.

[0082] Communication interface 1150 provides an interface to other communication networks and devices. Communication interface 1150 can be used as an interface for receiving data from other systems and transmitting data to other systems. Examples of communication interface 1150 typically include an Ethernet card, a modem (telephone, satellite, cable, Integrated Services Digital Network (ISDN)), an (asynchronous) Digital Subscriber Line (DSL) unit, a FireWire interface, a USB interface, and the like. For example, communication interface 1150 can connect to a computer network, a FireWire bus, and the like. In other embodiments, communication interface 1150 can be physically integrated into the motherboard of computer 1120 and can be a software program such as soft DSL.

[0083] In various embodiments, the computer system 1100 may also include software capable of communicating over a network such as HTTP, TCP / IP, RTP / RTSP, etc. In alternative embodiments of the present disclosure, other communication software and transport protocols may also be used, such as IPX, UDP, etc. In some embodiments, the computer 1120 includes one or more Xeon processors from Intel as processors 1160. TM Microprocessor. Further, in one embodiment, computer 1120 includes a UNIX-based operating system.

[0084] RAM 1170 and disk drive 1180 are examples of tangible media configured to store data, including executable computer code, human-readable code, etc., such as embodiments of the present disclosure. Other types of tangible media include: floppy disks; removable hard disks; optical storage media such as CD-ROMs, DVDs, and bar codes; semiconductor memories such as flash memory, non-transitory read-only memories (ROMs), and battery-powered volatile memories; network storage devices, etc. RAM 1170 and disk drive 1180 can be configured to store basic programming and data structures that provide the functionality of the present disclosure.

[0085] Software code modules and instructions that provide the functionality of the present disclosure may be stored in RAM 1170 and disk drive 1180. These software modules may be executed by processor 1160. RAM 1170 and disk drive 1180 may also provide repositories for storing data used in accordance with the present disclosure.

[0086] RAM 1170 and disk drive 1180 may include multiple memories, including a main RAM for storing instructions and data during program execution and a ROM for storing fixed, non-transitory instructions. RAM 1170 and disk drive 1180 may include a file storage subsystem that provides persistent (non-volatile) storage for program and data files. RAM 1170 and disk drive 1180 may also include a removable storage system, such as a removable flash memory.

[0087] The bus subsystem provides a mechanism for the various components and subsystems of the computer 1120 to communicate with each other as intended. Although the bus subsystem is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple busses.

[0088] Figure 11is a representation of a computer system capable of implementing the present disclosure. It will be apparent to one of ordinary skill in the art that many other hardware and software configurations are suitable for use with the present disclosure. For example, the computer may be a desktop, portable, rack-mounted, or tablet configuration. Additionally, the computer may be a series of networked computers. Further, it is contemplated that other microprocessors such as the Pentium TM ) or Itanium TM ) microprocessors, Opteron from Advanced Micro Devices, Inc. TM ) or AthlonXP TM ) microprocessors, etc. Further, it is contemplated to use other types of operating systems such as: Microsoft Corporation's etc.; Solaris of Sun Microsystems; LINUX; UNIX, etc. In other embodiments, the above technology can be implemented on a chip or an auxiliary processing board.

[0089] The various embodiments of the present disclosure may be implemented in the form of logic in software or hardware or a combination of both. The logic may be stored as a set of instructions in a computer-readable or machine-readable non-transitory storage medium, which is suitable for guiding a processor of a computer system to perform a set of steps disclosed in the embodiments of the present disclosure. The logic may form part of a computer program product, which is suitable for guiding an information processing device to perform a set of steps disclosed in the embodiments of the present disclosure. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will understand other ways and / or methods of implementing the present disclosure.

[0090] The data structures and code described herein may be stored, in part or in whole, on computer-readable storage media and / or hardware modules and / or hardware devices. Computer-readable storage media include, but are not limited to, volatile memory; non-volatile memory; magnetic and optical storage devices such as disk drives, tapes, CDs, and DVDs; or other media now known or later developed that can store code and / or data. The hardware modules or devices described herein include, but are not limited to, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated or shared processors, and / or other hardware modules or devices now known or later developed.

[0091] The methods and processes described herein may be implemented in part or in whole as code and / or data stored in a computer-readable storage medium or device, such that when a computer system reads and runs the code and / or data, the computer system performs the associated methods and processes. These methods and processes may also be implemented in part or in whole in a hardware module or device, such that when the hardware module or device is activated, the hardware module or device performs the associated methods and processes. The methods and processes disclosed herein may be implemented using a combination of code, data, and hardware modules or devices.

[0092] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the present disclosure is not limited to the details provided. There are many alternative ways to implement the present disclosure. The disclosed embodiments are illustrative and non-restrictive.

Claims

1. A method implemented by a system for performing a multiply and accumulate operation (MAC operation), the method comprising: Determine that the first number will be multiplied by the second number; and performing the MAC operation to multiply the first number by the second number, wherein performing the MAC operation comprises: determining an index of a quantization level associated with the first number, wherein the index is represented by 'm' bits, wherein the quantization level is represented by '1' bits, and wherein '1' is greater than 'm', determining "r" groups of "k" bits by using at least said index in a table lookup, wherein "r*k" equals "1" and wherein "1" bits comprise said "r" groups of "k" bits, and using said 'r' groups of 'k' bits in a shift and accumulate operation applied to an input buffer storing a bit representation of said second number; The MAC operation is applied during the execution of an artificial intelligence model, i.e., an AI model.

2. The method of claim 1 , wherein the system comprises an artificial intelligence inference engine (AI inference engine), wherein the first number comprises a weight of an AI model, wherein the second number comprises a variable of a transformation function of the AI ​​model, and wherein the transformation function comprises multiplying the variable by the weight.

3. The method of claim 1 , wherein using the “r” groups of “k” bits comprises: Input "m" bits of the index into the index buffer, determining a first group of "k" bits from the "r" groups of "r" bits by using "m" bits from the index buffer in a lookup of the table, wherein the first group includes "k" bits out of "l" bits, and wherein "k" is less than "l", Shifting the input buffer by the first group of "k" bits, and Accumulate the shifted input buffer in the accumulator buffer.

4. The method of claim 3, wherein using the 'r' groups of 'k' bits further comprises repeating the shift operation and the accumulate operation for each remaining group in the 'r' groups.

5. The method according to claim 1, further comprising: (a) Generate a set of candidate quantization values ​​"C" based on quadratic quantization, i.e., PoT quantization; (b) clustering the candidate quantization values ​​into Q clusters, wherein each of the Q clusters includes a subset of the candidate quantization values ​​and is represented by a statistical point based on the subset; (c) for a first of said "Q" clusters, replacing said statistical points with values ​​from said set "C"; and (d) generating the table, wherein the table includes a first index corresponding to the first cluster and the table associates the first index with a “1”-bit representation of the value.

6. The method according to claim 5, wherein the PoT quantification comprises a value represented as PoT is non-uniformly quantized, where and where β is a scaling factor. The method according to claim 5 , wherein the clustering is performed based on a target distribution of quantized values.

8. The method of claim 5, wherein the statistical point is replaced with the closest number from the set "C".

9. The method of claim 5, wherein the system comprises an artificial intelligence inference engine (AI inference engine), wherein the method further comprises: The performance of the AI ​​inference engine is determined by using 'Q' statistical points representing the 'Q' clusters, wherein each of the 'Q' statistical points comprises a weight of the AI ​​model, and wherein the table is generated based on the performance.

10. The method according to claim 9, further comprising: Based on the performance, at least one of the following is changed: the value of "r", the value of "k", or the value of "Q".

11. The method according to claim 10, further comprising: While changing the value of "r" or the value of "k", operations (a) to (c) are repeated.

12. The method according to claim 10, further comprising: While changing the value of "Q", operations (b) to (c) are repeated.

13. The method of claim 5, wherein the system comprises an AI inference engine, and wherein the method further comprises: Initializing the weights of the AI ​​model by using the quantization level bit representation from the table; and The AI ​​model is retrained based on the initialized weights.

14. A system comprising: One or more processors, wherein at least one of the one or more processors: Determine that the first number will be multiplied by the second number; and Performing a multiplication and accumulation operation (MAC operation) to multiply the first number by the second number, wherein performing the MAC operation includes: determining an index of a quantization level associated with the first number, wherein the index is represented by 'm' bits, wherein the quantization level is represented by '1' bits, and wherein '1' is greater than 'm', determining "r" groups of "k" bits by using at least said index in a table lookup, wherein "r*k" equals "1" and wherein "1" bits comprise said "r" groups of "k" bits, and Use the "r" in shift and accumulate operations applied to the input buffer. A group of "k" bits, wherein the input buffer stores a bit representation of the second number; The MAC operation is applied during the execution of an artificial intelligence model, i.e., an AI model.

15. The system of claim 14, wherein at least one of the one or more processors: Generate a set "C" of candidate quantization values ​​based on PoT quantization; clustering the candidate quantization values ​​into Q clusters, wherein each of the Q clusters includes a subset of the candidate quantization values ​​and is represented by a statistical point based on the subset; For a first of the "Q" clusters, replacing the statistical points with values ​​from the set "C"; and The table is generated, wherein the table includes a first index corresponding to the first cluster and associates the first index with a "1"-bit representation of the value.

16. The system of claim 15, wherein the PoT quantification comprises a value represented as PoT is non-uniformly quantized, where and where β is a scaling factor.

17. An apparatus comprising: One or more processors: Determine that the first number will be multiplied by the second number; and Performing a multiplication and accumulation operation (MAC operation) to multiply the first number by the second number, wherein performing the MAC operation includes: determining an index of a quantization level associated with the first number, wherein the index is represented by 'm' bits, wherein the quantization level is represented by '1' bits, and wherein '1' is greater than 'm', determining "r" groups of "k" bits by using at least said index in a table lookup, wherein "r*k" equals "1" and wherein "1" bits comprise said "r" groups of "k" bits, and using said 'r' groups of 'k' bits in a shift and accumulate operation applied to an input buffer storing a bit representation of said second number; The MAC operation is applied during the execution of an artificial intelligence model, i.e., an AI model.

18. The apparatus according to claim 17, further comprising: An artificial intelligence reasoning engine, i.e., an AI reasoning engine, wherein the first number comprises a weight of an AI model, wherein the second number comprises a variable of a transformation function of the AI ​​model, and wherein the transformation function comprises multiplying the variable by the weight.

19. The apparatus of claim 17, wherein using the "r" groups of "k" bits comprises: determining a first group of "k" bits from the "r" groups of "k" bits based on a lookup in the table, wherein the first group includes "k" bits out of "l" bits and wherein "k" is less than "l", Shifting the input buffer by the first group of "k" bits, and Accumulate the shifted input buffer in the accumulator buffer.

20. The apparatus of claim 19, wherein using the 'r' groups of 'k' bits further comprises repeating the shift operation and the accumulate operation for each remaining group in the 'r' groups.

Images