Hardware acceleration circuit, chip, data processing acceleration method, accelerator and device

Abstract

The application relates to a hardware acceleration circuit, a chip, a data processing acceleration method, an accelerator and equipment. The circuit comprises a lookup table circuit, which is used for outputting an exponential function value corresponding to a first index value of an i-th element in input data based on a first lookup table in response to the i-th element, and / or outputting an inverse of an exponential function value corresponding to a second index value of the i-th element based on a second lookup table; an adder, which is used for outputting an addition operation result of the exponential function values corresponding to at least part of the elements in the input data; and a multiplier, which is used for outputting a multiplication operation result of the inverse of the exponential function value corresponding to the i-th element and the addition operation result, so as to obtain an inverse of a specific function value corresponding to the i-th element. The application can reduce the entries of the lookup table and improve the accuracy of the specific function value.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a hardware acceleration circuit, an artificial intelligence chip, a data processing acceleration method, an artificial intelligence accelerator, and an electronic device. Background Technology

[0002] Nonlinear functions introduce nonlinear characteristics into artificial neural networks, playing a crucial role in their ability to learn and understand complex scenarios. Nonlinear functions include, but are not limited to, the softmax function and the sigmoid function.

[0003] The Softmax function, widely used in deep learning, is a prime example. In related technologies, its value can be calculated using general-purpose computing units such as CPUs or GPUs. However, when neural network processing is executed by hardware circuits such as Deep Learning Accelerators (DLAs) or Neural Network Processing Units (NPUs), if the Softmax function layer is located in the middle layer of the neural network, it leads to job migration overhead between the DLA / NPU and the CPU / GPU. This makes the scheme of using the CPU / GPU to determine the nonlinear function value inefficient, resulting in increased system bandwidth and higher power consumption. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a hardware acceleration circuit, an artificial intelligence chip, a data processing acceleration method, an artificial intelligence accelerator, and an electronic device, which can improve the data processing speed in the calculation of nonlinear functions and accelerate the acquisition of function values.

[0005] The first aspect of this application provides a hardware acceleration circuit, including a lookup table circuit, configured to, in response to the i-th element in an input dataset, output an exponential function value corresponding to a first index value of the i-th element based on a first lookup table, and / or output the reciprocal of the exponential function value corresponding to a second index value of the i-th element based on a second lookup table, wherein i is an integer greater than or equal to 1; an adder, configured to output the addition result of exponential function values ​​corresponding to at least some elements in the input dataset; and a multiplier, configured to output the multiplication result of the reciprocal of the exponential function value corresponding to the i-th element and the addition result, to obtain the reciprocal of a specific function value corresponding to the i-th element, thereby obtaining the specific function value corresponding to the i-th element.

[0006] A second aspect of this application provides an artificial intelligence chip, including the hardware acceleration circuit described above.

[0007] A third aspect of this application provides a data processing acceleration method, comprising obtaining the exponential function value corresponding to the i-th element in the input dataset, and obtaining the reciprocal of the exponential function value corresponding to the i-th element, wherein i is an integer greater than or equal to 1; obtaining the addition result of the exponential function values ​​corresponding to at least some elements in the input dataset; and obtaining a specific function value corresponding to the i-th element based on the reciprocal of the exponential function value corresponding to the i-th element and the addition result.

[0008] A fourth aspect of this application provides an artificial intelligence accelerator, comprising: a processor; and a memory storing executable code thereon, wherein when the executable code is executed by the processor, the processor performs the method described above.

[0009] A fifth aspect of this application provides an electronic device, comprising: a processor configured to send at least one of a first lookup table, a second lookup table, or a third lookup table to an artificial intelligence chip, wherein the first lookup table includes a first mapping relationship between the i-th element in the input dataset and the value of an exponential function; the second lookup table includes a second mapping relationship between the i-th element in the input dataset and the reciprocal of the exponential function value; the third lookup table includes a third mapping relationship between the multiplication result of the multiplication operation of the reciprocal of the exponential function value corresponding to the i-th element and the corresponding reciprocal of the multiplication operation result; and the addition operation result is the addition result of the exponential function values ​​corresponding to at least some elements in the input dataset; and an artificial intelligence chip configured to perform the above method based on at least one of the first lookup table, the second lookup table, or the third lookup table.

[0010] A sixth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0011] The seventh aspect of this application provides a computer program product including executable code that, when executed, implements the method described above.

[0012] The technical solution provided in this application may include the following beneficial effects:

[0013] In this embodiment, a specific function is inversely operated on, such that the sum of the exponential function values ​​corresponding to at least some elements in the input dataset (referred to as the accumulated value of the exponential function values) becomes the numerator, and the reciprocal of the exponential function value corresponding to the i-th element (referred to as the reciprocal of the exponential function value) becomes the denominator. When the reciprocal of the exponent and the accumulated value of the exponent are too large, the reciprocal of the multiplication result between them approaches zero. Since neural networks can focus more on specific function values ​​close to 1, these data approaching zero can be saturated and ignored. Based on the above operation of inversely operating on a specific function, when using a lookup table to obtain at least one of the exponential function values, the reciprocal of the exponential function values, or specific function values, a lookup table with fewer entries can be used, which is more acceptable for hardware implementation. In addition, the accuracy of the calculated specific function values ​​can be improved by reducing the number of entries in the lookup table.

[0014] Furthermore, in some embodiments of this application, the data format used for intermediate results and specific function values ​​generated during the determination of a specific function is set, thereby further improving the accuracy of the obtained specific function values.

[0015] Furthermore, in some embodiments of this application, the intermediate results are stored by using a storage space with a specified number of bits, and the intermediate results are truncated by a shift operation, which conveniently achieves the function of saturating and ignoring oversaturated data.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0018] Figure 1 This is a schematic diagram of the structure of a neural network shown in one embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of a neural network for classification shown in one embodiment of this application;

[0020] Figure 3 This is a block diagram of a hardware acceleration circuit shown in one embodiment of this application;

[0021] Figures 4 to 7 This is a block diagram of a hardware acceleration circuit shown in another embodiment of this application;

[0022] Figure 8This is a schematic flowchart illustrating a data processing acceleration method according to an embodiment of this application;

[0023] Figure 9 This is a schematic diagram illustrating the inverse operation in an embodiment of this application;

[0024] Figure 10 This is a logic diagram of a data processing acceleration method shown in an embodiment of this application;

[0025] Figure 11 This is a schematic diagram illustrating the format of each data item during the execution of the data processing acceleration method shown in the embodiments of this application;

[0026] Figure 12 This is a schematic diagram of the structure of the data processing acceleration device shown in the embodiments of this application;

[0027] Figure 13 This is a schematic diagram of the structure of an artificial intelligence accelerator shown in an embodiment of this application;

[0028] Figure 14 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0029] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0030] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a” and “the” as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0032] The computation of nonlinear functions may involve operations involving exponential functions and / or their reciprocals. For example, the computation of the Softmax function may involve operations involving the exponent (exp) and the reciprocal of the sum of exponents (1 / sum_of_exp). Dedicated hardware pipelines for the Softmax function are not feasible for large-scale computational needs. For instance, increased computational power leads to expensive hardware costs. One method for obtaining the Softmax function in related technologies uses a lookup table (LUT) to search for 16-bit integers (INT16). However, a 16-bit LUT is a very large table in terms of storage space; for example, if it contains (2^16, i.e., 65536) entries, it requires a large amount of static random access memory (SRAM) / dynamic random access memory (DRAM) to store the data, which makes the cost of the LUT combinational logic circuitry too high. On the other hand, completing a single LUT result requires 65536 cycles, resulting in excessively long processing times.

[0033] This application provides a hardware acceleration circuit, chip, data processing acceleration method, accelerator, and electronic device that performs inverse operations on nonlinear functions to obtain multiple logical components. For example, the Softmax function can be decomposed into the sum of exponential functions and the reciprocal of the exponents. At least some of the logical components of the function can be implemented using lookup tables that occupy less storage space, which can balance the power consumption, bandwidth, performance, and accuracy of determining the function value of the nonlinear function, thus meeting the requirements of neural networks.

[0034] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram of the structure of a neural network shown in one embodiment of this application.

[0036] See Figure 1 The diagram illustrates the topology of a neural network 100, including an input layer, hidden layers, and an output layer. This neural network 100 is capable of performing calculations or operations based on input data I1 and I2 received by the input layer, and generating output data O1 and O2 based on the results of the calculations.

[0037] For example, neural network 100 can be a deep neural network (DNN) that includes one or more hidden layers. Figure 1The neural network 100 in the diagram includes an input layer L1, two hidden layers L2 and L3, and an output layer L4. The DNN includes, but is not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer networks.

[0038] It should be noted that, Figure 1 The four layers shown are for illustrative purposes only and should not be construed as limiting the scope of this application. For example, a neural network may include more or fewer hidden layers.

[0039] Nodes in different layers of the neural network 100 can be connected to each other for data transmission. For example, a node can receive data from other nodes to perform calculations on the received data and output the calculation results to nodes in other layers.

[0040] Each node can determine its output data based on the output data and weights received from nodes in previous layers. For example, Figure 1 middle This represents the weight between the first node of the first layer and the first node of the second layer. This represents the output data of the first node in the first layer. Let represent the bias value of the first node in the second layer. Then, the output data of the first node in the second layer can be represented as: The calculation method for the output data of other nodes is similar, and will not be described in detail here.

[0041] In some embodiments, the neural network is configured with activation function layers, such as flexible maximization function layers, which can convert the result values ​​for each class into probability values.

[0042] In some embodiments, the neural network is configured with a loss function layer after the flexible maximum function layer, which is capable of calculating the loss as an objective function for training or learning.

[0043] Understandably, a neural network can respond to the data to be processed, and after processing the data, obtain a recognition result. The data to be processed includes, but is not limited to, at least one of: speech data, text data, and image data.

[0044] A typical type of neural network is a classification neural network. Classification neural networks determine the category of input data by calculating the probability of each class corresponding to the input data.

[0045] Figure 2 This is a schematic diagram of the structure of a neural network for classification, as shown in one embodiment of this application.

[0046] See Figure 2 The neural network 200 used for classification in this embodiment may include a hidden layer 210, a fully connected layer (FC layer) 220, a flexible maximum function layer 230, and a loss function layer 240.

[0047] The neural network 200 responds to the data to be classified by sequentially computing the hidden layer 210 and the fully connected (FC) layer 220. The FC layer 220 outputs a computational result s, which corresponds to the classification probability of the data to be classified. The FC layer 220 may include multiple nodes corresponding to multiple classes, each node outputting a result value corresponding to the probability that the data to be classified is assigned to the corresponding class. See also... Figure 1 FC layer corresponds to Figure 1 The output layer L4 has two nodes, each corresponding to one of two categories (Class 1 and Class 2). One node's output value represents the probability of the data being classified into Class 1, and the other node's output value represents the probability of the data being classified into Class 2. The functions of the hidden layer 210, the fully connected (FC) layer 220, the flexible maximum function layer 230, and the loss function layer 240 can be implemented using an AI chip. AI chips include, but are not limited to, DLA (Data Loss Algorithm) and NPU (Neural Processing Unit).

[0048] The FC layer 220 outputs the calculation result s to the flexible maximal function layer 230, which converts the calculation result s into a probability value y and can also normalize the probability value y. Specifically, the function of the FC layer can be implemented through a fully connected layer circuit. For example, the fully connected layer circuit is used to transmit the i-th element or the index value of the i-th element in the input dataset to the aforementioned hardware acceleration circuit.

[0049] The flexible maximum function layer 230 outputs the probability value y to the loss function layer 240, which can calculate the cross-entropy loss L of the result s based on the probability value y. Specifically, the function function layer 240 can be implemented through a loss function layer circuit. For example, the loss function layer circuit is used to transmit a loss value for a specified function value to the hardware acceleration circuit in response to the specified function value, so that the hardware acceleration circuit outputs the loss gradient value for the i-th element in the input dataset.

[0050] During the backpropagation learning process, the flexible maximum function layer 230 calculates the gradient of the cross-entropy loss L. Then, FC layer 220 performs gradient calculations based on the cross-entropy loss L. The learning process can be further refined. For example, the weights of the FC layer 220 can be updated using the gradient descent algorithm. Further, subsequent learning processes are performed in the hidden layer 210.

[0051] It should be noted that, Figure 2 At least a portion of the hidden layer 210, the FC layer 220, the flexible maximum layer 230, and the loss function layer 240 can be implemented in software, or in hardware circuitry, or in a combination of both. For example, Figure 2 The output s of the FC layer 220 can be transmitted to the CPU or GPU. The CPU or GPU calculates the Softmax value y and sends it back to the loss layer. The output L of the loss layer needs to be transmitted to the CPU or GPU for processing to obtain the gradient of the cross-entropy loss L, and then transmitted back to the FC layer 220. Alternatively, in the case of hardware implementation, the hidden layer 210, FC layer 220, flexible maximum function layer 230, and loss function layer 240 can all be implemented by hardware circuits, either integrated into a single AI chip or distributed across multiple chips. This avoids the data migration between other layers of the neural network and processors like the CPU / GPU when the flexible maximum function layer 230 is implemented by the CPU / GPU, improving the efficiency of neural network data processing, reducing data processing latency and power consumption, and avoiding increased bandwidth usage.

[0052] Figure 3 This is a block diagram illustrating a hardware acceleration circuit according to an embodiment of this application. In this application, the hardware acceleration circuit can be used, for example, but not limited to, to implement the flexible maximum function layer 230 in the neural network 200 described above. The hardware acceleration circuit can be, for example, but not limited to, a circuit component in a CPLD (Complex Programming logic device) chip, an FPGA (Field Programmable Gate Array) chip, a dedicated chip, etc.

[0053] See Figure 3 The hardware acceleration circuit 300 for artificial intelligence may include a lookup table circuit 310, an adder 320, and a multiplier 330.

[0054] The lookup table circuit 310 is configured to, in response to the i-th element in the input dataset, output the exponential function value corresponding to the first index value of the i-th element based on a first lookup table, and / or output the reciprocal of the exponential function value corresponding to the second index value of the i-th element based on a second lookup table. Here, i is an integer greater than or equal to 1. The first index value and the second index value can be the same or different. The index value can be data from the i-th element, or it can be obtained by transforming data from the i-th element, for example, it can be a portion of data extracted from the data of the i-th element.

[0055] For example, a first lookup table can be used to implement a first mapping relationship between the first index value of the i-th element and the exponential function value for that i-th element. A second lookup table can be used to implement a second mapping relationship between the second index value of the i-th element and the reciprocal of the exponential function value for that i-th element. These first and second mapping relationships can be pre-defined, such as a labeled correspondence. These first and second lookup tables allow the exponential function value and its reciprocal to be determined for the i-th element using a pre-defined mapping relationship without requiring complex function calculations.

[0056] It should be noted that the lookup table circuit 310 can implement at least one of the first and second mapping relationships. For example, the value of the exponential function can be determined by the first lookup table or calculated by the processor (such as a CPU or GPU) through software. Similarly, the reciprocal of the exponential function value can be determined by the second lookup table or calculated by the processor through software.

[0057] Adder 320 is used to output the result of an addition operation with the exponential function values ​​corresponding to at least some elements in the input dataset. The result of the addition operation 320 can be data with a specified bit width, such as 32-bit data, 8-bit data, etc.

[0058] Multiplier 330 is used to output the multiplication result of the reciprocal of the exponential function value corresponding to the i-th element and the addition operation result, to obtain the reciprocal of the specific function value corresponding to the i-th element, so as to obtain the specific function value corresponding to the i-th element.

[0059] It is understandable that the result of adding exponential function values ​​can be obtained by directly adding the exponential function values, or by performing a specific transformation on the exponential function values ​​before adding them. In the case of a specific transformation, the subsequent data processing results may be subjected to an inverse transformation or no inverse transformation may be performed, depending on the type of transformation. Similarly, the processing of other data should be broadly understood to include both of the above situations, and should not be limited to processing only the data itself. Other embodiments are similar and will not be described further below.

[0060] For example, the reciprocal of the exponential function value corresponding to the i-th element can be based on the reciprocal of the exponential function value corresponding to the i-th element itself, or it can be data determined after a specific transformation of that reciprocal. For example, the reciprocal can be data with a longer bit width obtained according to a preset precision, etc. However, in order to reduce the storage space occupied by the multiplier, thereby reducing hardware costs and improving response speed, a specific bit width of data can be extracted from the reciprocal of the specific function value corresponding to the i-th element, and this data with a specific bit width can be used as the reciprocal of the exponential function value for subsequent multiplication operations.

[0061] Let's take the Softmax function as an example for illustration. Suppose we have an array X, then the i-th element x... i The formula for calculating the Softmax function value is shown in equation (1).

[0062]

[0063] In equation (1), σ(x) i Represents the i-th element x i The value of the Softmax function, where e is the natural constant, x i Let x represent the i-th element of array X. max This represents the maximum element in array X. This represents the sum of the exponential function values ​​of at least some elements in array X.

[0064] The denominator in equation (1) has a very wide range, making it difficult to quantize it into a value range of appropriate integer bit widths. This results in a large number of entries in the lookup table or insufficient precision. For example, when the bit width is large, there are many entries in the lookup table, leading to excessive hardware costs. When the bit width is insufficient, the precision of a specific function value will decrease.

[0065] In some embodiments, the formula for calculating the Softmax function value can be reversed, as shown in equation (2).

[0066]

[0067] In equation (1), The value of is relatively large, and the range of its values ​​is also large. The value is relatively small. Compared to This effectively reduces the number of integer values ​​and decreases the range of value distribution. Compared to This effectively increases the number of decimal values. When using equation (2) to determine the value of the Softmax function, the difference between the two parts can be reduced, making it easier to quantize them into an appropriate integer bit width, which helps to improve the calculation accuracy.

[0068] Furthermore, because the Softmax function is inversely operated on, the addition result is in the denominator, and the reciprocal of the specific function value decreases as the addition result increases. In some applications, it is more important to ensure that the reciprocal of the specific function value approaches 1; when the addition result is large, the reciprocal approaches 0. Therefore, when the addition result is large, it may cause saturation due to excessive size and can be ignored. A smaller number of entries can be used to meet high precision requirements. For example, the above addition result can be a specific bit of data truncated from the above 32 bits, such as truncating 8 bits from the most significant bit downwards.

[0069] In this embodiment, the inverse operation on a specific function effectively reduces the number of entries required to determine the reciprocal of the exponential function sum in related technologies, making it more acceptable for hardware implementation. Furthermore, compared to the integer LUT schemes of related technologies (using formula (1)), the specific function value determined in this embodiment has higher precision.

[0070] In some embodiments, the lookup table circuit is also used to output a specific function value corresponding to the index value of the multiplication result based on a third lookup table in response to the multiplication result. Since the specific function is inversely operated on, the multiplication result and the specific function value are reciprocals. In related technologies, hardware-accelerated circuits have low efficiency in performing division operations, making it inconvenient to use the processor's computing power for inverse operations to quickly obtain the specific function value (i.e., the reciprocal of the multiplication result). This embodiment solves the above problem using a LUT. For example, the third lookup table can be used to implement a third mapping relationship between the multiplication result and its reciprocal. Of course, it is understood that this application does not exclude the possibility of implementing this inverse operation through a software module.

[0071] In some embodiments, the lookup table circuit 320 includes at least one basic lookup table circuit unit 410.

[0072] See Figure 4As shown, in some embodiments, the basic lookup table circuit unit 410 includes a logic circuit 411, an input terminal group 412, a control terminal group 413, and an output terminal group 414. The input terminal group 412 is connected to the memory 420 and inputs the lookup table data into the logic circuit 411. The logic circuit 411 selects the value corresponding to the index value (also called the address) input from the control terminal group 413 and outputs it from the output terminal group 414. The logic circuit 411 can be, for example, a logic gate circuit or a logic switch circuit. It is understood that in this application, a terminal group refers to a set of connection terminals, including one or more connection terminals. When the control terminal group 413 has A control terminals and the output terminal group 414 has B output terminals, the basic lookup table circuit unit 410 is referred to as A-input B-output.

[0073] The basic lookup table circuit unit 410 can perform lookup output based on the stored lookup table. Taking the first lookup table as an example, the lookup table also has A inputs and B outputs. The input data of the lookup table is the index value with a bit width of A bits, and the output data is the exponential function value with a bit width of B bits. The first lookup table in the storage area only stores the true value of the exponential function value, and the basic lookup table circuit unit is used to implement the mapping relationship between the index value and the true value of the exponential function value.

[0074] In some embodiments, the hardware acceleration circuitry may include at least one of the following memory areas.

[0075] The first static storage area is used to store the first lookup table, which is written into the first static storage area by the compiler.

[0076] The second static storage area is used to store the second lookup table, which is written into the second static storage area by the compiler.

[0077] The third static storage area is used to store the third lookup table, which is written into the third static storage area by the compiler.

[0078] The first, second, and third static storage areas are part of the hardware acceleration circuitry and can be located in static memory such as ROM or SRAM. The size of the static storage areas can be determined based on the value range of the data to be stored. For example, each of the first, second, and third static storage areas can store an 8-bit (2^8) LUT. Taking an 8-bit LUT as an example, the total number of entries in the three static storage areas is 3 × 2^3. 8 = 768 entries. If a 16-bit LUT is used, the total number of entries is 2. 16 =65536 entries. Using three 8-bit LUTs, compared to using one 16-bit LUT, effectively reduces the LUT's dependence on large storage space and reduces the area and cost of the lookup table circuit.

[0079] Understandably, in some embodiments, the lookup tables stored in the first static storage area, the second static storage area, and the third static storage area can be written into the aforementioned static storage areas by other control circuits or processors (such as CPUs or GPUs); or, each lookup table can also be written into dynamic memory, such as DRAM.

[0080] In some embodiments, the lookup table circuit may include a first memory area and a first basic lookup table circuit unit.

[0081] The first storage area is used to store the first lookup table.

[0082] The first basic lookup table circuit unit includes a first logic circuit, a first input terminal group, a first control terminal group, and a first output terminal group. The first input terminal group is connected to a first storage area. The first logic circuit is used to: respond to the first index value of the i-th element in the input data set input from the first control terminal group, and output the exponential function value corresponding to the i-th element from the first output terminal group based on the first lookup table.

[0083] In some embodiments, the lookup table circuit may include: a second storage area and a second basic lookup table circuit unit.

[0084] The second storage area is used to store the second lookup table.

[0085] The second basic lookup table circuit unit includes a second logic circuit, a second input terminal group, a second control terminal group, and a second output terminal group. The second input terminal group is connected to the second storage area. The second logic circuit is used to: respond to the second index value of the i-th element in the input data set input from the second control terminal group, and output the reciprocal of the exponential function value corresponding to the i-th element from the second output terminal group based on the second lookup table.

[0086] In some embodiments, the lookup table circuitry may include a third memory area and a third basic lookup table circuitry unit.

[0087] The third storage area is used to store the third lookup table.

[0088] The third basic lookup table circuit unit includes a third logic circuit, a third input group, a third control group, and a third output group. The third input group is connected to the third memory area. The third logic circuit is used to: respond to the index value of the multiplication result input from the third control group, and output a specific function value corresponding to the index value of the multiplication result from the third output group based on the third lookup table.

[0089] In a specific example, see Figure 5The hardware acceleration circuit includes: a lookup table circuit 510, a storage module 520, an adder 530, a first conversion circuit 540, a multiplier 550, and a second conversion circuit 560. The storage module 520 includes a first storage area 521 to a third storage area 523. The lookup table circuit 510 includes a first basic lookup table circuit unit 511 to a third basic lookup table circuit unit 513.

[0090] The first lookup table is stored in the first storage area 521, the second lookup table is stored in the second storage area 522, and the third lookup table is stored in the third storage area 523.

[0091] The first basic lookup table circuit unit 511 includes a first logic circuit 5111, a first input terminal group 5112, a first control terminal group 5113, and a first output terminal group 5114. The first input terminal group 5112 is connected to the first storage area 521; the first logic circuit 5111 is used to: in response to the index value of the input data input from the first control terminal group 5113, output the corresponding exponential function value stored in the first storage area 521 from the first output terminal group 5114.

[0092] The second basic lookup table circuit unit 512 includes a second logic circuit 5121, a second input terminal group 5122, a second control terminal group 5123, and a second output terminal group 5124. The second input terminal group 5122 is connected to the second storage area 522. The second logic circuit 5121 is used to: in response to the index value of the addition operation result input from the second control terminal group 5123, output the corresponding reciprocal stored in the second storage area 522 from the second output terminal group 5124.

[0093] The third basic lookup table circuit unit 513 includes a third logic circuit 5131, a third input group 5132, a third control group 5133, and a third output group 5134. The third input group 5132 is connected to the third storage area 523. The third logic circuit 5131 is used to: in response to the index value of the multiplication result input from the third control group 5123, output the corresponding reciprocal stored in the third storage area 523 from the third output group 5134.

[0094] For example, the first basic lookup table circuit unit 511 has N1 as input and N2 as output. The second basic lookup table circuit unit 512 has N1 as input and N4 as output. The third basic lookup table circuit unit 513 has N5 as input and N6 as output. In some implementations, the values ​​of N1, N2, N4, N5, and N6 are in the range [8, 12].

[0095] In some embodiments, the third conversion circuit 570 sequentially converts multiple input data points in the input dataset into index values ​​for those input data points. The first control terminal group 5113 sequentially inputs the index values ​​of the multiple input data points in the input dataset into the first logic circuit 5111. In response to the index values, the first logic circuit 5111 outputs the corresponding exponential function value from the first output terminal group 5114.

[0096] Adder 530 performs addition operations on multiple exponential function values ​​corresponding to multiple input data output from the first output group 5114 to obtain the addition result of multiple exponential function values.

[0097] The first conversion circuit 540 is used to convert the result of the addition operation into an index value with a corresponding bit width.

[0098] The second control terminal group 5123 inputs the index values ​​of multiple input data in the input dataset output by the third conversion circuit 570 into the second basic lookup table circuit unit 512. The second logic circuit 5121, in response to the index values ​​input from the second control terminal group 5123, outputs the reciprocal of the corresponding exponential function from the second output terminal group 5124.

[0099] The multiplier 550 performs a multiplication operation on the reciprocal of the exponential function value corresponding to the i-th input data output by the second output group 5124 and the converted data corresponding to the addition result output by the first conversion circuit 540 to obtain the multiplication result.

[0100] The second conversion circuit 560 converts the multiplication result into the index value of the multiplication result.

[0101] The third control terminal group 5133 inputs the index value of the multiplication result output by the second conversion circuit 560 into the third basic lookup table circuit unit 513. The third logic circuit 5131 responds to the index value input from the third control terminal group 5133 and outputs the reciprocal corresponding to the index value of the multiplication result from the third output terminal group 5134 to obtain the flexible maximum value corresponding to the i-th input data.

[0102] In some embodiments, time-sharing multiplexing can also be used to further reduce the lookup table's dependence on large storage space and / or multiple basic lookup table circuit units.

[0103] Specifically, the lookup table circuit includes: a reusable memory area and / or a reusable basic lookup table circuit unit.

[0104] In some embodiments, the storage area is used to store any one of the first lookup table, the second lookup table, or the third lookup table in each of the multiple time periods.

[0105] The basic lookup table circuit unit includes logic circuitry, an input group, a control group, and an output group. The input group is connected to the memory area. The logic circuitry is used to respond to the index value of the data input from the control group in each of the multiple time periods. Based on the corresponding lookup table, the output group outputs the data corresponding to the index value of the data.

[0106] For example, in the first time period, the logic circuit responds to the first index value of the i-th element in the input dataset input to the self-control terminal group, and outputs the exponential function value corresponding to the i-th element from the output terminal group based on a first lookup table. For example, in the second time period, the logic circuit responds to the second index value of the i-th element in the input dataset input to the self-control terminal group, and outputs the reciprocal of the exponential function value corresponding to the i-th element from the output terminal group based on a second lookup table. For example, in the third time period, the logic circuit responds to the index value of the multiplication result input to the self-control terminal group, and outputs the specific function value corresponding to the index value of the multiplication result from the third output terminal group based on a third lookup table.

[0107] The first lookup table, second lookup table, and third lookup table are written into the memory area by the processor during the compilation periods corresponding to the first, second, and third time periods, respectively. For example, when the basic lookup table circuit needs to use the first lookup table, the processor writes the first lookup table into this memory area. Since only one memory area needs to be configured for time-sharing storage of any one of the first, second, and third lookup tables, the space occupied by the memory area on the hardware acceleration circuit is effectively reduced, thus reducing hardware costs. It is understood that in another embodiment, the first, second, and third lookup tables can also be independently configured in three different memory areas.

[0108] See Figure 6 The hardware acceleration circuit 600 includes a storage module 620, a lookup table circuit 610, an adder 640, a multiplier 660, a first conversion circuit 650, a second conversion circuit 670, and a third conversion circuit 630. This embodiment is similar to... Figure 5 The hardware acceleration circuit shown is similar to 400, but the difference is:

[0109] The lookup table circuit 610 includes a basic lookup table circuit unit 611.

[0110] The basic lookup table circuit unit 611 includes: a logic circuit 6111, an input terminal group 6112, a control terminal group 6113, and an output terminal group 6114. The input terminal group 6112 is connected to the storage module 620. The logic circuit 6111 is used to: in response to a first index value of the i-th input data input from the control terminal group 6113 (this first index value is obtained by the third conversion circuit 630 after converting the i-th input data), output the exponential function value corresponding to the i-th input data from the output terminal group 6114 based on the first lookup table. In a second time period before or after the first time period, in response to a second index value of the i-th input data input from the control terminal group 6113, output the reciprocal of the exponential function value corresponding to the second index value from the output terminal group 6114 based on the second lookup table. In the third time period following the second time period, in response to the index value of the multiplication result input by the self-control terminal group 6113, the output terminal group 6114 outputs the reciprocal of the index value of the multiplication result corresponding to the index value of the multiplication result, i.e., the specific function value, based on the third lookup table.

[0111] Since only one storage area needs to be configured for time-sharing storage of any one of the first, second, and third lookup tables, the storage space occupied by the lookup tables is effectively reduced, thus reducing hardware costs.

[0112] Furthermore, the aforementioned hardware acceleration circuit 600 also includes a first selector 6113 and a second selector 630. The first selector 6113 is used to selectively input the first and second index values ​​of the i-th element output by the third conversion circuit 630 to the control terminal group 6113, and to input the index value of the multiplication result output by the second conversion circuit 670 to the control terminal group 6113. The second selector 630 is used to selectively transmit the data output by the output terminal group 6114 to either the adder 640 or the multiplier 660.

[0113] By reusing the basic lookup table circuit unit, only one basic lookup table circuit needs to be configured, which can effectively reduce the area and cost of the lookup table circuit.

[0114] In some embodiments, the bit width of the output data of the first lookup table, the second lookup table, and the third lookup table is between 8 and 12. In some embodiments, the bit width of the output data of the second lookup table is greater than the bit width of the output data of the first lookup table and / or the third lookup table.

[0115] Understandably, in some specific embodiments, the basic lookup table circuit unit can be configured to have input states with different bit widths and / or output states with different bit widths at different times, in order to adapt to situations where the input or output data bit widths of the first lookup table, the second lookup table, and the third lookup table are different.

[0116] See Figure 7In addition to the lookup table circuit 310, adder 320 and multiplier 330, the hardware acceleration circuit 700 may further include a subtractor 740.

[0117] The subtractor 740 is used to output the result of the subtraction operation between the i-th element in the initial dataset and the maximum value (max) in the initial dataset, so as to obtain the input dataset.

[0118] Referring to equations (1) and (2), when the specific function is the Softmax function, the parameters of the function include: x i -x max x k -x max i and k are integers greater than or equal to 0, and i and k may be equal or unequal. To facilitate determining the values ​​of these two parameters, subtraction can be performed using a subtractor located on the hardware acceleration circuitry.

[0119] In some embodiments, the accuracy of the obtained specific function values ​​can be further improved by setting the format of at least some of the data, or the dependence of the hardware acceleration circuit on a large storage space can be reduced, or the area and cost of the lookup table circuit can be controlled.

[0120] For example, the i-th element in the input dataset is data with a bit width of N1 bits, where the N1-bit data includes an integer with a bit width of M1 bits and a fractional number with a bit width of M2 bits. The sum of M1 and M2 can be equal to N1. For example, when N1 is 8, then M1 can be 3, 4, 5, 6, 7, 8, etc. For example, M1 can be greater than M2 to ensure that there are enough integer bits.

[0121] The exponential function value corresponding to the i-th element is a data with a width of N2 bits, where N2 bits includes an integer with a width of M3 bits and a decimal with a width of M4 bits. The sum of M3 and M4 can be equal to N2. For example, when N2 is 8, then M4 can be 3, 4, 5, 6, 7, 8, etc. For example, M3 can be less than M4. Understandably, through the subtraction operation of the above subtractor, the values ​​of each element in the input dataset are negative, thus normalizing the exponential function value of that element to the base e to the range (0, 1]. For example, M3 can be 0, and M4 can be equal to N2 to obtain sufficient precision.

[0122] It should be noted that the first conversion circuit can be used to convert the addition result output by the adder into data with a smaller bit width of N3 bits for subsequent multiplication operations. This conversion avoids the problem of excessive computational load on the multiplier caused by an excessively wide adder output. The first conversion circuit may include, for example, a shift circuit, and the converted data can be extracted from a specified position of the original data. The N3 bits of data include an integer with a bit width of M5 bits and a decimal with a bit width of M6 bits; the sum of M5 and M6 can be equal to N3. For example, when N3 is 8, M5 can be 3, 4, 5, 6, 7, 8, etc. Alternatively, M5 can be greater than M6 to ensure sufficient integer bits to represent addition results with a large value range.

[0123] It is understandable that any two values ​​among M1, M2, M3, M4, M5, and M6 can be the same or different. The specific values ​​can be set according to the range of the data being represented. For example, if the range of integer values ​​is small, more decimal places can be allocated to obtain higher data precision. If the range of integer values ​​is large, more integer places can be allocated.

[0124] In some embodiments, N1, N2, and N3 can take values ​​of 8, 9, 10, 11, or 12, and they can be the same or different. In one specific embodiment, N1, N2, and N3 can all be 8, which reduces the dependence on large storage space, reduces the space occupied by hardware acceleration circuitry, lowers costs, and has good compatibility with commonly used 8-bit hardware circuits. M1, M2, M3, M4, M5, and M6 are 5, 3, 0, 8, 7, and 1, respectively, which can meet the precision requirements of the data of the i-th element, the exponential function value corresponding to the i-th element, and the addition operation result while avoiding excessively wide data bits.

[0125] In some embodiments, the reciprocal of the exponential function value corresponding to the second index value of the i-th element is data with a width of N4 bits, where the N4 bits of data include an integer with a width of M7 bits and a fractional number with a width of M8 bits. N4 can be 6, 8, 9, 10, 12, 14, or 16, etc., and the sum of M7 and M8 can be equal to N4. For example, when N4 is 10, M7 can be 3, 4, 5, 6, 7, 8, 9, 10, etc. For example, M7 can be greater than M8. If M8 is 0, M7 is equal to N4. When the exponential function value is normalized to the range (0, 1), its reciprocal value is greater than 1, its value range is large, and it participates in subsequent multiplication operations; therefore, its decimal places can be ignored.

[0126] In some embodiments, the second conversion circuit is used to convert the multiplication result output by the multiplier into data with a smaller bit width of N5 bits. The second conversion circuit may include, for example, a shift circuit, and the converted data may be extracted from a specified position of the original data. The N5 bits of data include an integer with a bit width of M9 bits and a decimal with a bit width of M10 bits. N5 can be 6, 8, 9, 10, 12, 14, or 16, etc., and the sum of M9 and M10 can be equal to N5. For example, when N5 is 12, M7 can be 5, 6, 7, 8, 9, 10, 11, 12, etc. In some embodiments, M9 is greater than M10 to ensure that there are enough integer bits to represent the multiplication result with a large value range.

[0127] In some embodiments, the specific function value corresponding to the index value of the multiplication result is data with a bit width of N6 bits, where the N6 bits of data include an integer with a bit width of M11 bits and a fraction with a bit width of M12 bits. The sum of M11 and M12 can be equal to N2. For example, when N6 is 8, then M12 can be 3, 4, 5, 6, 7, 8, etc. In some embodiments, M11 is less than M12. It is understood that the output of the specific function value LUT is the reciprocal of the multiplication result, and its value range is in the range (0, 1). Thus, in a specific embodiment, M11 can be 0, and M12 is equal to N6 to obtain sufficient precision.

[0128] In some embodiments, the third conversion circuit is used to convert the i-th element in the input dataset into an index value. For example, the i-th element is converted into 8-bit wide data, where, to ensure precision, the 8-bit data may include 3 decimal bits. The third conversion circuit may include a shift circuit.

[0129] Because the Softmax function has a wide dynamic range, related technologies mostly implement this function using software modules. Some embodiments of this application provide a hardware circuit solution that is essentially based on 8 bits, effectively balancing important metrics such as circuit cost, power consumption, bandwidth, performance, and data accuracy.

[0130] Another aspect of this application provides a method for accelerating data processing.

[0131] Figure 8 This is a schematic flowchart illustrating a data processing acceleration method according to an embodiment of this application.

[0132] See Figure 8 The above-mentioned data processing acceleration method may include: operations S810 to S830.

[0133] In operation S810, the value of the exponential function corresponding to the i-th element in the input dataset is obtained, and the reciprocal of the value of the exponential function corresponding to the i-th element is obtained, where i is an integer greater than or equal to 1.

[0134] In this embodiment, the exponential function value corresponding to the i-th element in the input dataset can be obtained through lookup tables, arithmetic circuits, or external data. For example, the exponential function value corresponding to the i-th element can be determined based on a lookup table stored in the hardware acceleration circuit of the artificial intelligence chip. Alternatively, the artificial intelligence chip can send the i-th element to the processor, which calculates the exponential function value corresponding to the i-th element and sends that value back to the artificial intelligence chip. For example, the exponential function value corresponding to the i-th element can be determined based on a logic operation circuit set on the hardware acceleration circuit. Similarly, the reciprocal of the exponential function value corresponding to the i-th element can be obtained through lookup tables, arithmetic circuits, etc.

[0135] In operation S820, the result of the addition operation corresponding to the exponential function values ​​of at least some elements in the input dataset is obtained.

[0136] For example, an adder can be used to perform addition operations on the exponential function values ​​corresponding to at least some elements in the input dataset to obtain the addition result. Alternatively, the addition operation can be performed by a CPU or similar device.

[0137] In operation S830, based on the reciprocal of the exponential function value corresponding to the i-th element and the result of the addition operation, the specific function value corresponding to the i-th element is obtained.

[0138] In this embodiment, the specific function can be a nonlinear function, which can be expressed using an exponential function. Specific functions include, but are not limited to, the Softmax function, the Sigmoid function, and the TanH function. For example, the expression for the Softmax function can be found in equations (1) and (2). Taking equation (2) as an example, the Softmax function is inversely operated on. In operation S530, the reciprocal of the exponential function value corresponding to the i-th element and the result of the addition operation need to be multiplied to obtain the multiplication result. Then, the multiplication result is inversely operated on.

[0139] Figure 9 This is a schematic diagram illustrating the inverse operation in an embodiment of this application.

[0140] See Figure 9 This is a graph of the function y = 1 / x. When x is greater than zero, the larger the value of x, the smaller the value of the function y. The inverse operation of the Softmax function corresponding to the i-th element can be expressed as: The larger the value, the better σ(x) becomes. i The smaller σ(x) is. i The function values ​​can range from 1 to N, where N is an integer greater than 1. In some scenarios, more attention is paid to σ(x). i The case that approaches 1. Therefore, it can be ignored. In cases where the value is too large, more storage space is allocated to σ(x). i A value close to 1 helps improve the accuracy of the determined Softmax function value.

[0141] It should be noted that at least part of the result of a multiplication operation and its inverse operation can be determined by the processor executing the corresponding arithmetic logic. Furthermore, at least part of the result of a multiplication operation and its inverse operation can also be determined by a hardware accelerator using a lookup table.

[0142] In some embodiments, obtaining a specific function value corresponding to the i-th element based on the reciprocal of the exponential function value corresponding to the i-th element and the result of the addition operation may include the following operations: First, obtain the result of multiplying the reciprocal of the exponential function value corresponding to the i-th element and the result of the addition operation. Then, obtain the reciprocal of the multiplication result to obtain the flexible maximum function value corresponding to the i-th element.

[0143] Here, data A can be data A itself, or it can be data obtained after transforming data A. For example, the reciprocal can be data output by a lookup table circuit, or it can be data obtained after transforming the output of a lookup table circuit. For instance, if the result of a multiplication operation is 32-bit data, in order to reduce the number of entries in the LUT, the 32-bit data can be converted into data with a smaller bit width.

[0144] In some embodiments, obtaining the exponential function value corresponding to the i-th element in the input dataset includes: obtaining the exponential function value corresponding to the i-th element in the input dataset through a first mapping relationship pre-stored in a lookup table module. The first mapping relationship can be configured in the hardware acceleration circuit in the form of a first lookup table.

[0145] In some embodiments, obtaining the reciprocal of the exponential function value corresponding to the i-th element includes: obtaining the reciprocal of the exponential function value corresponding to the i-th element through a second mapping relationship pre-stored in a lookup table module. The second mapping relationship can be configured in the hardware acceleration circuit in the form of a second lookup table.

[0146] In some embodiments, obtaining the reciprocal of the multiplication result to obtain the flexible maximum function value corresponding to the i-th element includes: obtaining the reciprocal corresponding to the multiplication result through a pre-stored third mapping relationship in the lookup table module to obtain the flexible maximum function value corresponding to the i-th element. The third mapping relationship can be configured in the hardware acceleration circuit in the form of a third lookup table.

[0147] In some embodiments, before obtaining the exponential function value corresponding to the i-th element in the input dataset, the method may further include the following operation: subtracting the i-th element in the initial dataset from the maximum value in the initial dataset to obtain the i-th element in the input dataset.

[0148] Figure 10 This is a logic diagram of a data processing acceleration method illustrated in an embodiment of this application.

[0149] See Figure 10 Combined with Equation (2), the Softmax function is transformed to obtain two parts: the exponential function e x The sum and the exponential function e x The reciprocal of.

[0150] For example, by transforming the formula of the Softmax function, the exponential function e x The range becomes (0, 1], corresponding to the range of x in [-10, 0]. The range of x is divided into 256 points, and the exponential function value is calculated for each point. This allows all these results to be quantized and mapped to the range [0, 1]. All these quantized values ​​are then filled into an exponential function value LUT, achieving pre-storage of the exponential function values ​​using an 8-bit LUT.

[0151] This allows the search for the exponential function value in a pre-stored exponential function value LUT based on the value of x. For example, by using a lookup table, the search can find an element x in the exponential function value LUT that matches at least some (or all) of the elements in the array. i -x max The corresponding exponential function values.

[0152] Next, we can use an accumulator to convert all elements x in the array. i -x max The corresponding exponential function values ​​are added together to obtain all elements x in 32-bit format. i -x max The cumulative value of the exponential function.

[0153] Then, the accumulated value in 32-bit format is converted to obtain the accumulated value represented by N bits.

[0154] See Figure 10Furthermore, the reciprocal of the exponential function value can be found in the exponential function value reciprocal LUT.

[0155] Then, the accumulated value of the exponential function is multiplied by its reciprocal to obtain the reciprocal of the specific function value. For example, multiplying the accumulated value of the 8-bit exponential function by its reciprocal (10 bits) yields the reciprocal of the X-bit Softmax function. Here, X can be 8, 12, 16, or 32, etc.

[0156] Next, the reciprocal of a specific function value can be used to look up the Softmax function value in a pre-stored LUT of specific function values ​​from X bits to 8 bits.

[0157] It should be noted that, although Figure 10 The table involves three LUTs, but the number of entries in three small LUTs (such as 8-bit to 8-bit LUTs) is far less than the number of entries in one large table (such as 16-bit LUTs).

[0158] In one specific embodiment, the exponential function value LUT can be an 8-bit to 8-bit LUT, including 256 entries. The reciprocal exponential function value LUT can be an 8-bit to 10-bit LUT, including 1024 entries. The specific function value LUT can be a 12-bit to 8-bit LUT, including 4096 entries. The three LUTs together include 5376 entries. A single 16-bit LUT includes 65536 entries. Therefore, the number of entries in the LUT is significantly reduced, effectively lowering hardware costs while still meeting accuracy requirements.

[0159] For example, using the COCO2017 dataset, when running the technical solution shown in the previous specific embodiment on a DetectionTransformer neural network with 6 layers of encoder + 6 layers of decoder, the mean average precision (mAP) is 35.2. This demonstrates that this embodiment effectively reduces the LUT's dependence on large storage space while meeting the accuracy requirements.

[0160] In this embodiment, a specific function (such as the Softmax function) is inversely operated on, which allows the use of a LUT with fewer bits. This balances the power consumption, bandwidth, performance, and accuracy of implementing the specific function in hardware, thus meeting the needs of neural networks.

[0161] To further reduce the reliance on large storage spaces for intermediate results generated during the storage of lookup tables or function values, specific data formats can be used for data storage.

[0162] In some embodiments, the bit width of the i-th element in the input dataset, the exponential function value corresponding to the i-th element in the input dataset, and the addition result is the same. For example, an 8-bit storage space can be used to store the i-th element, the exponential function value corresponding to the i-th element in the input dataset, and the addition result.

[0163] After performing the reciprocal operation on the exponential function value, the bit width of the reciprocal corresponding to the i-th element can be set according to the required data precision. In some embodiments, to improve the precision of the obtained Softmax function value, a storage space with more bits can be used to store the reciprocal. Specifically, the bit width of the reciprocal corresponding to the i-th element is greater than the bit width of the i-th element in the input dataset. For example, the bit width of the i-th element can be 8 bits, and the bit width of the reciprocal can be greater than 8 bits to improve precision, such as 8 bits, 9 bits, 10 bits, 12 bits, 13 bits, 14 bits, etc.

[0164] Furthermore, the data format can include the number of digits and the number of digits where the decimal point is located. For example, the Q5.3 format can be used to represent 8 bits of data, where 5 bits are integers and 3 bits are decimals. When storing data for a specific parameter, it is not necessary to store the decimal position information; the decimal position information can be determined directly based on the data format set for that parameter.

[0165] In some embodiments, the bit width of the i-th element is equal to the bit width of the specific function value corresponding to the i-th element. For example, both the bit width of the i-th element and the bit width of the specific function value can be 8 bits.

[0166] To convert intermediate results or specific function values ​​into a preset data format, bit shifting can be used. For example, if the intermediate result is 32 bits and the data format of the intermediate result is 8 bits, then the 8 bits starting from the most significant bit of the intermediate result can be taken, and the remaining bits can be discarded.

[0167] For example, after obtaining the specific function value corresponding to the i-th element, the bit width of the specific function value is transformed. If the specific function value is 12 bits, 8 bits can be extracted from it.

[0168] In some embodiments, the i-th element is data with a bit width of N1 bits, where the N1 bits of data include an integer with a bit width of M1 bits and a fractional number with a bit width of M2 bits. The exponential function value corresponding to the i-th element is data with a bit width of N2 bits, where the N2 bits of data include an integer with a bit width of M3 bits and a fractional number with a bit width of M4 bits. The result of the addition operation is data with a bit width of N3 bits, where the N3 bits of data include an integer with a bit width of M5 bits and a fractional number with a bit width of M6 bits.

[0169] In this context, any two values ​​of M1, M2, M3, M4, M5, and M6 are the same or different. For example, N1, N2, and N3 are all 8. The values ​​of M1, M2, M3, M4, M5, and M6 are all different. For example, the values ​​of M1 to M12 include at least one of the following: M1 greater than M2, M3 less than M4, M5 greater than M6, M7 greater than M8, M9 greater than M10, M11 less than M12, M9 greater than or equal to M5, M9 less than or equal to M7, M10 greater than or equal to M6, and M10 greater than or equal to M8.

[0170] In some embodiments, the bit width of the reciprocal of the exponential function value corresponding to the i-th element is greater than the bit width of the i-th element in the input dataset.

[0171] In some embodiments, the bit width of the reciprocal of the exponential function value corresponding to the i-th element is greater than the bit width of the exponential function value corresponding to the i-th element and the bit width of the specific function value.

[0172] In some embodiments, the bit width of the i-th element is equal to the bit width of the specific function value corresponding to the i-th element.

[0173] For example, to ensure the precision of the i-th input element and reduce the risk of saturation, M1 can be set to 5 and M2 to 3. Since the exponential function value corresponding to the i-th element is usually a decimal, the numerical format of the exponential function value can be set to have multiple decimal places to improve data precision. For example, the format of the exponential function value could be Q0.8. If the sum of multiple exponential function values ​​is large, M5 can be set to be greater than M3, or M5 greater than M1. Furthermore, to ensure the precision of the sum of multiple exponential function values, the number of decimal places can be set, such as M6 being 1, 2, or 3.

[0174] In some embodiments, the reciprocal of the exponential function value corresponding to the i-th element is a data with a bit width of N4 bits, where the N4 bits include an integer with a bit width of M7 bits and a decimal with a bit width of M8 bits. Any two of M1, M2, M7, and M8 can be the same or different. For example, if the exponential function value corresponding to the i-th element has a wide distribution range, more entries are needed to improve the precision of the reciprocal obtained through the lookup table. For instance, the exponential function reciprocal LUT could be a 10-bit to 8-bit LUT, a 12-bit to 8-bit LUT, or a 14-bit to 8-bit LUT. Furthermore, if the reciprocal of the exponential function value corresponding to the i-th element is mainly distributed in the interval greater than 1, then only the integer part needs to be set, and the decimal part is not required.

[0175] In some embodiments, the result of the multiplication operation is data with a bit width of N5 bits, wherein the N5 bits of data include an integer with a bit width of M9 bits and a fraction with a bit width of M10 bits.

[0176] For example, M9 is greater than or equal to M5, M9 is less than or equal to M7, M10 is greater than or equal to M6, and M10 is greater than or equal to M8. After performing a multiplication operation, to improve the accuracy of the result, a larger number of bits can be assigned to the multiplication result. For example, the number of bits in the multiplication result can be greater than or equal to the number of bits in the sum of a specified function value, or the number of bits in the reciprocal of an exponential function value. Furthermore, to reduce the number of entries included in the LUT and thus lower hardware costs, the number of bits in the multiplication result should not be too large. For example, the number of bits in the multiplication result can be 10 bits, 12 bits, or 14 bits. The number of bits in the multiplication result can include integer bits and decimal bits.

[0177] In some embodiments, the bit width of the multiplication result is greater than the bit width of the reciprocal of the exponent function value corresponding to the i-th element and / or the bit width of the addition result. This can improve the accuracy of the multiplication result.

[0178] In some embodiments, the reciprocal of the multiplication result is a data with a width of N6 bits, where the N6 bits include an integer with a width of M11 bits and a fraction with a width of M12 bits. M11 is less than M1, M11 is less than M9, M12 is greater than M2, and M12 is greater than M10. For example, N6 can be 8 bits, 10 bits, or 12 bits, etc.

[0179] Figure 11 This is a schematic diagram illustrating the format of each data item during the execution of the data processing acceleration method shown in the embodiments of this application.

[0180] See Figure 11 x i -x max The value can be in Q5.3 format, the output value of the exponential function value LUT can be in Q0.8 format, the accumulated value of the exponential function value can be in Q7.1 format, the output value of the reciprocal value of the exponential function value LUT can be in Q10.0 format, the result of multiplication operation can be in Q9.3 format, and the output value of the specific function value LUT can be in Q0.8 format.

[0181] It should be noted that the above formats are merely illustrative examples, and other formats may be used depending on the required precision. They should not be construed as limiting this application.

[0182] The relevant features of the data processing acceleration method in this application embodiment can be found in the relevant content of the foregoing hardware acceleration circuit embodiment, and will not be repeated here.

[0183] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a data processing acceleration device, an electronic device, and corresponding embodiments.

[0184] Figure 12 This is a schematic diagram of the structure of the data processing acceleration device shown in the embodiments of this application.

[0185] See Figure 12 The data processing acceleration device 1200 includes a function value and reciprocal acquisition module 1210, an addition operation module 1220, and a multiplication operation module 1230.

[0186] The function value and reciprocal acquisition module 1210 is used to obtain the exponential function value corresponding to the i-th element in the input dataset, and to obtain the reciprocal of the exponential function value corresponding to the i-th element, where i is an integer greater than or equal to 1.

[0187] The addition module 1220 is used to obtain the addition result of the exponential function values ​​corresponding to at least some elements in the input dataset.

[0188] The multiplication module 1230 is used to obtain a specific function value corresponding to the i-th element based on the reciprocal of the exponential function value corresponding to the i-th element and the result of the addition operation.

[0189] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0190] The data processing acceleration method according to the embodiments of this application can be applied to artificial intelligence accelerators.

[0191] Figure 13 This is a schematic diagram of the structure of an artificial intelligence accelerator according to an embodiment of this application. See also... Figure 13 The AI ​​accelerator 1300 includes a memory 1310 and a processor 1320.

[0192] The artificial intelligence accelerator 1320 can be a general-purpose processor, such as a CPU (Central Processing Unit), or an artificial intelligence processor (IPU) for performing artificial intelligence operations. Artificial intelligence operations can include machine learning operations, neuromorphic operations, etc. Machine learning operations include neural network operations, k-means operations, support vector machine operations, etc. The artificial intelligence processor can, for example, include one or a combination of GPU (Graphics Processing Unit), DLA (Deep Learning Accelerator), NPU (Neural-Network Processing Unit), DSP (Digital Signal Processing Unit), Field-Programmable Gate Array (FPGA), and Application Specific Integrated Circuit (ASIC). This application does not limit the specific type of processor.

[0193] Memory 1310 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1320 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1310 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1310 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0194] The memory 1310 stores executable code, which, when processed by the processor 1320, can cause the processor 1320 to execute part or all of the methods described above.

[0195] In one possible implementation, the artificial intelligence accelerator may include multiple processors, each of which can independently run various assigned tasks. This application does not limit the processors or the tasks they run.

[0196] It is understood that, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in software program modules.

[0197] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage module can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0198] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0199] In one possible implementation, an artificial intelligence chip is also disclosed, which includes the aforementioned hardware acceleration circuitry.

[0200] In one possible implementation, a board is also disclosed, which includes a storage device, an interface device, a controller, and the aforementioned artificial intelligence chip; wherein the artificial intelligence chip is connected to the storage device, the controller, and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the artificial intelligence chip and external devices; and the controller is used to monitor the status of the artificial intelligence chip.

[0201] In one possible implementation, an electronic device is disclosed that includes the aforementioned artificial intelligence chip. The electronic device includes data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, dashcams, navigators, sensors, cameras, servers, cloud servers, cameras, camcorders, projectors, watches, headphones, mobile storage, wearable devices, vehicles, home appliances, and / or medical devices. Vehicles include airplanes, ships, and / or vehicles; home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs.

[0202] Figure 14 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0203] See Figure 14 The electronic device 1400 includes a processor 1410 and an artificial intelligence chip 1420.

[0204] The processor 1410 is used to send at least one of a first lookup table, a second lookup table, or a third lookup table to the artificial intelligence chip 1420. The first lookup table is used to realize a first mapping relationship between the i-th element in the input dataset and the value of the exponential function. The second lookup table is used to realize a second mapping relationship between the i-th element in the input dataset and the reciprocal of the exponential function value. The third lookup table is used to realize a third mapping relationship between a multiplication result and its reciprocal. The multiplication result is obtained by multiplying the reciprocal of the exponential function value corresponding to the i-th element and the addition result. The addition result is the addition result of the exponential function values ​​corresponding to at least some elements in the input dataset.

[0205] The artificial intelligence chip 1420 is used to perform the method as described above based on at least one of a first lookup table, a second lookup table, or a third lookup table.

[0206] The processor 1410 can be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0207] The AI ​​chip 1420 may include a fully connected layer circuit, a loss function layer circuit, and a hardware accelerator as shown above.

[0208] The fully connected layer circuit is used to transmit the i-th element of the input dataset to the hardware acceleration circuit and to receive the loss gradient value of the i-th element of the input dataset from the hardware acceleration circuit.

[0209] The loss function layer circuit is used to transmit a loss value for the specified function value to the hardware acceleration circuit in response to the specified function value from the hardware acceleration circuit.

[0210] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0211] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0212] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A hardware acceleration circuit, characterized in that, include: A lookup table circuit is configured to, in response to the i-th element in the input dataset, output the exponential function value corresponding to the first index value of the i-th element based on a first lookup table, and output the reciprocal of the exponential function value corresponding to the second index value of the i-th element based on a second lookup table, where i is an integer greater than or equal to 1; An adder is used to output the result of an addition operation with the exponential function values ​​corresponding to at least some elements in the input dataset; The multiplier is used to output the multiplication result of the exponential function value corresponding to the i-th element and the addition operation result to obtain the reciprocal of the specific function value corresponding to the i-th element, so as to obtain the specific function value corresponding to the i-th element.

2. The hardware acceleration circuit according to claim 1, characterized in that, The lookup table circuit is also used to respond to the result of the multiplication operation by outputting a specific function value corresponding to the index value of the result of the multiplication operation based on the third lookup table.

3. The hardware acceleration circuit according to claim 2, characterized in that: The i-th element is data with a bit width of N1 bits, wherein the N1-bit data includes an integer with a bit width of M1 bits and a decimal with a bit width of M2 bits; The exponential function value corresponding to the i-th element is data with a bit width of N2 bits, wherein the N2 bits of data include an integer with a bit width of M3 bits and a decimal with a bit width of M4 bits; The hardware acceleration circuit further includes: a first conversion circuit, used to convert the addition result into data with a bit width of N3 bits as the addition result, wherein the N3 bits of data include an integer with a bit width of M5 bits and a decimal with a bit width of M6 bits, and any two of M1, M2, M3, M4, M5 and M6 can be the same or different.

4. The hardware acceleration circuit according to claim 3, characterized in that: The reciprocal of the exponential function value corresponding to the second index value of the i-th element is data with a bit width of N4 bits, wherein the N4 bits of data include an integer with a bit width of M7 bits and a decimal with a bit width of M8 bits; The hardware acceleration circuit further includes a second conversion circuit for converting the multiplication result into N5-bit data, wherein the N5-bit data includes an integer with a bit width of M9 bits and a fraction with a bit width of M10 bits.

5. The hardware acceleration circuit according to claim 4, characterized in that, The specific function value corresponding to the index value of the multiplication result is data with a bit width of N6 bits, wherein the N6 bits of data include an integer with a bit width of M11 bits and a fraction with a bit width of M12 bits. The values ​​of M1 to M12 include at least one, more, or all of the following: M1 is greater than M2; M3 is less than M4; M5 is greater than M6; M7 is greater than M8; M9 is greater than M10; M11 is less than M12.

6. The hardware acceleration circuit according to claim 2, characterized in that: The bit width of the output data of the first lookup table, the second lookup table, and the third lookup table is between 8 and 12. and / or The bit width of the output data of the second lookup table is greater than the bit width of the output data of the first lookup table and / or the third lookup table.

7. The hardware acceleration circuit according to claim 2, characterized in that, The lookup table circuit includes: A first storage area and a first basic lookup table circuit unit, wherein the first storage area is used to store a first lookup table, and the first basic lookup table circuit unit includes a first logic circuit, a first input terminal group, a first control terminal group, and a first output terminal group, wherein the first input terminal group is connected to the first storage area; the first logic circuit is used to: in response to a first index value of the i-th element in the input data set input from the first control terminal group, output an exponential function value corresponding to the i-th element from the first output terminal group based on the first lookup table; and / or A second storage area and a second basic lookup table circuit unit, wherein the second storage area is used to store a second lookup table, and the second basic lookup table circuit unit includes a second logic circuit, a second input terminal group, a second control terminal group, and a second output terminal group, wherein the second input terminal group is connected to the second storage area; the second logic circuit is used to: in response to a second index value of the i-th element in the input data set input from the second control terminal group, output the reciprocal of the exponential function value corresponding to the i-th element from the second output terminal group based on the second lookup table; and / or A third storage area and a third basic lookup table circuit unit, wherein the third storage area is used to store a third lookup table, and the third basic lookup table circuit unit includes a third logic circuit, a third input terminal group, a third control terminal group, and a third output terminal group, wherein the third input terminal group is connected to the third storage area; the third logic circuit unit is used to: in response to the index value of the multiplication result input from the third control terminal group, based on the third lookup table, output a specific function value corresponding to the index value of the multiplication result from the third output terminal group.

8. The hardware acceleration circuit according to claim 2, characterized in that, The lookup table circuit includes: A storage area is used to store any one of the first lookup table, the second lookup table, or the third lookup table in each of multiple time periods; or, it includes multiple storage areas for storing the first lookup table to the third lookup table respectively. The basic lookup table circuit unit includes logic circuitry, an input group, a control group, and an output group. The input group is connected to the storage area. The basic lookup table circuit is used to respond to the index value of the data input from the control group in each of multiple time periods, and output the data corresponding to the index value of the data from the output group based on the corresponding lookup table.

9. The hardware acceleration circuit according to claim 3, characterized in that, Also includes: A subtractor is used to output the result of subtracting the i-th element in the initial dataset from the maximum value in the initial dataset, so as to obtain the input dataset; The third conversion circuit is used to convert the i-th element in the input dataset into an index value.

10. An artificial intelligence chip, comprising the hardware acceleration circuitry as described in any one of claims 1 to 9.

11. A data processing acceleration method, characterized in that, The method, applied to the hardware acceleration circuit according to any one of claims 1 to 9, comprises: Obtain the value of the exponential function corresponding to the i-th element in the input dataset, and obtain the reciprocal of the value of the exponential function corresponding to the i-th element, where i is an integer greater than or equal to 1; Obtain the addition result of the exponential function values ​​corresponding to at least some elements in the input dataset; Based on the reciprocal of the exponential function value corresponding to the i-th element and the result of the addition operation, the specific function value corresponding to the i-th element is obtained.

12. The method according to claim 11, characterized in that, The process of obtaining a specific function value corresponding to the i-th element based on the reciprocal of the exponential function value corresponding to the i-th element and the result of the addition operation includes: Obtain the reciprocal of the exponential function value corresponding to the i-th element and the result of the multiplication operation of the addition operation result; Obtain the reciprocal of the multiplication result to obtain the maximum flexible function value corresponding to the i-th element.

13. The method according to claim 11, characterized in that: Obtaining the exponential function value corresponding to the i-th element in the input dataset includes: obtaining the exponential function value corresponding to the i-th element in the input dataset through a pre-stored first mapping relationship in the lookup table module; and / or Obtaining the reciprocal of the exponential function value corresponding to the i-th element includes: obtaining the reciprocal of the exponential function value corresponding to the i-th element through a pre-stored second mapping relationship in the lookup table module; and / or Obtaining the reciprocal of the multiplication result to obtain the flexible maximum function value corresponding to the i-th element includes: obtaining the reciprocal of the multiplication result through a pre-stored third mapping relationship in the lookup table module to obtain the flexible maximum function value corresponding to the i-th element.

14. The method according to claim 12, characterized in that: The i-th element is data with a bit width of N1 bits, wherein the N1-bit data includes an integer with a bit width of M1 bits and a decimal with a bit width of M2 bits; The exponential function value corresponding to the i-th element is data with a bit width of N2 bits, wherein the N2 bits of data include an integer with a bit width of M3 bits and a decimal with a bit width of M4 bits; The result of the addition operation is data with a bit width of N3 bits, wherein the N3 bits of data include an integer with a bit width of M5 bits and a fraction with a bit width of M6 bits; The reciprocal of the exponential function value corresponding to the i-th element is data with a bit width of N4 bits, wherein the N4 bits of data include an integer with a bit width of M7 bits and a decimal with a bit width of M8 bits; The result of the multiplication operation is data with a bit width of N5 bits, wherein the N5 bits of data include an integer with a bit width of M9 bits and a decimal with a bit width of M10 bits. The reciprocal of the multiplication result is a data with a width of N6 bits, wherein the N6 bits of data include an integer with a width of M11 bits and a fraction with a width of M12 bits. in, The values ​​of M1 to M12 include at least one, more or all of the following: M1 is greater than M2, M3 is less than M4, M5 is greater than M6, M7 is greater than M8, M9 is greater than M10, M11 is less than M12, M9 is greater than or equal to M5, M9 is less than or equal to M7, M10 is greater than or equal to M6, and M10 is greater than or equal to M8.

15. The method according to claim 12, characterized in that: The bit width of the multiplication result is greater than the bit width of the reciprocal of the exponent function value corresponding to the i-th element and \or the bit width of the addition result; and / or The bit width of the reciprocal of the exponential function value corresponding to the i-th element is greater than the bit width of the i-th element in the input dataset; and / or The bit width of the reciprocal of the exponential function value corresponding to the i-th element is greater than the bit width of the exponential function value corresponding to the i-th element and the bit width of the specific function value; and / or The bit width of the i-th element is equal to the bit width of the specific function value corresponding to the i-th element.

16. The method according to any one of claims 11 to 15, characterized in that, The data bit width of the i-th element in the input dataset, the exponential function value corresponding to the i-th element in the input dataset, and the addition result is between 8 and 12.

17. The method according to any one of claims 11 to 15, characterized in that, The method further includes: Before obtaining the exponential function value corresponding to the i-th element in the input dataset, the i-th element in the initial dataset is subtracted from the maximum value in the initial dataset to obtain the i-th element in the input dataset; and / or, After obtaining the specific function value corresponding to the i-th element, the bit width of the specific function value is transformed.

18. The method according to any one of claims 11 to 15, characterized in that, The method is used to implement a flexible maximum function layer in a neural network, which is used to classify data to be classified; wherein, The data to be processed includes at least one of voice data, text data, and image data.

19. An artificial intelligence accelerator, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 11-15.

20. An electronic device, characterized in that, include: The processor is configured to send at least one of a first lookup table, a second lookup table, or a third lookup table to the artificial intelligence chip. The first lookup table is configured to establish a first mapping relationship between the i-th element in the input dataset and the value of the exponential function; the second lookup table is configured to establish a second mapping relationship between the i-th element in the input dataset and the reciprocal of the exponential function value; and the third lookup table is configured to establish a third mapping relationship between a multiplication result and its reciprocal, wherein the multiplication result is obtained by multiplying the reciprocal of the exponential function value corresponding to the i-th element and the result of an addition operation, and the addition result is the result of adding the exponential function values ​​corresponding to at least some elements in the input dataset. An artificial intelligence chip is used to perform the method as described in any one of claims 11-18 based on at least one of the first lookup table, the second lookup table, or the third lookup table.

Images