Convolution operation circuit, compiling method and related product

Abstract

The present disclosure discloses a convolution operation circuit, a convolution operator compiling method and related products. The convolution operation circuit can be included in a computing device. The computing device can be included in a combined processing device, which can further include an interface device and other processing devices. The computing device interacts with the processing devices to jointly complete a user-specified computing operation. The combined processing device can further include a storage device connected with the computing device and the processing devices respectively, for storing data of the computing device and the processing devices. The scheme of the present disclosure provides a convolution operation circuit and a convolution operator compiling method, which can optimize the convolution operation of neurons based on asymmetric quantization and improve the overall operation performance.

Description

Technical Field

[0001] This disclosure generally relates to the field of intelligent computing, and more particularly to the field of neural networks. More specifically, this disclosure relates to a convolution operation circuit, a computing device, a chip, a board, and a method for compiling a convolution operator. Background Technology

[0002] With the development of artificial intelligence technology, the computational workload of neural network operations is increasing, requiring more and more computing resources. Quantizing the data used in neural network operations is a good way to reduce the computational workload and save computing resources.

[0003] Various quantization schemes exist, such as symmetric quantization, asymmetric quantization, channel quantization, and fixed-point quantization. Different quantization schemes are suitable for different application requirements and data characteristics. In neural network models, for convolutional operators, the data involved in the computation includes input neurons and weights. The weights are generally evenly distributed, while the neuron values ​​are usually asymmetrically distributed relative to 0. For example, neurons after ReLU only have positive values ​​and no negative values.

[0004] In view of this, there is an urgent need for a quantization scheme suitable for the data characteristics of neural network models, so as to improve the computational accuracy as much as possible while reducing computational overhead. Summary of the Invention

[0005] To address at least one or more of the technical problems mentioned above, this disclosure proposes hybrid quantization schemes in several aspects, and provides specific implementation circuits for convolution operators in particular.

[0006] In a first aspect, this disclosure provides a convolution operation circuit, comprising: an asymmetric quantization path for converting a floating-point input neuron into a fixed-point input neuron through asymmetric quantization; a symmetric quantization path for converting floating-point weights into fixed-point weights through symmetric quantization; a convolution circuit for performing a convolution operation on the fixed-point input neuron and the fixed-point weights; an output conversion circuit for converting the output of the convolution circuit into floating-point output data; a multiplication circuit for scaling the floating-point output data; and an addition circuit for superimposing a bias on the output of the multiplication circuit.

[0007] In a second aspect, this disclosure provides a computing device including a convolution operation circuit according to the first aspect.

[0008] In a third aspect, this disclosure provides a chip including a computing device according to the second aspect.

[0009] In the fourth aspect, this disclosure provides a board including the chip according to the third aspect.

[0010] In a fifth aspect, this disclosure provides a method for compiling a convolution operator, comprising: obtaining a file to be compiled containing the convolution operator; determining whether the input neuron of the convolution operator uses an asymmetric quantization scheme; in response to determining that the input neuron of the convolution operator uses an asymmetric quantization scheme, updating a bias new_bias for the convolution operator, wherein the new_bias integrates the original bias of the convolution operator and the offset introduced by the asymmetric quantization of the input neuron; and generating a runtime file containing the updated new_bias.

[0011] Through the convolution operation circuit and the compilation scheme of the convolution operator provided above, the embodiments disclosed herein provide a specific implementation scheme for hybrid quantization of the convolution operator. By reasonably allocating the calculation timing of each parameter / variable, it can effectively improve the overall operation performance and obtain the expected accuracy. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0013] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0014] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0015] Figure 3a This diagram illustrates the internal structure of a computing device according to an embodiment of the present disclosure.

[0016] Figure 3b This diagram illustrates the internal structure of the processor core in an embodiment of this disclosure.

[0017] Figure 4a and Figure 4b Exemplary schematic diagrams of symmetric quantization and asymmetric quantization are shown respectively;

[0018] Figure 5 An exemplary operational circuit block diagram of a convolution operator according to an embodiment of this disclosure is shown;

[0019] Figure 6 An exemplary flowchart of a method for compiling a convolution operator implemented by a processing device according to an embodiment of this disclosure is shown. Detailed Implementation

[0020] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0022] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0023] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0024] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0025] First, we provide explanations of the technical terms that may be used in this disclosure.

[0026] Floating-point numbers: The IEEE floating-point standard uses the following format to represent a number. Here, `sign` is the sign bit, where 0 represents a positive number and 1 represents a negative number; `E` represents the exponent, which is a weighted sum of 2 raised to the power of `E` (which may be a negative power); `mantissa` represents the mantissa, a binary fraction ranging from 1 to 2⁻¹, or 0 to ¹⁻¹. The representation of a floating-point number in a computer is divided into three fields, each encoded separately:

[0027] (1) A single sign bit s directly encodes the sign s.

[0028] (2) The k-bit exponent field encodes the exponent.

[0029] (3) The n-bit mantissa is used to encode the mantissa. However, the encoding result depends on whether the exponent stage is all 0.

[0030] Fixed-point numbers consist of three parts: a shared exponent, a sign bit, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign bit indicates whether the fixed-point number is positive or negative; and the mantissa determines the number of significant digits, i.e., the precision. Taking an 8-bit fixed-point number as an example, its numerical calculation method is as follows: value = (-1) sign ×(mantissa)×2 (exponent-127)

[0031] Data bit width: How many bits are used to represent the data.

[0032] Quantization is the process of converting high-precision numbers previously expressed in 32-bit or 64-bit form into fixed-point numbers that occupy less memory space, typically 16-bit or 8-bit. This conversion process inevitably results in some loss of precision.

[0033] Overflow: In a fixed-point arithmetic unit, the representation of numbers has a certain range. During the operation, if the size of the number exceeds the range that the fixed-point number can represent, it is called "overflow". Exemplary hardware environment

[0034] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0035] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0036] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0037] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0038] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0039] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0040] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0041] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0042] When the computing device 201 runs the neural network, it is generally necessary to first compile the neural network using the processing device 203 to obtain an executable file. This executable file contains device information, that is, on which device in the heterogeneous computer system the executable file needs to be executed. After the executable file is assembled and linked, the executable program of the neural network is obtained, and the executable program is stored in the storage device 204.

[0043] The processing device 203 can read the executable program from its storage location and obtain multiple tasks of the program based on the executable program. These tasks are distributed to the computing device 201 for execution via the interface device 202, and finally the calculation results are obtained.

[0044] Figure 3aA schematic diagram of the internal structure of computing device 201 is shown. Computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. As a system-on-a-chip, computing device 201 includes multiple clusters, and each cluster includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor core.

[0045] From the perspective of system-on-a-chip hierarchy, such as Figure 3a As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple clusters 305.

[0046] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to respond to access requests from the processor core to access external storage devices, such as… Figure 2 The DRAM 204 in the chip allows the computing device 201 to read data from external sources or write data to external sources. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302, and multiple clusters 305 to transmit data and control signals between the modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201. Four are shown exemplary in the figure; however, with hardware development, the computing device 201 of this invention can also include 8, 16, 64, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0047] From the perspective of cluster hierarchy, such as Figure 3a As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0048] Four processor cores 306 are shown exemplarily in the figure, but the present invention does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 3b As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0049] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0050] The computation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0051] Storage module 43 is used to store or move relevant data, including neuron RAM (NRAM) 431, weight RAM (WRAM) 432, input / output direct memory access (IODMA) 433, and move direct memory access (MVDMA) 434. NRAM 431 stores feature maps for computation by processor core 306 and intermediate results; WRAM 432 stores the convolution kernels, i.e., weights, of the deep learning network; IODMA 433 controls memory access between NRAM 431 / WRAM 432 and DRAM 204 via broadcast bus 309; MVDMA 434 controls memory access between NRAM 431 / WRAM 432 and SRAM 308. It should be noted that NRAM and WRAM here can be two storage regions formed by dividing the same memory in logical storage space, or they can be two independent memories; no specific limitation is made here.

[0052] Back Figure 3a The storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between cluster 305 and DRAM 204, communication between clusters 305, and communication between processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0053] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same cluster 305 does not need to be obtained from DRAM 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0054] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between clusters 305, and data transfer between cluster 305 and DRAM 204, respectively. These will be explained below.

[0055] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0056] CDMA 310 is used to control SRAM 308 access between different clusters 305 within the same computing device 201.

[0057] GDMA 311 works in conjunction with external memory controller 301 to control memory access from SRAM 308 to DRAM 204 in cluster 305, or to read data from DRAM 204 into SRAM 308. As described above, communication between DRAM 204 and NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between DRAM 204 and NRAM 431 or WRAM 432 via IODAM 433; the second channel involves first transmitting data between DRAM 204 and SRAM 308 via GDMA 311, and then transmitting data between SRAM 308 and NRAM 431 or WRAM 432 via MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is significantly greater than that of the first channel. Therefore, communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments of the present invention can select the data transmission channel according to their own hardware conditions.

[0058] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, GDMA 311 and IODMA 433 are considered different components. For those skilled in the art, any component whose functions and technical effects are similar to those of this invention falls within the scope of protection of this invention. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose functions and technical effects are similar to those of this invention falls within the scope of protection of this invention.

[0059] In the aforementioned hardware structure, the components inside the computing device 201 are referred to as on-chip, and the components outside the computing device 201 are referred to as off-chip. When performing calculations at each layer of deep learning, a large number of on-chip and off-chip accesses are required, especially reading data from DRAM 204 into the computing device 201 and then storing the calculation results of the computing device 201 into DRAM 204. Exemplary Quantization Scheme

[0060] As mentioned earlier, the neuron values ​​in a neural network model are asymmetrically distributed, while the weights are usually more evenly distributed. If both are uniformly quantized using symmetric quantization, there will be a significant loss of accuracy. Therefore, this disclosure provides a hybrid quantization scheme, that is, asymmetric quantization is used for the input neurons, while symmetric quantization is used for the weights and other constants.

[0061] Figure 4a and Figure 4b Exemplary schematic diagrams of symmetric quantization and asymmetric quantization are shown respectively.

[0062] Figure 4a The number field of the data to be quantized is distributed with "0" as the center of symmetry. Z is the maximum absolute value of all floating-point numbers in the number field of the data to be quantized. Figure 4a In this context, A represents the maximum floating-point number that can be represented by an n-bit fixed-point number. Converting a floating-point number A to a fixed-point number yields 2^n 1 ...3^n 12^n 12^n n-1 -1. To avoid overflow, A needs to contain Z.

[0063] Figure 4b The data domain shown is not symmetrically distributed with "0" as its center. `min` is the minimum value of all floating-point numbers in the data domain, and `max` is the maximum value of all floating-point numbers in the data domain. `P` is the center point between `min` and `max`. The entire data domain is shifted so that the shifted data domain is symmetrically distributed with "0" as its center. The maximum absolute value in the shifted data domain is `Z`. Figure 4b It can be seen that the offset is the horizontal distance between point "0" and point "P", which is called the offset O. Where O = (max + min) / 2, Z = (max - min) / 2.

[0064] Regardless of whether it is symmetric or asymmetric quantization, in this disclosed embodiment, the quantization formula can be unified as follows: Where X is the raw data, which is a floating-point type; X quantize This is the quantized number, of type int; n is the quantized data bit width; scale_factor is used to standardize the formula form and is not used in actual calculations, scale_factor = 2 position / scale; round performs rounding operations. It should be noted that this is not limited to rounding operations alone; other rounding operations can also be used, such as rounding up, rounding down, and rounding to zero, to replace the rounding operation in formula (1). clamp(a, b, x) is a clamping function used to restrict x to the interval a to b, which can be expressed as: The clamp and round operations can be implemented using float2int. Offset, position, and scale are quantization parameters, which are determined as follows.

[0065] The type of offset is int32, and its range is the range represented by int n;

[0066] The type of position is int32:

[0067] The scale type is fp32:

[0068] Because it's asymmetric quantization, the data distribution to be quantized is asymmetrical; some numbers are more negative, and some are more positive. Therefore, both the maximum and minimum values ​​must be considered. In the formula above, max = Max(0, Max(X)), min = Min(0, Min(X)). max - min represents the range from min to max. max - min equals 0 only when min == max == 0. Therefore, position is 0 in this case.

[0069] Accordingly, the formula for inverse quantization can be expressed as: X dequantize =(X quantize -offset)×scale_factor (2)

[0070] Therefore, the calculation formulas for quantization and dequantization are given above. Although the formulas have been provided, in actual technical implementation, careful design is needed for how to represent and calculate each parameter and variable, as well as how to arrange the calculation order to minimize the overall performance overhead and achieve a better accuracy range.

[0071] More specifically, for convolutional operators commonly used in neural network models, when employing hybrid quantization as disclosed in this embodiment, the computational performance of convolutional operators can be optimized by designing the timing of participation of each quantization parameter in the overall computation process. The convolutional operators mentioned herein can include various convolutional operations, such as, but not limited to, two-dimensional convolution, depth-wise convolution, reverse convolution, etc., and this disclosure embodiment is not limited in this respect.

[0072] Generally speaking, a complete convolution operation can be represented as: That is, the output neuron is equal to the convolution of the input neuron and the weights, plus the bias. For the floating-point convolution operation fp_conv, it can be expressed as:

[0073] In practical implementations, fixed-point arithmetic is typically used on AI chips, so it is desirable that the result of fixed-point arithmetic is as close as possible to the result of floating-point arithmetic. In other words, it is desirable that the result obtained after dequantizing the quantized data is close to the original floating-point number X. Substituting the floating-point number obtained from the previous dequantization formula (2) into formula (4), we can obtain: Among them, X q The offset represents the input after asymmetric quantization. input α (corresponding to the previous position) and α (corresponding to the previous scale) are the quantization parameters of the input neuron, respectively; W q This represents the weights after symmetric quantization, where b (corresponding to the previous position) and β (corresponding to the previous scale) are the quantization parameters of the weights.

[0074] As can be seen from formula (5), floating-point convolution operation can be converted into fixed-point convolution operation multiplied by a coefficient. Subtract a constant Add bias. This transformation simplifies the convolution operation.

[0075] Figure 5 An exemplary operational circuit block diagram of a convolution operator according to an embodiment of this disclosure is shown. As shown, the convolution operation circuit 500 includes an asymmetric quantization path 510 and a symmetric quantization path 520. The asymmetric quantization path 510 is used to convert floating-point input neurons into fixed-point input neurons through asymmetric quantization. The symmetric quantization path 520 is used to convert floating-point weights into fixed-point weights through symmetric quantization.

[0076] More specifically, the asymmetric quantization path 510 includes a first symmetric quantization circuit 501 and a conversion circuit 503. The first symmetric quantization circuit 501 is used to symmetrically quantize the floating-point input neuron and convert it into a fixed-point input.

[0077] The first symmetric quantization circuit 501 performs the operation: Where x symm-q denoted as the symmetrically quantized input neuron, x represents the floating-point input neuron, and a and α are the quantization parameters position and scale of the input neuron, respectively.

[0078] The conversion circuit 503 is used to convert the symmetric quantized input into the asymmetric quantized input.

[0079] Specifically, the conversion circuit 503 performs the operation: x asy-q =x symm-q +offset input , That is, the input x after symmetric quantization symm-q Add offset input offset input It is the quantization parameter offset of the input neuron, thus obtaining the asymmetric quantized input x. asy-q .

[0080] The symmetric quantization path 520 includes a second symmetric quantization circuit 502. The second symmetric quantization circuit 502 is used to symmetrically quantize floating-point weights and convert them into fixed-point weights.

[0081] Similarly, the second symmetric quantization circuit 502 performs the operation: Where w symm-q represents the weights after symmetric quantization, w represents the floating-point weights, and b and β are the quantization parameters position and scale of the weights, respectively.

[0082] The convolution operation circuit 500 also includes a convolution circuit 504 for performing convolution operations on asymmetric quantized fixed-point input neurons and symmetric quantized fixed-point weights.

[0083] As can be seen from formula (5), the convolution operation, which converts floating-point operations to fixed-point operations, requires multiplication by a coefficient. In some embodiments, the coefficients can be The space is divided into two parts: 2 a+b and Since the convolution circuit 504 performs fixed-point convolution operations, a+b is the quantization parameter position, which, from a hardware perspective, is a shift operation and is performed together with the convolution operation of the convolution circuit 504. This refers to the quantization parameter "scale," which is a multiplication operation; therefore, multiplication circuit 506 performs this operation. Both convolution circuit 504 and multiplication circuit 506 belong to... Figure 3b In the arithmetic circuit section of the arithmetic module 42, the chip performs these operations in parallel during actual calculations, reducing execution time and saving the number of times the on-chip SRAM 308 is accessed, thus reducing the storage space occupied by intermediate results.

[0084] Specifically, in some embodiments, the convolution circuit 504 is configured to perform operations: Where y fixIndicates the output of fixed-point convolution, x asy-q w represents the input neuron with asymmetric quantization. symm_q This represents the weights after symmetric quantization, where 'a' is the quantization parameter position of the input neuron and 'b' is the quantization parameter position of the weights.

[0085] The convolution operation circuit 500 also includes an output conversion circuit 505, used to convert the output result of the convolution circuit into floating-point output data. Specifically, it converts the output y of the convolution circuit 504 into floating-point output data. fix Converting from fixed-point to floating-point type y fp Therefore, all subsequent circuits can use floating-point operations.

[0086] The convolution operation circuit 500 also includes a multiplication circuit 506 for scaling its preceding floating-point output data. In some embodiments, the multiplication circuit 506 is configured to multiply by the previously split second-part coefficients. That is, the following operation is performed: Where y1 represents the output of multiplication circuit 506, y fp Indicates y fix The converted floating-point data has α as the quantization parameter scale of the input neuron and β as the quantization parameter scale of the weights.

[0087] coefficient Typically very small, for example, in 10 -10 ~10 -11 The order of magnitude is [missing information]. The 506 multiplication circuit uses floating-point arithmetic, which allows for higher precision calculations and thus more accurate results.

[0088] The convolution operation circuit 500 also includes an adder circuit 507 for adding a bias to the output of the multiplication circuit 506. In some embodiments, the bias incorporates an additional portion resulting from the offset of asymmetric quantization. That is, in the adder circuit 507, its configuration is used to perform the operation: y2 = y1 + new_bias, Where y2 represents the output of adder circuit 507, y1 represents the output of multiplier circuit 506, and new_bias combines the original bias of the convolution operation with the offset introduced by the asymmetric quantization of the input neuron.

[0089] As shown in Equation (5), the asymmetric quantization of the input neuron results in an offset. input This offset will also be convolved with the weights, resulting in an additional part: In some embodiments, this additional portion is merged with the bias in the convolution operation to form a new bias `new_bias`, thereby reducing computational overhead. Also, the `new_bias` above is calculated as follows:

[0090] Furthermore, the calculation of the new bias `new_bias` itself can be performed at compile time, that is, pre-calculated during the compilation of the convolution operation instructions. Compilation is typically implemented on a processing device such as a CPU, and the compiled model file contains the pre-calculated `new_bias` value. When running the network model on the computing device, the model file is loaded, and the `new_bias` value in the model file is used directly, thus saving the computational overhead of calculating `new_bias` on the computing device. Furthermore, since there is no bit-width limitation when calculating `new_bias` on the CPU, high-precision implementation of `new_bias` can be used to obtain a more accurate `new_bias` value, with precision closer to floating-point representation. Moreover, because `new_bias` uses high-precision calculation, the risk of precision overflow is avoided. Additionally, due to the coefficients... Typically very small, for example, in 10 -10 ~10 -11 With a magnitude of [value], high-precision calculations can yield more accurate results.

[0091] When calculating new_bias on the CPU, the calculation can be performed based on the values ​​of the quantization parameters determined during the calibration phase. Typically, during the calibration phase, the user can provide several sets of input data, such as several sets of image inputs. The calibration process then calculates the maximum and minimum values ​​of each operator corresponding to these sets of inputs in floating-point format based on the user-provided inputs. After obtaining the maximum and minimum values ​​of each operator's inputs, the quantization parameters offset, position, and scale corresponding to each operator can be determined. Therefore, at compile time, based on the values ​​of the quantization parameters determined during the calibration phase, the calculation of formula (6) can be performed to directly obtain new_bias for runtime use.

[0092] In some embodiments, the convolution operation itself may not have a bias. In these embodiments, a bias can be created, and the value of the bias can be directly calculated as...

[0093] In other embodiments, boundary padding may occur during convolution operations. Typically, the main purpose of boundary padding is to ensure that the input and output feature maps of the neural network layer are of the same size; this is achieved by padding the image with a ring of zero values ​​around its perimeter. In this disclosed embodiment, to match asymmetric quantization, the values ​​in the padding region are also subjected to corresponding asymmetric quantization. According to the quantization formula (1), the padding value is equal to the offset. input Therefore, in these embodiments, the fill value can be directly set to the offset when performing the fill operation. input .

[0094] The circuit implementation of the convolution operation using hybrid quantization in this disclosure embodiment has been described above with reference to the accompanying drawings. As can be seen from the above description, by reasonably designing the timing of operations involving constants in the convolution operation—for example, by advancing operations related to offset to the compilation stage—the overall performance overhead can be minimized while maintaining a certain level of accuracy. Furthermore, by merging the offset and bias operations together, the computation process can be further simplified.

[0095] This disclosure also provides a method for compiling a convolution operator. Figure 6 An exemplary flowchart illustrating a method for compiling a convolution operator implemented by a processing device according to an embodiment of this disclosure is shown. The processing device may be, for example, […]. Figure 2 The processing device 203.

[0096] As shown in the figure, in step 610, the file to be compiled, containing convolution operators, is obtained. In programming frameworks, computation graphs are typically used to describe the computation process of machine learning algorithms, and operators are used to represent various operations. Convolution operators are included as computation nodes in the computation graph for compilation.

[0097] Next, in step 620, it is determined whether the input neurons of the convolution operator use an asymmetric quantization scheme. In some embodiments, the user can specify whether the convolution operator uses an asymmetric quantization scheme through a configuration file. By checking the configuration file, it can be determined whether the input neurons of the convolution operator use an asymmetric quantization scheme.

[0098] Next, in step 630, in response to determining that the input neuron of the convolution operator adopts an asymmetric quantization scheme, the bias new_bias used for the convolution operator is updated, wherein new_bias combines the original bias of the convolution operator and the offset introduced by the asymmetric quantization of the input neuron.

[0099] Furthermore, updating the bias `new_bias` used for the convolution operator may include: determining whether the convolution operator has an original bias; if it does, loading the original bias; otherwise, creating the original bias and setting it to 0; and updating `new_bias` according to the following formula: Where bias represents the original bias of the convolution operator, and offset represents the offset of the convolution operator. input It is the quantization parameter offset of the input neuron, w q Let represent the weights after symmetric quantization, where a and α are the quantization parameters position and scale of the input neuron, respectively, and b and β are the quantization parameters position and scale of the weights, respectively.

[0100] Finally, in step 640, a runtime file is generated that contains the updated new_bias. Thus, when the runtime file is loaded onto the computing device, the computing device can directly use the pre-calculated new_bias to perform operations without needing to call resources for computation on-chip.

[0101] In some embodiments, when the input neuron of the convolution operator has boundary padding, the padding value of the padding portion of the input neuron is set to offset. input offset input It is the offset of the quantization parameter of the input neuron.

[0102] Therefore, this disclosure provides a method for compiling a convolution operator that supports pre-calculating the bias of the convolution operator during compilation when the input neuron of the convolution operator requires an asymmetric quantization scheme. This bias integrates the original bias of the convolution operator and the offset introduced by the asymmetric quantization of the input neuron, thereby saving on-chip computation for this part. Furthermore, since compilation is typically performed on the CPU and there is no bit-width limitation, high-precision calculation of `new_bias` can be used to obtain a more accurate value, with precision closer to floating-point representation. Moreover, the use of high-precision calculation for `new_bias` avoids the risk of precision overflow.

[0103] This disclosure also provides a processing apparatus that can compile a convolution operator according to the method described above. This disclosure also provides a computing apparatus that may include the convolution operation circuitry described above.

[0104] In this disclosure embodiment, a computer-readable storage medium is also provided, storing program instructions that, when loaded and executed by a processor, cause the processor to perform the compilation method for the convolution operator described in this disclosure embodiment. In this disclosure embodiment, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the compilation method for the convolution operator described in this disclosure embodiment.

[0105] This disclosure also provides a chip that may include the aforementioned computing device. Furthermore, this disclosure also provides a board that may include the aforementioned chip.

[0106] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0107] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0108] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0109] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0110] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0111] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A convolution operation circuit, comprising: The asymmetric quantization pathway is used to convert floating-point input neurons into fixed-point input neurons through asymmetric quantization. Symmetric quantization path is used to convert floating-point weights into fixed-point weights through symmetric quantization; A convolution circuit is used to perform convolution operations on the fixed-point input neurons and fixed-point weights; Output conversion circuit, used to convert the output of the convolution circuit into floating-point output data; A multiplication circuit is used to scale and adjust the floating-point output data; as well as An adder circuit is used to superimpose a bias onto the output of the multiplication circuit; The addition circuit described above is configured to perform the operation: , in This indicates the output of the adder circuit. This represents the output of the multiplication circuit. It combines the original bias of the convolution operation with the offset introduced by the asymmetric quantization of the input neuron; The It is pre-calculated during the compilation of the instructions for the convolution operation according to the following formula: ， in, This represents the original bias of the convolution operation. It is the quantization parameter offset of the input neuron. Let a and represent the weights after symmetric quantization. These are the quantization parameters of the input neuron: position and scale, b and... These are the quantization parameters for the weights: position and scale.

2. The convolution operation circuit according to claim 1, wherein the asymmetric quantization path comprises: The first symmetric quantization circuit is used to symmetrically quantize the floating-point input neuron to obtain a fixed-point symmetrically quantized input neuron. as well as A conversion circuit is used to add an offset to the fixed-point symmetric quantized input neuron to convert it into a fixed-point asymmetric quantized input neuron.

3. The convolution operation circuit according to claim 2, wherein the first symmetric quantization circuit is configured to perform the operation: in Let x represent the symmetrically quantized input neuron, and let a and y represent the floating-point input neuron. These are the quantization parameters position and scale of the input neuron, respectively.

4. The convolution operation circuit according to claim 3, wherein the conversion circuit is configured to perform the operation: ， in This represents an input neuron that has undergone asymmetric quantization. It is the offset of the quantization parameter of the input neuron.

5. The convolution operation circuit according to claim 1, wherein the symmetric quantization path comprises: The second symmetric quantization circuit is used to symmetrically quantize the floating-point weights to obtain symmetrically quantized fixed-point weights.

6. The convolution operation circuit according to claim 5, wherein the second symmetric quantization circuit is configured to perform the operation: ， in Let w represent the weights after symmetric quantization, and b represent the floating-point weights. These are the quantization parameters for the weights: position and scale.

7. The convolution operation circuit according to any one of claims 1-6, wherein the convolution circuit is configured to perform the operation: ， in This indicates the output of a fixed-point convolution. This represents an input neuron that has undergone asymmetric quantization. This represents the weights after symmetric quantization, where 'a' is the quantization parameter position of the input neuron and 'b' is the quantization parameter position of the weights.

8. The convolution operation circuit according to claim 7, wherein the multiplication circuit is configured to perform the operation: ， in This represents the output of the multiplication circuit. express The converted floating-point data, It is the quantization parameter scale of the input neuron. It is the quantization parameter scale of the weights.

9. The convolution operation circuit according to claim 8, wherein the multiplication circuit is performed using high-precision floating-point operations.

10. The convolution operation circuit according to claim 1, wherein, During the compilation of the instructions for the convolution operation, if the original bias of the convolution operation does not exist, the original bias of the convolution operation is created and set to 0.

11. The convolution operation circuit according to any one of claims 1-6, wherein when the input neuron of the convolution operation has boundary padding, the padding value of the padding portion of the input neuron is set to 0. The It is the offset of the quantization parameter of the input neuron.

12. A computing device comprising a convolution operation circuit according to any one of claims 1-11.

13. A chip comprising the computing device according to claim 12.

14. A circuit board comprising the chip according to claim 13.

15. A method for compiling a convolution operator, comprising: Obtain the file to be compiled containing convolution operators; Determine whether the input neurons of the convolution operator use an asymmetric quantization scheme; In response to determining that the input neurons of the convolution operator employ an asymmetric quantization scheme, the bias used for the convolution operator is updated. , wherein It combines the original bias of the convolution operator with the offset introduced by the asymmetric quantization of the input neuron; as well as Generate a runtime file containing the updated... ; Where the bias used for the convolution operator is updated include: Determine whether the convolution operator has an original bias. If it exists, load the original bias; otherwise, create the original bias and set it to 0. Update the following formula : ， in, This represents the original bias of the convolution operator. It is the quantization parameter offset of the input neuron. Let a and represent the weights after symmetric quantization. These are the quantization parameters of the input neuron: position and scale, b and... These are the quantization parameters for the weights: position and scale.

16. The compilation method according to claim 15, further comprising: When the input neuron of the convolution operator has boundary padding, the padding value of the padding part of the input neuron is set to 0. The It is the offset of the quantization parameter of the input neuron.

Images