Reconfigurable processing units for deep learning

Abstract

A reconfigurable processing unit for deep learning includes: a configuration control unit for receiving configuration information and bus commands from an external bus, generating mode instructions and control instructions, and obtaining configuration parameters; a first-in-first-out (FIFO) memory for receiving external data and outputting computation result data; a multiplexer for inputting external data into an arithmetic logic unit (ALU) or a static random access memory (SRAM); an ALU for performing calculations on the external data using the configuration parameters, obtaining computation result data, and inputting the computation result data into the SRAM or FIFO; and a SRAM for storing the computation result data or external data, and outputting the computation result data or external data to the ALU, or outputting the computation result data to the FIFO.

Description

Technical Field

[0001] This disclosure relates to the field of network devices, and more specifically, to a reconfigurable processing unit for deep learning. Background Technology

[0002] With the development of computer technology, deep learning has become one of the hottest research topics in machine learning in recent years. Deep learning requires a large amount of data computation for training and inference of target features. The processors used for deep learning computations mainly include general-purpose processors (GPPs) and application-specific integrated circuits (ASICs). Common deep learning operators include convolution, pooling, and activation functions.

[0003] GPPs can handle various types of computing applications, including mathematical and logical operations, but they cannot fully utilize algorithmic parallelism for data processing. ASICs are integrated circuits designed and manufactured to meet specific user requirements and the needs of specific electronic systems, resulting in long design cycles and high design costs. Reconfigurable processors are processing structures that connect a certain number of processing units through specific interconnection methods to achieve parallel processing of different algorithms. In reconfigurable processor research, based on reconfigurable granularity, they can be mainly divided into fine-grained and coarse-grained types. Fine-grained reconfigurable processors have high versatility, but during computation, their configuration time is too long, failing to meet the dynamic real-time configuration requirements of some real-time applications. Summary of the Invention

[0004] In view of this, this disclosure provides a reconfigurable processing unit for deep learning, comprising:

[0005] The configuration control unit is used to receive configuration information and bus commands from an external bus, generate mode instructions and control instructions based on the configuration information and bus commands, and obtain configuration parameters.

[0006] The first-in-first-out (FIFO) memory is used to receive external data and output calculation results data according to the above control instructions.

[0007] A multiplexer is used to input the external data into the arithmetic logic unit or static random access memory according to the aforementioned mode instructions and control instructions.

[0008] The arithmetic logic unit, in response to the control information, uses the configuration parameters to perform calculations on the external data from the multiplexer or static random access memory to obtain calculation result data with 8-bit or 16-bit data precision, and inputs the calculation result data into the static random access memory or the first-in-first-out memory.

[0009] In addition, a static random access memory, in response to the above mode instructions and control instructions, is used to store calculation result data or external data generated by the above arithmetic logic unit, output the calculation result data or external data to the above arithmetic logic unit, or output the calculation result data to the above first-in-first-out memory.

[0010] According to embodiments of this disclosure, the above-mentioned configuration control unit includes:

[0011] The configuration register is used to receive configuration information from the external bus and generate the aforementioned mode instructions.

[0012] And a controller, used to receive bus commands from an external bus and generate the aforementioned enable signal.

[0013] The configuration register is also used to obtain the configuration parameters based on the configuration information according to the enable signal from the controller.

[0014] Furthermore, the controller is also configured to obtain the control instructions based on the configuration information in response to the mode instructions.

[0015] According to embodiments of this disclosure, the configuration registers include: a cell mode register, a row register, a column register, a pooling mode register, a channel register, and a quantization register.

[0016] The aforementioned cell mode register receives configuration information from the external bus and generates mode instructions based on the configuration information. The mode instructions include convolution mode instructions and pooling mode instructions.

[0017] Furthermore, when the aforementioned convolution mode instruction is generated by the aforementioned cell mode register, the aforementioned row register, column register, channel register, and quantization register generate the aforementioned configuration parameters based on the aforementioned configuration information in response to the enable signal from the aforementioned controller.

[0018] When the above-mentioned unit mode register generates the above-mentioned pooling mode instruction, the above-mentioned row register, column register, and pooling mode register respond to the enable signal from the above-mentioned controller and generate the above-mentioned configuration parameters based on the above-mentioned configuration information.

[0019] According to embodiments of this disclosure, the controller includes: a decoder, an accumulator state machine, a transmit state machine, a load state machine, a maximum value state machine, and an output state machine.

[0020] The aforementioned accumulator state machine and transmitter state machine are used to receive bus commands from the external bus in response to the aforementioned convolution mode instruction, generate the aforementioned enable signal, and obtain the aforementioned control instructions based on the aforementioned configuration information decoded by the decoder.

[0021] The aforementioned loading state machine, maximum value state machine, and output state machine are used to respond to the aforementioned pooling mode instruction, receive bus commands from the external bus, generate the aforementioned enable signal, and obtain the aforementioned control instruction based on the aforementioned configuration information decoded by the decoder.

[0022] According to embodiments of this disclosure, the above-described configuration control unit is used for:

[0023] Determine the state of the aforementioned first-in-first-out memory.

[0024] When the aforementioned FIFO memory is determined to be empty, the aforementioned FIFO memory does not output external data to the aforementioned multiplexer, so that the aforementioned reconfigurable processing unit does not perform calculations.

[0025] Furthermore, when it is determined that the aforementioned first-in-first-out (FIFO) memory is full, the aforementioned FIFO memory does not acquire the calculation result data generated by the aforementioned reconfigurable processing unit, so that the aforementioned reconfigurable processing unit does not perform calculations.

[0026] When the aforementioned FIFO memory is not empty or full, the aforementioned FIFO memory outputs external data to the aforementioned multiplexer and outputs calculation result data to the aforementioned reconfigurable processing unit, so that the aforementioned reconfigurable processing unit starts to perform calculations.

[0027] According to embodiments of this disclosure, the multiplexer described above is used for:

[0028] In response to the control instructions and convolution mode instructions, the multiplexer inputs the external data from the first input queue received by the first-in-first-out memory into the arithmetic logic unit or the static random access memory.

[0029] In response to the control instructions and pooling mode instructions, the multiplexer inputs the external data from the second input queue received by the first-in-first-out memory into the arithmetic logic unit or the static random access memory.

[0030] According to embodiments of this disclosure, the static random access memory is further used for:

[0031] In response to the aforementioned control command, the external data from the aforementioned multiplexer is cached as lookup table data; and the intermediate calculation data of the aforementioned arithmetic logic unit is stored.

[0032] According to embodiments of this disclosure, the arithmetic logic unit includes: an adder, a multiplier, a shifter, an output graph counter, a channel counter, and a partial register.

[0033] The above convolution mode instructions include multiplication and addition instructions.

[0034] The controller responds to the convolution mode instruction, and the accumulator state machine and transmitter state machine decode the configuration information based on the decoder to obtain the control instruction; the first-in-first-out (FIFO) memory responds to the convolution mode instruction and the control instruction generated by the accumulator state machine and transmitter state machine, receives external data and outputs calculation result data; the multiplexer responds to the convolution mode instruction and the control instruction generated by the accumulator state machine, inputting the external data of the first input queue received by the FIFO memory into the arithmetic logic unit; the output graph counter of the arithmetic logic unit responds to the control instruction generated by the accumulator state machine, determines the number of rows and columns of the output graph calculated by the arithmetic logic unit, and controls the size of the output graph; the arithmetic logic... The channel counter of the logic unit responds to the control instruction generated by the transmission state machine to determine the number of channel accumulations and the number of data accesses to the static random access memory during calculation. The adder, multiplier, shifter, and partial sum register of the arithmetic logic unit use the number of channel accumulations and the number of data accesses to the static random access memory during calculation to perform calculations on the external data, realizing vector multiplication, matrix multiplication, and matrix addition operations on the external data to obtain the calculation result data. In response to the convolution mode instruction, the controller decodes the configuration parameters of the quantization register, and the decoded configuration parameters are multiplied by the calculation result data through the multiplier to obtain calculation result data with 8-bit or 16-bit data precision.

[0035] According to embodiments of this disclosure, the arithmetic logic unit further includes a maximum value register, a maximum value counter, and a comparator.

[0036] The above pooling mode instructions include max pooling instructions and average pooling instructions.

[0037] In response to the pooling mode instruction, the controller, loading state machine, maximum value state machine, and output state machine decode the configuration information based on the decoder to obtain the control instruction; the first-in-first-out (FIFO) memory, in response to the pooling mode instruction and the control instruction generated by the loading and output state machines, receives external data and outputs calculation result data; the multiplexer, in response to the convolution mode instruction and the control instruction generated by the accumulation state machine, inputs the external data from the second input queue received by the FIFO memory into the arithmetic logic unit; the output graph of the arithmetic logic unit... The counter responds to the control instructions generated by the loading state machine to determine the number of rows and columns of the output graph calculated by the arithmetic logic unit and controls the size of the output graph. The maximum value register of the arithmetic logic unit responds to the control instructions generated by the maximum value state machine and is used to store the maximum value of the max pooling kernel during the calculation of each channel. The maximum value register, maximum value counter, partial sum register, comparator, adder, and output graph counter of the arithmetic logic unit use the external data to perform calculations, realize the max pooling and average pooling operations on the external data, and obtain the calculation result data.

[0038] According to embodiments of this disclosure, the aforementioned static random access memory is further configured to: in response to control information, perform activation function operations on the 8-bit data precision calculation result data based on the aforementioned lookup table; in response to control information, store the 16-bit data precision calculation result data in a multi-segment polynomial fitting mode, and perform activation function operations on each segment of the 16-bit data precision calculation result data based on the aforementioned lookup table to obtain multiple first-order activation function operation results; and the aforementioned arithmetic logic unit is further configured to accumulate the multiple first-order activation function operation results to obtain the 16-bit data precision activation function operation result of the aforementioned calculation result data. According to embodiments of this disclosure, the reconfigurable processing unit's function is controlled by a configuration control unit. The configuration control unit controls a first-in-first-out (FIFO) memory to receive external data; controls a multiplexer to input the external data into an arithmetic logic unit (ALU) or a static random access memory (SRAM); controls the ALU to perform calculations on the external data to obtain calculation results with 8-bit or 16-bit precision; controls the SRAM to store the calculation results or external data generated by the ALU, and outputs the calculation results or external data to the ALU; and controls the FIFO memory to output the calculation results.

[0039] The aforementioned reconfigurable processing unit performs calculations on coarse-grained data, reducing the size of configuration information and improving computation speed. Different mode instruction control implements vector multiplication, matrix multiplication and matrix addition, max pooling, average pooling, Sigmoid function, and Tanh function calculations. The functions of the aforementioned arithmetic logic unit are reused during the calculation of the Tanh function, and the aforementioned static random access memory implements the function of caching data, thereby improving the utilization of hardware resources. Attached Figure Description

[0040] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0041] Figure 1 The schematic diagram illustrates a structural block diagram of a reconfigurable processing unit for deep learning according to an embodiment of the present disclosure;

[0042] Figure 2 This schematically illustrates a structural block diagram of the configuration control unit according to an embodiment of the present disclosure;

[0043] Figure 3 The diagram schematically illustrates the microarchitecture of hardware for reconfigurable computation in convolutional mode according to embodiments of the present disclosure.

[0044] Figure 4 The diagram schematically illustrates a microarchitecture diagram of hardware for reconfigurable computing in pooling mode according to embodiments of the present disclosure. Detailed Implementation

[0045] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0047] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0048] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0049] In realizing the concept disclosed herein, the inventors discovered that fine-grained reconfigurable devices in related technologies are mainly represented by Field-Programmable Gate Arrays (FPGAs) that perform look-up table (LUT) operations at the bit level. The fine-grained nature of FPGAs results in a finer operator abstraction level and requires more configuration information, increasing signal transmission delay and reducing computation speed. To meet computational demands, the number of processors needs to be increased, leading to a larger chip area. Furthermore, the configuration time of fine-grained reconfigurable devices becomes too long, failing to meet the dynamic real-time configuration requirements of some real-time applications.

[0050] In order to at least partially solve the technical problems existing in the related art, embodiments of this disclosure provide a reconfigurable processing unit for deep learning.

[0051] Figure 1 A schematic block diagram of a reconfigurable processing unit for deep learning according to an embodiment of the present disclosure is shown.

[0052] like Figure 1 As shown, the reconfigurable processing unit for deep learning includes:

[0053] The configuration control unit is used to receive configuration information and bus commands from an external bus, generate mode instructions and control instructions based on the configuration information and bus commands, and obtain configuration parameters.

[0054] The first-in-first-out (FIFO) memory is used to receive external data and output calculation results data according to the above control instructions.

[0055] A multiplexer is used to input the external data into the arithmetic logic unit or static random access memory according to the aforementioned mode instructions and control instructions.

[0056] The arithmetic logic unit, in response to the control information, uses the configuration parameters to perform calculations on the external data from the multiplexer or static random access memory to obtain calculation result data with 8-bit or 16-bit data precision, and inputs the calculation result data into the static random access memory or the first-in-first-out memory.

[0057] The static random access memory, in response to the aforementioned mode instructions and control instructions, is used to store calculation result data or external data generated by the aforementioned arithmetic logic unit, output the aforementioned calculation result data or external data to the aforementioned arithmetic logic unit, or output the aforementioned calculation result data to the aforementioned first-in-first-out memory.

[0058] According to embodiments of this disclosure, a control unit is configured, and configuration parameters can be input into an arithmetic logic unit to perform calculations with external data. By controlling the configuration parameters, the precision of the calculation result data can be adjusted to meet different calculation precision requirements.

[0059] Figure 2 A schematic block diagram of a configuration control unit according to an embodiment of the present disclosure is shown.

[0060] like Figure 2 As shown, the above configuration control unit includes:

[0061] A configuration register is used to receive configuration information from an external bus and generate the aforementioned mode instruction; and a controller is used to receive a bus command from an external bus and generate the aforementioned enable signal; the configuration register is also used to obtain the aforementioned configuration parameters based on the configuration information according to the enable signal from the controller; and the controller is also used to obtain the aforementioned control instruction based on the configuration information in response to the aforementioned mode instruction.

[0062] According to embodiments of this disclosure, the configuration control unit receives configuration information and bus commands from an external bus, obtains mode instructions and control instructions, and sends them to a static random access memory, a multiplexer, an arithmetic logic unit, and a first-in-first-out memory. The configuration control unit specifies the function of the reconfigurable processing unit and controls the calculation process of the reconfigurable processing unit.

[0063] According to embodiments of this disclosure, the configuration registers may include: cell mode registers, row registers, column registers, pooling mode registers, channel registers, and quantization registers.

[0064] The aforementioned unit mode register receives configuration information from an external bus and generates mode instructions based on the configuration information. The mode instructions include convolution mode instructions and pooling mode instructions. When the aforementioned unit mode register generates the convolution mode instruction, the aforementioned row register, column register, channel register, and quantization register generate the aforementioned configuration parameters based on the configuration information in response to an enable signal from the aforementioned controller. When the aforementioned unit mode register generates the aforementioned pooling mode instruction, the aforementioned row register, column register, and pooling mode register generate the aforementioned configuration parameters based on the aforementioned configuration information in response to an enable signal from the aforementioned controller.

[0065] According to embodiments of this disclosure, the channel register in the configuration register may include a channel accumulator register (chan_add_reg) and a repeat accumulator register (chan_add_re_reg). In the configuration parameters generated by the channel register, the high 8 bits represent the configuration parameters generated by the repeat accumulator register, and the low 8 bits represent the configuration parameters generated by the channel accumulator register. The quantization register in the configuration register may include a quantization shift register and a quantization parameter register. In the configuration parameters generated by the quantization register, the quantization register responds to control signals and can output a 37-bit string of configuration parameters to the arithmetic logic unit. The high 5 bits represent the configuration parameters generated by the quantization shift register, and the low 32 bits represent the configuration parameters generated by the quantization parameter register. The configuration parameters generated by the quantization register are combined with the calculation data in the arithmetic logic unit under different mode instruction control to determine the precision of the calculation result data. The quantization shift register indicates the data bit width that needs to be shifted left.

[0066] The configuration register generates configuration parameters for different registers based on configuration information from the external bus. These parameters determine the row and column information of the computation data and specify different registers. This enables a single configuration of the reconfigurable processing unit via the external bus, allowing for multiple computations. To change the configuration parameters, the configuration signal can be re-input via the external bus, enabling real-time switching between different computation modes.

[0067] According to embodiments of this disclosure, the controller includes: a decoder, an accumulator state machine, a transmit state machine, a load state machine, a maximum value state machine, and an output state machine.

[0068] The aforementioned accumulation state machine and transmission state machine are used to receive bus commands from the external bus in response to the aforementioned convolution mode instruction, generate the aforementioned enable signal, and obtain the aforementioned control instruction based on the aforementioned configuration information decoded by the decoder; the aforementioned loading state machine, maximum value state machine, and output state machine are used to receive bus commands from the external bus in response to the aforementioned pooling mode instruction, generate the aforementioned enable signal, and obtain the aforementioned control instruction based on the aforementioned configuration information decoded by the decoder.

[0069] According to embodiments of this disclosure, the accumulator state machine (Cadd_state) and sender state machine (Send_state) in the controller respond to the above convolution mode and jointly control the multiplexer, arithmetic logic unit, and static random access memory; the accumulator state machine controls the first-in-first-out memory to receive external data, and the sender state machine outputs the calculation result data from the first-in-first-out memory.

[0070] The accumulator state machine (Cadd_state) in the controller represents the process of receiving external data to perform multiplication and accumulation operations, obtaining partial sum data, and storing the partial sum data in static random access memory. If the accumulator state machine does not complete the calculation of all partial sum data, it is necessary to retrieve multiple partial sum data from the static random access memory and re-accumulate them to complete the data calculation.

[0071] According to the embodiments of this disclosure, the loading state machine, the maximum value state machine, and the output state machine in the controller jointly control the multiplexer, the arithmetic logic unit, and the static random access memory in response to the above-mentioned pooling mode instructions; the loading state machine controls the first-in-first-out memory to receive external data, and the output state machine outputs the calculation result data from the first-in-first-out memory.

[0072] According to embodiments of this disclosure, the above-described configuration control unit is used for:

[0073] The state of the aforementioned first-in-first-out (FIFO) memory is determined; if the aforementioned FIFO memory is determined to be empty, the aforementioned FIFO memory does not output external data to the aforementioned multiplexer, so that the aforementioned reconfigurable processing unit does not perform calculations; and if the aforementioned FIFO memory is determined to be full, the aforementioned FIFO memory does not acquire the calculation result data generated by the aforementioned reconfigurable processing unit, so that the aforementioned reconfigurable processing unit does not perform calculations.

[0074] When the aforementioned FIFO memory is not empty or full, the aforementioned FIFO memory outputs external data to the aforementioned multiplexer and outputs calculation result data to the aforementioned reconfigurable processing unit, so that the aforementioned reconfigurable processing unit starts to perform calculations.

[0075] According to embodiments of this disclosure, the configuration control unit first configures the reconfigurable processing unit based on configuration information and bus commands from the external bus. Then, the first-in-first-out (FIFO) memory executes data reception and output according to control instructions, achieving data control. This allows the reconfigurable processing unit to configure the internal sub-unit states before computation begins, and then receive external data to perform specific operations after configuration. The relative independence of each sub-unit within the reconfigurable processing unit module necessitates individual control by the configuration control unit, eliminating the need for joint control of all sub-units during computation, thus reducing the amount of configuration information and bus commands from the external bus.

[0076] According to embodiments of this disclosure, the multiplexer described above is used for:

[0077] In response to the control command and the convolution mode command, the multiplexer inputs the external data of the first input queue received by the first-in-first-out memory into the arithmetic logic unit or the static random access memory.

[0078] In response to the control instructions and pooling mode instructions, the multiplexer inputs the external data from the second input queue received by the first-in-first-out memory into the arithmetic logic unit or the static random access memory.

[0079] According to embodiments of this disclosure, by using a single multiplexer in response to the control instructions and different mode instructions, it is possible to select the source of external data received from the first-in-first-out memory in a reconfigurable processing unit and input the external data into the arithmetic logic unit or static random access memory. This reduces the number of multiplexers used, decreases the size of the reconfigurable processing unit, and improves the utilization of hardware resources.

[0080] According to embodiments of this disclosure, the arithmetic logic unit includes: an adder, a multiplier, a shifter, an output graph counter, a channel counter, and a partial register.

[0081] The above convolution mode instructions include multiplication and addition instructions.

[0082] In response to the convolution mode instruction, the controller obtains the control instruction by decoding the configuration information based on the decoder using the accumulation state machine and the transmission state machine.

[0083] The aforementioned first-in-first-out memory responds to the aforementioned convolution mode instruction and the aforementioned control instructions generated by the aforementioned accumulation state machine and transmission state machine, receiving external data and outputting calculation result data.

[0084] The multiplexer, in response to the convolution mode instruction and the control instruction generated by the accumulation state machine, inputs the external data from the first input queue received by the first-in-first-out memory into the arithmetic logic unit.

[0085] The output graph counter of the arithmetic logic unit responds to the control instructions generated by the accumulator state machine to determine the number of rows and columns of the output graph calculated by the arithmetic logic unit and control the size of the output graph.

[0086] The channel counter of the arithmetic logic unit responds to the control instructions generated by the sending state machine to determine the number of channel accumulations and the number of data accesses to the static random access memory during calculation.

[0087] The aforementioned arithmetic logic unit, including the adder, multiplier, shifter, and partial sum register, utilizes the number of channel accumulations and the number of data accesses to static random access memory during computation to perform calculations on the aforementioned external data, implementing vector multiplication, matrix multiplication, and matrix addition operations on the external data to obtain the calculation result data.

[0088] In response to the convolution mode instruction, the controller decodes the configuration parameters of the quantization register. The decoded configuration parameters are multiplied by the calculation result data through the multiplier to obtain the calculation result data with 8-bit or 16-bit data precision.

[0089] Figure 3 The diagram schematically illustrates a microarchitecture diagram of hardware for reconfigurable computation in convolutional mode according to embodiments of the present disclosure.

[0090] Figure 3 The dashed lines represent control signals in the reconfigurable processing unit, while the solid lines represent data signals in the reconfigurable processing unit. Empty and full signals generated by the first-in-first-out memory in either an empty or full state do not need to be reflected in the diagram. Mode configuration, configuration enable, channel configuration, and quantization configuration are all configuration and enable signals of the configuration control unit.

[0091] like Figure 3As shown, the cell mode register receives configuration information from the external bus and is set to a specific convolution mode instruction. The accumulator state machine and the transmitter state machine generate corresponding enable signals based on the specific convolution mode instruction. The row register, column register, channel register, and quantization register, in response to the corresponding enable signals, generate configuration parameters based on the configuration information received from the external bus. The decoder decodes the configuration parameters to generate control signals and inputs these control signals into the arithmetic logic unit. The control signals may include control signals generated by the accumulator state machine concerning the row registers and column registers, control signals generated by the transmitter state machine, and control signals concerning the channel registers and quantization registers.

[0092] The FIFO (First-In, First-Out) memory receives external data in response to a receive request from the accumulator state machine. This external data may include external data from both the first and second input queues. The FIFO memory outputs the computation result data in response to a send request from the sender state machine. The multiplexer receives external data from the first input queue of the FIFO memory and inputs it into the arithmetic logic unit in response to control instructions and specific convolution mode instructions.

[0093] The arithmetic logic unit (ALU) may include an output graph counter and a partial sum register. The output graph counter responds to control signals from the accumulator state machine regarding row and column registers, controlling the size of the output graph. The partial sum register responds to control signals from the accumulator state machine, temporarily storing partial sum data. The ALU responds to control signals from the channel register and quantizer register, performing multiplication-accumulation and data bit-width processing on the data. Furthermore, the ALU may also include adders, multipliers, activation units, shift registers, and channel counters. The ALU is primarily used to implement operations such as multiplication, addition, multiplication-accumulation, shifting, and comparison.

[0094] According to embodiments of this disclosure, the convolution mode mainly implements vector multiplication and matrix addition functions, and the specific configuration information is shown in Table 1 below:

[0095] Table 1

[0096]

[0097] Vector multiplication and addition function: For existing convolutional neural networks, the kernel size is usually 3×3. The kernel can be expanded into a one-dimensional vector operation with a length of 9.

[0098] As shown in function "1" in Table 1 above, setting spe_mode = "vect_mac", row_reg = "1", col_reg = "1", chan_add_reg = "9", and chan_add_re_reg = "1" completes a 9×1 vector accumulation operation and outputs the result outside the reconfigurable unit. The unit mode (spe_mode) parameter register sets the function of the arithmetic logic unit to multiply-accumulate operation and places the multiply-accumulate result in the internal partial sum (Psum) register. The multiply-accumulate unit in the arithmetic logic unit is obtained by using an adder and a multiplier in a second-order pipelined design, which can reduce the latency of the critical path. The multiply-accumulate unit only processes 8-bit input data, and the corresponding bit width of the partial sum register is set to 32 bits to ensure that the output result of the partial sum data will not overflow. The calculated result is truncated by multiplying by a scaling factor to ensure the accuracy of the output result. The repeat accumulation register (chan_add_re_reg) = 1 indicates that the result of the vector calculation is directly output to the output first-in-first-out memory.

[0099] As shown in function "2" of Figure 1 above, setting spe_mode = "vect_mac", row_reg = "2", col_reg = "2", chan_add_reg = "9", and chan_add_re_reg = "2" completes a convolution operation with a 2×2 kernel (both rows and columns), a kernel size of 9, and 2 input channels. First, the calculation result of the first channel is stored in static random access memory (SRAM). Then, the result of the second channel is calculated, and the result of the second channel is added to the result of the first channel to complete the multiplication and accumulation calculation of the two channels.

[0100] Matrix accumulation function: The reconfigurable processing unit supports streaming mode operation. Specifically, it retrieves a matrix through the first-in-first-out (FIFO) memory, stores the matrix in the static random access memory (SRAM), and then waits for the next configuration control unit to input a second matrix. At this time, it retrieves the first matrix from the SRAM and accumulates the second matrix. If chan_add_re_reg = "1", it outputs directly to the FIFO memory. If chan_add_re_reg is set to n, it waits for n matrices of the same size to be input before sending the output result to the output FIFO memory.

[0101] After completing the multiplication and accumulation operations of all input channels and convolution kernels, the output image needs to be quantized. To meet the different precision requirements of various deep learning algorithms, quantization of the output image data with different precision requirements can be performed by loading external input quantization parameters into the quantization parameter register, multiplying the output data by the input quantization parameters, and truncating to a specified bit width.

[0102] According to embodiments of this disclosure, the arithmetic logic unit further includes a maximum value register, a maximum value counter, and a comparator.

[0103] The above pooling mode instructions include max pooling instructions and average pooling instructions.

[0104] In response to the pooling mode instruction, the controller decodes the configuration information based on the decoder to obtain the control instruction.

[0105] The aforementioned first-in-first-out memory responds to the aforementioned pooling mode instruction and the aforementioned control instructions generated by the aforementioned loading state machine and output state machine, receiving external data and outputting calculation result data.

[0106] The convolution mode instruction of the multiplexer and the control instruction generated by the accumulator state machine input the external data of the second input queue received by the first-in-first-out memory into the arithmetic logic unit.

[0107] The output graph counter of the arithmetic logic unit responds to the control instructions generated by the loading state machine to determine the number of rows and columns of the output graph calculated by the arithmetic logic unit and control the size of the output graph.

[0108] The maximum value register of the arithmetic logic unit responds to the control instructions generated by the maximum value state machine and is used to store the maximum value of the max-pooling kernel during the calculation of each channel.

[0109] The maximum value register, maximum value counter, partial sum register, comparator, adder, and output graph counter of the above arithmetic logic unit use the above external data to perform calculations, realize the maximum pooling and average pooling operations on the above external data, and obtain the calculation result data.

[0110] Figure 4 The diagram schematically illustrates a microarchitecture diagram of hardware for reconfigurable computing in pooling mode according to embodiments of the present disclosure.

[0111] Figure 4 The dashed and solid lines indicate the empty and full signals generated by the first-in-first-out (FIFO) memory. Figure 3 The same applies, so I won't repeat it here.

[0112] like Figure 4As shown, the cell mode register receives configuration information from the external bus and is set to a specific convolution mode instruction. The load state machine, maximum state machine, and output state machine generate corresponding enable signals based on the specific convolution mode instruction. The row register, column register, and pooling mode register, in response to the corresponding enable signals, generate configuration parameters based on the configuration information received from the external bus. The decoder decodes the configuration parameters to generate control signals and inputs these control signals into the arithmetic logic unit. The control signals may include control signals sent by the load state machine regarding the row registers and column registers, as well as control signals sent by the load state machine regarding the pooling mode registers.

[0113] The FIFO (First-In, First-Out) memory receives external data in response to the receive request from the load state machine. The FIFO memory outputs the calculation result data in response to the send request from the output state machine. The multiplexer receives external data from the second input queue of the FIFO memory in response to control instructions and convolution mode instructions, and inputs the external data into the arithmetic logic unit.

[0114] In the arithmetic logic unit (ALU), the output graph counter responds to control signals from the loading state machine regarding row and column registers, controlling the size of the output graph. The maximum value register responds to control signals from the loading state machine regarding the pooling mode register, controlling the ALU to perform specific pooling operations. Partial pooling registers buffer data generated during max pooling, average pooling, and other operations. The ALU can be specifically expanded into adders, multipliers, comparators, a maximum value register, a maximum value counter, and another adder. The ALU primarily implements max pooling and average pooling operations.

[0115] According to embodiments of this disclosure, pooling operations with different requirements are completed by reusing static random access memory and corresponding arithmetic logic units, such as pooling operations with variable step sizes of pooling cores of 2 or 3. When an average pooling operation with a pooling core of 7 is completed, the calculation of the division unit is completed using the quantization factor cached in the quantization register, avoiding the use of a divider and reducing the size of the reconfigurable processing unit.

[0116] More specifically, you can set spe_mode = "max_pool" mode to enable the arithmetic logic unit to perform comparator functions. Before each max pooling calculation, the input graph is loaded into static random access memory via a multiplexer. Although this increases the data storage cycle compared to a dedicated pooling circuit, the "max_pool" mode ensures that the input graph can be pooled directly after convolution, and can perform pooling with different strides and different convolution kernels.

[0117] The static random access memory (SRAM) can be designed to be 1024×32. If the input graph is 8-bit data, it can support up to four channels pooling simultaneously at a time. At the beginning of each session, the first number is placed in the maximum value register (max_reg). If the maximum pooling kernel (kernel_size) is set to 2, the maximum result of 4 numbers in each window is output and placed in the maximum value register. If the maximum pooling kernel is set to 3, the maximum result of 9 numbers in each window is output and placed in the maximum value register.

[0118] The `spe_mode` option can be set to "avg_pool". In this mode, the arithmetic logic unit (ALU) calls the previous multiply-accumulate unit to perform the multiply-accumulate operation of the partial sums of the window. The ALU then multiplies the data obtained from the multiply-accumulate operation of the partial sums of the window by a scaling factor to perform the average calculation. If a 7×7 average pooling is performed, after the multiply-accumulate calculation of a vector of length 49, 49 is cached in the quantization register as a quantization factor. When used, it is multiplied by the reciprocal of 49, and then a truncation operation is performed to perform the division calculation, achieving the effect of average pooling.

[0119] According to embodiments of this disclosure, the reconfigurable processor can perform data calculations with arithmetic logic units of 8-bit or 16-bit data precision and storage units of 32-bit data precision as the smallest reconfiguration granularity. Increasing the data granularity can reduce the amount of configuration information required by the reconfigurable processor. The interconnection structure between reconfigurable units can increase the signal transmission bandwidth, thereby reducing the chip area while ensuring normal operation speed.

[0120] More specifically, the adders and multipliers in the arithmetic logic unit of the reconfigurable processor can perform data calculations with 8-bit or 16-bit precision. A multiply-adder, obtained by combining adders and multipliers through a second-order pipelined design, can perform data calculations with 8-bit precision. The static random access memory (SRAM) has a memory bus width of 32 bits. The reconfigurable processor's larger data granularity reduces the amount of configuration information required and improves its processing speed.

[0121] According to embodiments of this disclosure, the aforementioned static random access memory is further used for: responding to control information, performing activation function operations on the calculation result data with 8-bit data precision based on the aforementioned lookup table; responding to control information, storing the calculation result data with 16-bit data precision in a multi-segment polynomial fitting mode, and performing activation function operations on each segment of the calculation result data with 16-bit data precision based on the aforementioned lookup table to obtain multiple first-order activation function operation results.

[0122] The aforementioned arithmetic logic unit is also used to accumulate the results of the above multiple first-order activation function operations to obtain the activation function operation result with 16-bit data precision of the above calculation result data.

[0123] According to embodiments of this disclosure, the static random access memory (SRAM) can also store intermediate computational data of the arithmetic logic unit, such as storage portion and data. Using SRAM to store the intermediate computational data of the arithmetic logic unit can increase the access speed of the arithmetic logic unit to the intermediate computational data. On the other hand, when storing data, the SRAM performs quantization processing, and after the SRAM outputs the data, it reduces the storage space required for the data.

[0124] According to embodiments of this disclosure, after completing the calculation of the input channels, the network layer may also perform activation function operations, which may include ReLU function, Sigmoid function, Tanh function, etc.

[0125] The ReLU function outputs a positive value if it is greater than 0 when compared to 0, and can be directly calculated using the comparator of the arithmetic logic unit (ALU). In common reconfigurable processing unit designs, the Sigmoid and Tanh functions are typically implemented using lookup tables. Using the address of static random access memory (SRAM) as an index, the results of the Tanh and Sigmoid functions are pre-loaded into SRAM, completing the lookup table configuration. For typical 8-bit data, an 8-bit lookup table can be used in SRAM to retrieve the result. For 16-bit data, due to the size limitation of SRAM, a lookup table and polynomial fitting method are used to retrieve the result, achieving high-precision lookup of the 16-bit Sigmoid and Tanh functions. The Sigmoid function can be set to a multi-segment polynomial fitting mode, and the coefficients of the multi-segment first-order fitting function can be stored in the lookup table using the polyfit function. The multiply-accumulate function in the ALU is then used to implement the fitting method using a multi-segment linear function.

[0126] According to embodiments of this disclosure, since partial and register data can be accumulated into static random access memory, convolutional layers and pooling layers can perform layer fusion calculations. The spe_mode="conv_pool" mode can be set, and reconfigurable units can perform pooling after completing the convolution mode, reducing access to off-chip memory and improving calculation speed.

[0127] According to the embodiments of this disclosure, the calculation is configured to perform matrix multiplication and matrix addition on two 16×16 input graphs, max pooling with a 2×2 kernel and a stride of 2, and to calculate the Sigmoid function values ​​of 16 points of the result data. The performance of the configured calculation is evaluated, and the calculation results are shown in Table 2 below:

[0128] Table 2

[0129]

[0130] Area and power consumption evaluation (performance under SMIC 28nm process): The resource consumption and power consumption of the reconfigurable processing unit were evaluated at a clock frequency of 400MHz. The area and power consumption performance are shown in Table 3 below.

[0131] Table 3

[0132]

[0133] Reconfigurable processing units can achieve energy efficiency close to that of ASICs.

[0134] Error accuracy was evaluated, and the performance of the activation function was assessed in both 8-bit and 16-bit modes. The performance values ​​are shown in Table 4 below:

[0135] Table 4

[0136]

[0137] According to embodiments of this disclosure, the reconfigurable processing unit can also be configured to receive various configuration information and bus commands from an external bus to perform different computational operations. The function configuration table of the reconfigurable processing unit can be represented as shown in Table 5 below:

[0138] Table 5

[0139]

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0141] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A reconfigurable processing unit for deep learning, comprising: The configuration control unit is used to receive configuration information and bus commands from an external bus, generate mode instructions and control instructions based on the configuration information and bus commands, and obtain configuration parameters. A first-in-first-out (FIFO) memory is used to receive external data and output calculation result data according to the control instructions. A multiplexer is used to input the external data into an arithmetic logic unit or a static random access memory according to the mode instructions and control instructions. The arithmetic logic unit, in response to the control command, uses the configuration parameters to perform calculations on the external data from the multiplexer or static random access memory to obtain calculation result data with 8-bit or 16-bit data precision, and inputs the calculation result data into the static random access memory or the first-in-first-out memory. as well as A static random access memory, in response to the mode instructions and control instructions, is used to store calculation result data or external data generated by the arithmetic logic unit, output the calculation result data or external data to the arithmetic logic unit, or output the calculation result data to the first-in-first-out memory; The configuration control unit includes: A configuration register, used to receive configuration information from an external bus and generate the mode instructions; and The controller is used to receive bus commands from the external bus and generate enable signals; The configuration register is also used to obtain the configuration parameters based on configuration information according to the enable signal from the controller; and The controller is also configured to obtain the control command based on configuration information in response to the mode command; The configuration registers include: cell mode register, row register, column register, pooling mode register, channel register, and quantization register; The cell mode register receives configuration information from an external bus and generates mode instructions based on the configuration information. These mode instructions include convolution mode instructions and pooling mode instructions. When the convolution mode instruction is generated by the cell mode register, the row register, column register, channel register, and quantization register generate the configuration parameters based on the configuration information in response to the enable signal from the controller. When the pooling mode instruction is generated by the cell mode register, the row register, column register, and pooling mode register generate the configuration parameters based on the configuration information in response to the enable signal from the controller.

2. The reconfigurable processing unit according to claim 1, wherein, The controller includes: a decoder, an accumulator state machine, a transmit state machine, a load state machine, a maximum value state machine, and an output state machine; The accumulator state machine and the transmitter state machine are used to receive bus commands from the external bus in response to the convolution mode instruction, generate the enable signal, and obtain the control instruction based on the configuration information decoded by the decoder. The loading state machine, the maximum value state machine, and the output state machine are used to respond to the pooling mode instruction, receive bus commands from the external bus, generate the enable signal, and obtain the control instruction based on the configuration information decoded by the decoder.

3. The reconfigurable processing unit according to claim 2, wherein, The configuration control unit is used for: Determine the state of the first-in-first-out memory; When it is determined that the FIFO memory is empty, the FIFO memory does not output external data to the multiplexer, so that the reconfigurable processing unit does not perform calculations. as well as If the first-in-first-out (FIFO) memory is determined to be full, the FIFO memory will not acquire the calculation result data generated by the reconfigurable processing unit, so that the reconfigurable processing unit will not perform calculations. When the FIFO memory is not empty or full, the FIFO memory outputs external data to the multiplexer and outputs calculation result data to the reconfigurable processing unit, so that the reconfigurable processing unit can start calculation.

4. The reconfigurable processing unit according to claim 3, wherein, The multiplexer is used for: The multiplexer, in response to the control command and the convolution mode command, inputs the external data of the first input queue received by the first-in-first-out memory into the arithmetic logic unit or the static random access memory. The multiplexer, in response to the control command and the pooling mode command, inputs the external data from the second input queue received by the first-in-first-out memory into the arithmetic logic unit or the static random access memory.

5. The reconfigurable processing unit according to claim 4, wherein, The static random access memory is also used for: In response to the control command, the external data from the multiplexer is cached as lookup table data; and the intermediate calculation data of the arithmetic logic unit is stored.

6. The reconfigurable processing unit according to claim 5, wherein, The arithmetic logic unit includes: an adder, a multiplier, a shifter, an output graph counter, a channel counter, and a partial register; The convolution mode instructions include multiplication and addition instructions; The controller responds to the convolution mode instruction, and the accumulator state machine and the transmitter state machine decode the configuration information based on the decoder to obtain the control instruction; The first-in-first-out memory responds to the convolution mode instruction and the control instructions generated by the accumulation state machine and the sending state machine, receiving external data and outputting calculation result data. The multiplexer, in response to the convolution mode instruction and the control instruction generated by the accumulator state machine, inputs the external data of the first input queue received by the first-in-first-out memory into the arithmetic logic unit; The output graph counter of the arithmetic logic unit responds to the control command generated by the accumulator state machine to determine the number of rows and columns of the output graph calculated by the arithmetic logic unit and control the size of the output graph; The channel counter of the arithmetic logic unit responds to the control command generated by the transmitting state machine to determine the number of channel accumulations during calculation and the number of data accesses to the static random access memory during calculation. The arithmetic logic unit, comprising adders, multipliers, shifters, and partial registers, performs calculations on the external data using the number of channel accumulations and the number of data accesses to static random access memory during computation. This enables vector multiplication, matrix multiplication, and matrix addition operations on the external data to obtain the computational result data. In response to the convolution mode instruction, the controller decodes the configuration parameters of the quantization register, and the decoded configuration parameters are multiplied by the calculation result data through the multiplier to obtain the calculation result data with 8-bit data precision or 16-bit data precision.

7. The reconfigurable processing unit according to claim 5, wherein, The arithmetic logic unit also includes a maximum value register, a maximum value counter, and a comparator; The pooling mode instructions include max pooling instructions and average pooling instructions; The controller responds to the pooling mode instruction, and the loading state machine, the maximum value state machine, and the output state machine decode the configuration information based on the decoder to obtain the control instruction; The first-in-first-out memory responds to the pooling mode instruction and the control instructions generated by the loading state machine and the output state machine, receiving external data and outputting calculation result data. The convolution mode instruction of the multiplexer and the control instruction generated by the accumulator state machine input the external data of the second input queue received by the first-in-first-out memory into the arithmetic logic unit; The output graph counter of the arithmetic logic unit responds to the control command generated by the loading state machine to determine the number of rows and columns of the output graph calculated by the arithmetic logic unit and control the size of the output graph; The maximum value register of the arithmetic logic unit responds to the control instruction generated by the maximum value state machine and is used to store the maximum value of the max-pooling kernel during the calculation of each channel; The maximum value register, maximum value counter, partial sum register, comparator, adder, and output graph counter of the arithmetic logic unit use the external data to perform calculations, realizing maximum pooling and average pooling operations on the external data to obtain the calculation result data.

8. The reconfigurable processing unit according to any one of claims 6-7, wherein, The static random access memory is further used for: responding to a control command, performing activation function operations on the 8-bit data precision calculation result data based on the lookup table; responding to a control command, storing the 16-bit data precision calculation result data in a multi-segment polynomial fitting mode, and performing activation function operations on each segment of the 16-bit data precision calculation result data based on the lookup table to obtain multiple first-order activation function operation results; as well as The arithmetic logic unit is also used to accumulate the results of the multiple first-order activation functions to obtain the activation function operation result with 16-bit data precision of the calculation result data.

Images