A configurable convolution operation acceleration device and method based on systolic array

Abstract

The present invention discloses a configurable convolution operation acceleration device and method based on a systolic array, and belongs to the technical field of neural network hardware acceleration of ultra-large-scale integrated circuits. Aiming at the problems existing in the existing convolution operation acceleration device, the present invention proposes a systolic array with an improved operation mode, that is, an operation mode of broadcasting in the row direction and pulsating in the column direction is adopted, and the intermediate calculation results are fixed in PE, and there is no need to stop the operation to update the weight, which greatly improves the PE utilization rate, makes the actual computing power close to the theoretical peak computing power, and because the row direction broadcasting mode is adopted, the working state of each row of PE is the same, so that the number of PEs that need to output the operation results in each cycle is the same, which effectively ensures the stability of the input and output data throughput rate and fully utilizes the data bandwidth of the output channel.

Description

Technical Field

[0001] The invention relates to a configurable convolution operation acceleration device and method based on a systolic array, and belongs to the technical field of neural network hardware acceleration of very large scale integrated circuits. Background Art

[0002] Convolution operations are core operations in the field of modern computing, especially in application areas such as image processing, computer vision, and deep learning. With the rapid development of artificial intelligence and machine learning technologies, the computational complexity of convolution operations has increased dramatically, placing higher demands on computing performance and efficiency. Systolic arrays, as a hardware architecture for parallel computing, can multiplex input data to achieve efficient parallel computing by distributing computing tasks to multiple processing units (PEs) and transmitting data through fixed communication paths. Therefore, they are widely used for hardware acceleration of convolution operations. However, with the continuous emergence of convolution operations of different scales and types, the traditional systolic array and its peripheral circuit design will reflect the following problems when accelerating convolution operations:

[0003] 1. When the traditional systolic array uses an array working mode in which the intermediate calculation results are fixed in each PE and the input data and weights are transmitted in a pulsating manner in the row and column directions, the PEs in the same operation state during the convolution operation are arranged along the diagonal direction of the array. The number of PEs that need to output the operation results in different cycles is different. The overall throughput of the systolic array fluctuates greatly, and the output channel bandwidth cannot be fully utilized, resulting in performance loss. At the same time, the above irregular array working mode also requires the design of additional control and data re-arrangement logic in the input data cache and weight cache, which increases the design complexity.

[0004] 2. When the traditional systolic array performs convolution operations by fixing the weights in each PE, pulsating the input data in the row direction, and pulsating the output data in the column direction, the systolic array can solve the problem of large fluctuations in the overall array throughput in the previous problem after starting pipeline operations. However, since the existing array size is usually unable to meet the requirements of fixing all the weights of the convolution layer on the array, it is necessary to stop the operation at intervals and update the fixed weights in the PE. Therefore, the PE utilization rate of the array is low, and the actual computing power is greatly different from the theoretical peak computing power;

[0005] 3. In traditional systolic arrays, each PE in the array needs to communicate not only with adjacent PEs, but also with caches outside the array to update weights or output calculation results. Therefore, as the size of a single array continues to expand, the systolic array will be prone to wiring congestion, making it difficult to deploy a complete convolutional layer.

[0006] 4. The design of the traditional systolic array peripheral circuits has not been jointly optimized to improve the versatility of the convolution mode and save hardware overhead. When supporting configurable convolution modes, the existing array peripheral circuit designs (including input data cache, weight cache, scheduling controller, etc.) design dedicated control logic for different modes or use larger data caches, which does not fully utilize the high data reusability of convolution calculations. The instruction operation of configurable convolution modes is not designed, and it is more complicated to schedule it as a heterogeneous acceleration core;

[0007] Therefore, how to design an efficient and configurable systolic array architecture to meet the needs of different convolution operations remains the focus and difficulty of current research. Summary of the invention

[0008] In order to optimize the above problems, reduce the area and power consumption of hardware, improve the performance and energy efficiency of the system, realize configurable acceleration of convolution operations in multiple modes, and provide a hardware architecture that is easy to integrate into a general processing system as a heterogeneous acceleration core, the present invention provides a configurable convolution operation acceleration device and method based on a systolic array.

[0009] The technical solution adopted by the present invention is as follows:

[0010] A configurable convolution operation acceleration device based on a systolic array includes an instruction memory, a register group, a scheduler, an arbitrator, a weight cache, an input cache, a sliding window cache and a systolic array; wherein the instruction memory is used to store instruction programs applicable to the device; the register group is used to store the device working status such as the feature map size, the number of input / output channels and related instruction operands; the scheduler is used to read the instructions stored in the instruction memory and control the operation of the corresponding module; the arbitrator is used to select to send the input data of the device to the input cache or the weight cache under the control of the scheduler; the weight cache is used to store the weight data for convolution operation; the input cache is used to store part of the feature map data by row and send the sliding window data in batches according to the size of the systolic array; the sliding window cache is used to receive the sliding window data output by the input cache and output it in the format applicable to the systolic array; the systolic array is used to broadcast in the row direction and pulsate in the column direction to complete the operation. The device realizes the function of accelerating the convolution operation based on the systolic array with improved operation mode and the configurable weight cache, input cache and sliding window cache matching its data stream; the function of configuring the above structure is realized based on the instruction memory, the register group, the scheduler and the arbitrator. The device supports acceleration of convolution operations of different modes, including convolution modes with convolution kernel sizes of 3*3 and 1*1 and sliding window step sizes of arbitrary values.

[0011] Furthermore, the systolic array of the improved operation method is composed of processing units PE that are broadcast in the row direction and pulsate in the column direction on a scale of n rows and m columns. Each PE is composed of a multiplier and two registers. The multiplier implements the multiplication and accumulation function, and the two registers respectively store the intermediate results of the multiplication and accumulation and the data pulsating in the column direction. When performing convolution operations, the data from the sliding window cache is broadcast in the row direction and input to all PEs in each row, and the data from the weight cache is input in the column direction and input to all PEs in the first row, and pulsatingly transmitted to the corresponding PE in the next row in each subsequent clock cycle. The systolic array controls each PE to output the result from the register after performing several multiplication and accumulation operations according to the number of input channels Chin in the register group. As a result, the PEs in the row direction in the systolic array are all in the same working state, and the array outputs the calculation results of a row of PEs in each clock cycle.

[0012] Furthermore, the device adopts a custom instruction set to realize the scheduling of the systolic array peripheral circuit. In addition to the classic ADD, SUB, JMP and other instructions, WWGT, WACT and RACT instructions are designed. WWGT is a write weight instruction, which controls the weight cache to receive the weight data of the immediate number WLen length; WACT is a write stimulus instruction, which controls the SRAM width address, Bank address, start address and write length of the stimulus written by the input cache when executing the current instruction according to the register values ​​of the register group address corresponding to WdAddr, BkAddr, StAddr and WLen, and the immediate number Zero controls whether the written value is set to zero; RACT is a read stimulus instruction, which controls the read address interval, SRAM width address, Bank address, start address and read length of the stimulus read out by the input cache when executing the current instruction according to the register values ​​of the register group address corresponding to RInvl, WdAddr, BkAddr, StAddr and RLen.

[0013] Furthermore, the input buffer is composed of at least five dual-port RAMs with a width of 8n bits, each RAM is named a Bank, and each Bank is used to store a row of data in the input feature map. At the same time, the input buffer contains an instruction transaction queue adapted for WACT and RACT instructions, which can receive several WACT and RACT instructions according to the configuration of the register group, and the scheduler can execute other subsequent instructions in sequence.

[0014] When the scheduler decodes the write input instruction WACT, the input cache uses the bit write function of the RAM to write the data from the arbitrator to the corresponding bit of the corresponding address of the corresponding Bank according to the decoded width address WdAddr, Bank address BkAddr, start address StAddr and write length WLen, combined with the input channel number Chin and input feature map size FSize in the register group. In particular, when the zero value flag Zero is 1, the input cache writes the zero value to the corresponding bit of the corresponding address of the corresponding Bank. When the scheduler decodes the read input instruction RACT, the input cache reads the 8n bit data of the corresponding address of the corresponding Bank according to the decoded WdAddr, BkAddr, StAddr, read length RLen and read interval RInvl, combined with Chin and FSize in the register group, and sends it to the sliding window cache.

[0015] Furthermore, the sliding window cache is composed of n 8n bit register groups, and the data of each register group corresponds to a row of PE in the systolic array. When receiving data sent by the input cache, a currently idle register group is selected (if multiple idle register groups are selected, the register group corresponding to the upper row of the array is selected first). According to the information obtained by decoding the instruction RACT and Chin and FSize, the valid data in the 8n bit is shifted to the low bit and stored, and then 8 bits of data are sent to the systolic array each time by shifting. When all valid data are sent to the systolic array, the corresponding register group is idle and can continue to receive data from the input cache. The weight cache outputs the weights in the sliding window cache in a first-in-first-out manner to each PE in the first row of the systolic array by pre-fetching.

[0016] The present invention provides an acceleration method using the configurable convolution operation acceleration device based on the systolic array, comprising:

[0017] Step (1), before starting the operation, the external controller writes the instruction program into the instruction memory through the instruction address and instruction data port. The instruction program first includes a number of CONST instructions for configuring the register group, then includes a number of WWGT instructions for writing the weights into the weight cache, and finally includes a number of WACT, RACT, ADD, SUB, MUL, DIV, CMP and JMP instructions for cyclically writing and reading the input cache to perform convolution operations;

[0018] Step (2), the external controller sends a start signal to the scheduler. After receiving the signal, the scheduler starts to sequentially read the instruction program in the instruction memory and execute it. When the CONST instruction is executed, the register group is configured, and then the input feature map size FSize, the number of input channels Chin, the number of output channels Chout and other information in the current convolution mode are configured;

[0019] Step (3), when the scheduler executes the WWGT instruction, the control arbitrator sends the input data to the weight cache until the weight data of the write length WLen is successfully written;

[0020] Step (4), when the scheduler executes the WACT instruction, the arbitrator is controlled to send the input data to the input cache, and the input cache caches the data according to the SRAM width address, bank address, start address and write length information of the instruction decoding; when the scheduler executes the RACT instruction, the input cache is controlled to read the data into the sliding window cache. At the same time, since the input cache contains an instruction transaction queue, when the scheduler executes the above two instructions, the subsequent instructions can be executed sequentially when the instructions are not executed. Therefore, the device of the present invention can realize the pipeline of reading and writing the input cache to ensure the high utilization rate of the systolic array.

[0021] Step (5), when the scheduler executes the ADD, SUB, and MUL instructions, it adds / subtracts / multiplies the register values ​​at the corresponding addresses according to the instruction operand addresses Rs and Rt and stores them in the register at the address corresponding to the instruction operand address Rd; when the scheduler executes the DIV instruction, it divides the register values ​​at the corresponding addresses according to the Round value to determine whether to round up or down and stores them in the register at the address corresponding to the instruction operand address Rd; when the scheduler executes the CMP instruction, it compares the register values ​​at the addresses corresponding to Rs and Rt and stores them in the comparison result register; when the scheduler executes the JMP instruction, it compares the jump condition value Jug and the value of the comparison result register. If the two are equal, the value of the instruction counter in the scheduler is updated to the immediate number Imm. Combining the above three instructions, the instructions in step (4) can be executed cyclically while updating information.

[0022] Step (6): When the scheduler executes the DONE instruction, the scheduler sends a completion signal, indicating that the current convolutional layer calculation is completed.

[0023] The device and method described in the present invention can realize parallel convolution operations of feature maps and 32 convolution kernels in a convolution mode with a convolution kernel size of 3*3 or 1*1, a convolution kernel step size of 2 or 1, and a maximum data amount of 10KB per row of the input feature map.

[0024] The beneficial effects of the present invention are:

[0025] The present invention adopts a systolic array with an improved operation mode, that is, the systolic array adopts an operation mode of broadcasting in the row direction and pulsating in the column direction, and fixes the intermediate calculation results in PE, without stopping the operation to update the weight. Therefore, the PE utilization rate of the present invention is high, and the actual computing power is close to the theoretical peak computing power; and because the broadcast mode in the row direction is adopted, the working state of each row of PE is the same, so that the number of PEs that need to output the calculation results in each cycle is the same, which effectively ensures the stability of the input and output data throughput rate and fully utilizes the data bandwidth of the output channel.

[0026] In addition, the present invention selects a balanced systolic array size, that is, considering the needs of most application scenarios, the systolic array is set to 32×32, which can effectively calculate while avoiding wiring congestion. For the case of more than 32 convolution kernels in the convolution mode, thanks to the custom instruction set and device use method, the device of the present invention can perform parallel computing between device clusters by multiplexing the device of the present invention to form an on-chip network.

[0027] Finally, while adapting to convolution operations in various modes, the device and method fully reuse the input feature map data during convolution operations, thereby significantly reducing the hardware resource overhead caused by data caching, thereby improving the performance and energy efficiency of the system; its customized instruction set and device usage method facilitate the integration of the device with a general processing system, providing an effective solution for the hardware acceleration deployment of convolutional neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A schematic diagram of the structure of a configurable convolution operation acceleration device based on a systolic array provided by the present invention.

[0030] Figure 2 The figure is a schematic diagram of the operation principle of the systolic array with improved operation mode in the device of the present invention.

[0031] Figure 3 It is a structure diagram of a custom instruction set suitable for the device of the present invention.

[0032] Figure 4 This is a diagram of the operation principle of the convolutional neural network.

[0033] Figure 5 Schematic diagram of data flow of sliding window cache and weight cache data entering the systolic array under the feature map viewing angle in an embodiment of the present invention.

[0034] Figure 6 Schematic diagram of data flow of sliding window cache and weight cache data entering the systolic array from the array viewing angle in an embodiment of the present invention.

[0035] Figure 7 Schematic diagram of the state machine in the device scheduler of the present invention.

[0036] Figure 8 A schematic diagram of a state machine in the input buffer of the device of the present invention.

[0037] Fig. 9 Schematic diagram of a storage structure in which the input cache is composed of five RAMs in an embodiment of the present invention.

[0038] Fig.10 Schematic diagram of the order in which the input cache reads the convolution sliding window data.

[0039] Fig.11 Schematic diagram of the data types that are initially written into RAM according to an embodiment of the present invention.

[0040] Fig.12 Schematic diagram of the address mapping relationship after the input feature map is written into the RAM in an embodiment of the present invention.

[0041] Fig.13 It is a schematic diagram of reading out the first 32 channels of the upper left data in 32 sliding windows in an embodiment of the present invention.

[0042] Fig.14 It is a schematic diagram of reading out the first 32 channels of data directly above the 32 sliding windows in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] Embodiment 1

[0045] This embodiment provides a configurable convolution operation acceleration device based on a systolic array, see Figure 1The configurable convolution operation acceleration device includes: an instruction memory, a register group, a scheduler, an arbiter, a weight cache, an input cache, a sliding window cache and a systolic array; wherein the instruction memory is used to store instruction programs applicable to the device; the register group is used to store device working states such as feature map size, input / output channel number and related instruction operands; the scheduler is used to read the instructions stored in the instruction memory and control the operation of the corresponding module; the arbiter is used to select to send the input data of the device to the input cache or the weight cache under the control of the scheduler; the weight cache is used to store weight data for convolution operation; the input cache is used to store part of the feature map data by row and send sliding window data in batches according to the size of the systolic array; the sliding window cache is used to receive the sliding window data output by the input cache and output it in a format applicable to the systolic array; the systolic array is used to broadcast in the row direction and pulsate in the column direction to complete the operation.

[0046] like Figure 1 As shown, the instruction memory is connected to the scheduler to send the instruction program to the scheduler for execution; the scheduler is connected to the register group to update the status register value therein; the scheduler is also connected to the arbitrator, input cache and weight cache to control its working status under different instructions; the register group is connected to the input cache, sliding window cache, weight cache and pulsation array to send the status register value to the above modules; the arbitrator is also connected to the input cache and the weight cache, and under the control of the scheduler, the input data is sent to the input cache or the weight cache according to different instructions; the input cache is connected to the sliding window cache to send the data required for each sliding window to the latter; the sliding window cache is connected to the pulsation array to transmit the input data in the row direction; the weight cache is connected to the pulsation array to transmit the weight data in the column direction.

[0047] like Figure 2 As shown, the systolic array consists of processing units PE that are broadcast in the row direction and pulsated in the column direction with a scale of n rows and m columns. Each PE consists of a multiplier and two registers. The multiplier implements the multiplication and accumulation function, and the two registers store the intermediate results of the multiplication and accumulation and the data pulsating in the column direction. When performing convolution operations, the data from the sliding window buffer is broadcast in the row direction and input to all PEs in each row, and the data from the weight buffer is input in the column direction and pulsated to the corresponding PE in the next row in each subsequent clock cycle. The systolic array controls each PE to output the result from the register after performing several multiplication and accumulation operations according to the number of input channels Chin in the register group. As a result, the PEs in the row direction in the systolic array are all in the same working state, and the array outputs the calculation results of a row of PEs in each clock cycle.

[0048] The device uses a custom instruction set to implement the scheduling of the systolic array peripheral circuits, such as Figure 3As shown, in addition to the classic ADD, SUB, JMP and other instructions, the scheme of the present invention also designs WWGT, WACT and RACT instructions, wherein WWGT is a write weight instruction, which controls the weight cache to receive the weight data of the immediate number WLen length; WACT is a write stimulus instruction, which controls the SRAM width address, Bank address, start address and write length of the stimulus written by the input cache when executing the current instruction according to the register values ​​of the register group address corresponding to WdAddr, BkAddr, StAddr and WLen, and the immediate number Zero controls whether the written value is set to zero; RACT is a read stimulus instruction, which controls the read address interval, SRAM width address, Bank address, start address and read length of the stimulus read out by the input cache when executing the current instruction according to the register values ​​of the register group address corresponding to RInvl, WdAddr, BkAddr, StAddr and RLen. In the acceleration device, the input cache is composed of at least five dual-port RAMs with a width of 8n bits, each RAM is named a Bank, and each Bank is used to store a row of data in the input feature map. At the same time, the input buffer contains a command transaction queue adapted for WACT and RACT instructions, and its internal working state is as shown in the attached figure. Figure 8 As shown, according to the configuration of the register group, several WACT and RACT instructions can be received, and the scheduler can execute other subsequent instructions in sequence. When the scheduler decodes the write input instruction WACT, the input cache uses the bit write function of the RAM to write the data from the arbitrator to the corresponding bit of the corresponding address of the corresponding Bank according to the width address WdAddr, Bank address BkAddr, start address StAddr and write length WLen obtained by decoding, combined with the number of input channels Chin and the input feature map size FSize in the register group. In particular, when the zero value flag Zero is 1, the input cache writes the zero value to the corresponding bit of the corresponding address of the corresponding Bank. When the scheduler decodes the read input instruction RACT, the input cache reads the 8n bit data of the corresponding address of the corresponding Bank according to the decoded WdAddr, BkAddr, StAddr, read length RLen and read interval RInvl, combined with Chin and FSize in the register group, and sends it to the sliding window cache.

[0049] In the acceleration device, the sliding window cache is composed of n 8n bit register groups, and the data of each register group corresponds to a row of PE in the systolic array. When receiving the data sent by the input cache, a currently idle register group is selected (if multiple idle register groups are selected, the register group corresponding to the upper row of the array is selected first). According to the information obtained by decoding the instruction RACT and Chin and FSize, the valid data in the 8n bit is shifted to the low bit and stored, and then 8 bits of data are sent to the systolic array each time by shifting. When all the valid data are sent to the systolic array, the corresponding register group is idle and can continue to receive data from the input cache. The weight cache outputs the weights in the sliding window cache in a first-in-first-out manner to each PE in the first row of the systolic array by pre-fetching.

[0050] The classical instructions involved in the scheme of the present invention include CONST instruction, ADD instruction, SUB instruction, MUL instruction, DIV instruction, CMP instruction, JMP instruction and DONE instruction, etc.; wherein, the CONST instruction is used to store the information of the feature map size, the number of input channels and the number of output channels in the register group, the ADD, SUB and MUL instructions are respectively used to add / subtract / multiply the register values ​​corresponding to the instruction operand addresses Rs and Rt and store them in the register corresponding to the address of the instruction operand address Rd, the DIV instruction is used to divide the corresponding register values ​​according to the instruction operand addresses Rs and Rt by judging whether to round up or down according to the value of Round and store them in the register corresponding to the address of the instruction operand address Rd, the CMP instruction is used to compare the register values ​​corresponding to the instruction operand addresses Rs and Rt and store them in the comparison result register, the JMP instruction is used to compare the jump condition value Jug and the value of the comparison result register, if the two are equal, the value of the instruction counter in the scheduler is updated to the immediate number Imm, and the DONE instruction is used to send a completion signal to the external controller to indicate that the calculation is completed.

[0051] Embodiment 2

[0052] This embodiment provides an acceleration method using the configurable convolution operation acceleration device based on the systolic array provided in the first embodiment. In this embodiment, a convolution layer with a convolution kernel size of 3*3, a step size of 1, a feature map size of 64*64, an input channel number of 128, and an output channel number of 128 is deployed on the acceleration device provided by the present invention, and the method of the present invention is described by taking the process of executing the instruction program of the device of the present invention as an example. Figure 4 As shown, in this embodiment, the first 32 output channels are deployed, and the remaining 96 output channels are deployed on three other identical acceleration devices.

[0053] Before describing the overall workflow of the acceleration device, the working principle of the systolic array in the acceleration device is first described. Specifically, in this embodiment, the systolic array is composed of 32 rows and 32 columns of PEs. The data from the sliding window cache is broadcasted in the row direction and input to all PEs in each row. The data from the weight cache is input in the column direction and systolic to the PEs corresponding to the next row in each subsequent clock cycle. The systolic array controls each PE to output the result from the register after performing several multiplication and accumulation operations according to the number of input channels Chin in the register group. Specifically, in the convolution operation that completes the complete input feature map, as shown in FIG. Figure 5 and Figure 6 As shown, each row of PE receives the data of the continuous sliding window of the input feature map in the row direction, such as the first row of PE receives the data of all input channels in the 9 points of the first sliding window, and the second row of PE receives the data of all input channels in the 9 points of the second sliding window. At the same time, each column of PE receives the weight data corresponding to the input sliding window data in the column direction, so that after each PE calculates 3*3*Chin times of multiplication and accumulation, it can output the convolution operation result of the corresponding sliding window. Among them, the 32 data output by each row of PE is the data of 32 channels at a certain point in the output feature map. When each row of PE completes the operation of the corresponding sliding window data, it can start to receive the next sliding window data, and the weight data only needs to be repeatedly sent to the array to realize the pipelining of the operation.

[0054] After the description of the working principle of the systolic array is completed, the overall working process of the device is described below, including the input buffer (its structure in this embodiment is shown in the attached figure). Fig. 9 As shown), the workflow of sliding window cache, weight cache and other modules:

[0055] Before starting the convolution operation, the instruction program is written into the instruction memory through the instruction address and instruction data. Then, a start signal is sent to the device, and the scheduler immediately starts to read and execute instructions from the instruction memory in sequence. The instruction execution state machine in the scheduler is shown in the attached figure. Figure 7 As shown, the state machine is in the standby state when it is reset, and enters the instruction fetch, decoding, and execution states in sequence after startup. After the instruction is executed, the state machine enters the update state from the execution state, which updates the value of the register group. Then the state machine enters the instruction fetch state again until the DONE instruction is executed, then it no longer enters the instruction fetch state but enters the completion state, and finally returns to the standby state.

[0056] Generally, when the state machine is in the execution state, it must wait for the instruction to be executed before entering the update state. In particular, when executing instructions WACT and RACT, if the corresponding transaction queue in the input cache is not full, the state machine can directly enter the update state to execute the next instruction without waiting for the instruction to be executed; if the corresponding transaction queue is full, it is necessary to wait. The instruction program steps that need to be executed in this embodiment are as follows (since the purpose of designing instructions such as ADD and SUB is to assist the execution of WACT and RACT instructions, the WACT and RACT instructions are mainly described here):

[0057] In step (1), the first thing to be executed is a number of CONST instructions. When executing these instructions, the scheduler writes information such as the convolution kernel size, step size, number of feature map rows, number of feature map columns, number of input / output channels, number of systolic array rows, etc. into the register group, and sets the relevant counters used subsequently to zero and stores them in the registers.

[0058] Step (2) The scheduler executes instruction WWGT, controls the arbiter to receive input data and distribute it to the weight cache. After the weight cache receives data at 1152 addresses, the instruction is executed and the scheduler executes the next instruction. The reason why data at 1152 addresses is received here is that 1152=3*3*128 is the amount of data for a convolution kernel, and the width of the RAM in the weight cache is 32*8 bits. Therefore, this instruction writes the weight data of 32 convolution kernels into the weight cache.

[0059] Step (3) The scheduler executes the instruction WACT to control the arbiter to receive the input data and distribute it to the input buffer. Due to the existence of padding, a total of 7 WACT instructions are executed, as shown in the attached figure. Fig.11 As shown in the figure, the gray shadow is the data of the input feature map, the green shadow is the padding zero value, and the serial number in the figure indicates the data position written each time the WACT instruction is executed. The first WACT writes all the padding zero values ​​above the input feature map into Bank0; the second WACT writes the padding zero value on the left side of the first row of the input feature map into the starting four addresses of Bank1 (because one address can store 32 channels of input data, and the number of input channels in this embodiment is 128); the third WACT writes the first row of the input feature map to Bank1; the fourth WACT writes the padding zero value on the right side of the first row of the input feature map into the next four addresses of Bank1; the fifth to seventh WACTs are similar to the second to fourth times, except that the second row of the input feature map and its corresponding padding zero value are written into Bank2. The storage method of the input feature map data in each Bank of SRAM after writing is completed in this embodiment is shown in the attached figure. Fig.12As shown, taking Bank1 as an example, the width of the SRAM is 32*8 bits and the depth is 320 bits. Each address stores 32-channel data of a point in the feature map. Four consecutive addresses store the complete 128-channel data of a point, and a total of 66 points of data including left and right padding are stored.

[0060] Step (4) After completing step (3) above, the device can use the data from Bank 0 to 2 to start the convolution sliding window operation. In this embodiment, since the number of rows of the systolic array is 32, the array can perform convolution operations on 32 sliding windows in a row in parallel. From the perspective of the feature map, the data in a sliding window is read out in a row-by-row reading manner. In order to achieve the above purpose, a total of 36 RACT instructions need to be executed. From the perspective of the SRAM, the first RACT instruction is executed as follows: Fig.13 The method in the above example sequentially converts the corresponding 32 consecutive sliding windows into Fig.10 The data of the first 32 channels in the upper left center are read out and sent to the sliding window buffer. Fig.13 The first 32 sliding windows in three banks are shown in the figure, where the shaded area indicates the address location of the corresponding read data in the bank. The sliding window cache then stores the 32 sets of data in 32 256-bit register groups in its module and reads them according to the following formula: Figure 6 The form in the above example sends 8 bits of data to the systolic array each time by shifting. After the data in the register group is output, the input cache can execute the next RACT instruction. When the RACT instruction is executed for the second to fourth time, the corresponding bits in the continuous 32 sliding windows are Fig.10 The data of the subsequent 96 channels in the upper left corner of the data are read out and sent to the sliding window buffer, and the above process is repeated. When the RACT instruction is executed for the fifth time, the following steps are performed: Fig.14 The method in the above example sequentially converts the corresponding 32 consecutive sliding windows into Fig.10 The data of the first 32 channels in the data just above the center are read out and sent to the sliding window buffer. When the RACT instruction is executed for the 6th to 8th time, the data of the subsequent 96 channels are read out similarly to the 2nd to 4th time. When the RACT instruction is executed for the 9th to 12th time, the corresponding 32 channels in the continuous sliding window are read out. Fig.10 The data in the upper right corner of the figure is read out and sent to the sliding window buffer. Similarly, when the RACT instruction is executed for the 13th to 24th time, the corresponding data in the 32 consecutive sliding windows are read out and sent to the sliding window buffer. Fig.10 The data of the center left, center center, and center right are read out and sent to the sliding window buffer; when the RACT instruction is executed for the 25th to 36th time, the corresponding data in the 32 consecutive sliding windows are read out in turn. Fig.10 The data at the lower left, bottom, and bottom right are read out and sent to the sliding window buffer.

[0061] After completing the above 36 RACT instructions, the device completes a row of 32 consecutive sliding windows convolution operations. Since the number of columns of the input feature map in this embodiment is 64, a row needs to perform 64 sliding window convolution operations. Repeating the above operation can complete the subsequent 32 sliding windows.

[0062] Step (5) During the execution of step (4), the SRAM corresponding to Bank0~2 is in the read state and is responsible for executing several RACT instructions to perform sliding window convolution operations. At this time, the SRAM corresponding to Bank3 is in the write state to execute the WACT instruction in step (3), so that the device can store the feature map data of the next row while performing the sliding window convolution operation.

[0063] Step (6) After both step (4) and step (5) are completed, step (4) and step (5) are repeated, as shown in Table 1 below. The first execution of step (4) and step (5) is loop number 1, in which Bank0~2 is read and Bank3 is written. In this step, step (4) and step (5) are executed for loop number 2. The difference from the previous execution is that when step (4) is executed this time, the device uses the data of Bank1~3 to start the convolution sliding window operation. When step (5) is executed this time, the device writes a new row of input feature map data into Bank0, as shown in loop number 2 in the table below. In this embodiment, the number of input feature map rows is 64. Therefore, after completing the loop in the table below, in the 64th loop, the device uses the data of Bank3~1 to perform the convolution sliding window operation. At this time, there is no need to write data to Bank2. When the convolution sliding window operation of the data of Bank3~1 is completed, the scheduler executes the DONE instruction, indicating that the convolution operation of this layer has been completed, and then outputs a completion signal.

[0064] Table 1, Bank numbers read and written at different cycle times

[0065]

[0066] Finally, in step (7), if a single device is selected for serial operation, steps (2) to (6) must be repeated to complete the operation of the remaining 96 convolution kernels. If a device cluster is selected for parallel operation, the above steps are performed using a cluster of four devices to complete the operation of each 32 convolution kernel.

[0067] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A configurable convolution operation acceleration device based on a systolic array, characterized in that: The device includes the following modules: an instruction memory, a register group, a scheduler, an arbitrator, a weight cache, an input cache, a sliding window cache and a systolic array; wherein the instruction memory is used to store an instruction program applicable to the device; the register group is used to store the working status of the device and instruction operands; the scheduler is used to read the instructions stored in the instruction memory and control the operation of the corresponding modules; the arbitrator is used to select to send input data to the input cache or the weight cache under the control of the scheduler; the weight cache is used to store weight data for convolution operations; the input cache is used to store feature map data by row and send sliding window data in batches according to the size of the systolic array; the sliding window cache is used to receive the sliding window data output by the input cache and output it in a format applicable to the systolic array; the systolic array is used to broadcast in the row direction and pulsate in the column direction to complete the operation; The systolic array is composed of processing units PE that are broadcast in the row direction and pulsated in the column direction with a scale of n rows and m columns. Each PE is composed of a multiplier and two registers. The multiplier implements the multiplication and accumulation function, and the two registers respectively store the intermediate results of the multiplication and accumulation and the data pulsating in the column direction. When the systolic array performs convolution operation, the data from the sliding window buffer is broadcasted in the row direction and input into all PEs of each row, and the data from the weight buffer is input into all PEs of the first row in the column direction, and pulsatingly transmitted to the corresponding PE of the next row in each subsequent clock cycle. The systolic array controls each PE to output the result from the register after performing several multiplication and accumulation operations according to the number of input channels in the register group. All PEs in the row direction of the systolic array are in the same working state, and the array outputs the calculation result of a row of PEs in each clock cycle.

2. The device according to claim 1, characterized in that The instruction memory stores write weight instructions WWGT, write excitation instructions WACT and read excitation instructions RACT.

3. The device according to claim 2, characterized in that The input buffer is composed of at least five dual-port RAMs with a width of 8n bits, each of which is used to store a row of data in the input feature map. At the same time, the input buffer contains an instruction transaction queue adapted for WACT and RACT instructions.

4. The device according to claim 3, characterized in that The sliding window buffer is composed of n 8n bit register groups, and the data of each register group corresponds to a row of PE in the systolic array. When receiving data sent by the input buffer, a currently idle register group is selected for storage, and then 8 bits of data are sent to the systolic array each time by shifting.

5. The device according to claim 4, characterized in that The instruction memory is connected to the scheduler to send the instruction program to the scheduler for execution; the scheduler is connected to the register group to update the status register value therein; the scheduler is also connected to the arbiter, input cache and weight cache to control their working status under different instructions; The register group is connected to the input buffer, the sliding window buffer, the weight buffer and the systolic array to send the state register value to the above module; the arbiter is also connected to the input buffer and the weight buffer, and sends the input data to the input buffer or the weight buffer according to different instructions under the control of the scheduler; the input buffer is connected to the sliding window buffer to send the data required for each sliding window to the latter; The sliding window buffer is connected to the systolic array to transmit input data in a row direction; the weight buffer is connected to the systolic array to transmit weight data in a column direction.

6. An acceleration method applied to the configurable convolution operation acceleration device based on a systolic array according to any one of claims 1 to 5, characterized in that: The method comprises: Step 1, before starting the operation, write the instruction program into the instruction memory through the instruction address and instruction data, the instruction program includes CONST instruction, WWGT instruction, WACT instruction, RACT instruction, ADD instruction, SUB instruction, MUL instruction, DIV instruction, CMP instruction, JMP instruction and DONE instruction; Step 2: Send a start signal to the scheduler. After receiving the start signal, the scheduler starts to sequentially read the instruction program in the instruction memory and execute it. When the CONST instruction is executed, the register group is configured to configure the input feature map size FSize, the number of input channels Chin, and the number of output channels Chout in the current convolution mode. Step 3: When the scheduler executes the WWGT instruction, the control arbiter sends the input data to the weight cache until the weight data with a write length of WLen is successfully written; Step 4: When the scheduler executes the WACT instruction, the arbitrator is controlled to send the output data to the input buffer, and the input buffer caches the data; when the scheduler executes the RACT instruction, the input buffer is controlled to read the data into the sliding window buffer; Step 5, when the scheduler executes the ADD, SUB, and MUL instructions, it adds / subtracts / multiplies the register values ​​at the corresponding addresses according to the instruction operand addresses and stores them in the registers at the corresponding addresses of the instruction operand addresses; when it executes the DIV instruction, it divides the register values ​​at the corresponding addresses by rounding up or down according to the instruction operand addresses and stores them in the registers at the corresponding addresses of the instruction operand addresses; when it executes the CMP instruction, it compares the register values ​​at the instruction operand addresses and stores them in the comparison result register; when it executes the JMP instruction, it compares the jump condition value Jug and the value of the comparison result register. If the two are equal, the value of the instruction counter in the scheduler is updated to an immediate value; Step 6: When the scheduler executes the DONE instruction, it sends a completion signal, indicating that the current convolutional layer calculation is completed.

7. The acceleration method according to claim 6, characterized in that: The WWGT instruction is a write weight instruction, which controls the weight cache to receive weight data of immediate length; the WACT instruction is a write excitation instruction, which controls the input cache to cache the corresponding number of input feature map excitation data to the corresponding address of the corresponding SRAM inside it according to the SRAM width address, Bank address, starting address, write length and immediate number Zero obtained by instruction decoding; the RACT instruction is a read excitation instruction, which controls the input cache to read the corresponding number of input feature map excitation data from the corresponding address of the corresponding SRAM according to the read address interval, SRAM width address, Bank address, starting address and read length obtained by instruction decoding and send it to the sliding window cache.

Images