Processor and method for controlling processor

Abstract

To provide a processor having a systolic array capable of efficiently executing an operation.SOLUTION: Determining, by a controller that controls a plurality of processing cores, a processing core to be used for an operation for each layer of a neural network, wherein one or more processing cores respectively used for operations of a first layer and a second layer included in the neural network are at least partially different, each of the plurality of processing cores includes one or more memories, the one or more memories used for the operation of the first layer and the one or more memories used for the operation of the second layer are at least partially different, and each of the plurality of processing cores includes at least one output memory and at least one operation unit; The at least one output memory receives at least one operation result of the at least one operation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a processor and a method for controlling a processor. [Background technology]

[0002] In recent years, deep learning using deep neural networks (DNNs) has enabled highly accurate image recognition, speech recognition, and other tasks. Furthermore, processors specialized for deep learning convolution processing have been proposed. This type of processor has a systolic array containing multiple arithmetic cells arranged in a matrix. Input data is supplied along the rows of the systolic array, weight data is supplied along the columns of the systolic array, and the multiple input data and multiple weight data are sequentially multiplied and integrated to perform convolution processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 9,747,546 [Patent Document 2] U.S. Patent No. 9,952,831 [Patent Document 3] U.S. Patent No. 9,940,573 [Patent Document 4] U.S. Patent No. 9,842,293 [Patent Document 5] U.S. Patent No. 9,805,303 [Patent Document 6] U.S. Patent No. 9,721,203 [Patent Document 7] U.S. Patent No. 9,691,019 [Patent Document 8] US Patent Application Publication No. 2018 / 0165577 [Patent Document 9] U.S. Patent Application Publication No. 2017 / 0103315 Summary of the Invention [Problem to be solved by the invention]

[0004] When the size of a systolic array is fixed, the size of the systolic array may be insufficient if the amount of data to be processed in deep learning is large. In this case, the data to be processed must be loaded multiple times and the convolution process must be performed multiple times, resulting in reduced processing efficiency. Furthermore, when performing convolution processes while sequentially outputting input data from a buffer in which the input data is stored, for example, if the stride value of the input data is changed, the input data must be re-stored in the buffer to match the stride value.

[0005] The embodiments of the present invention have been made in view of the above points, and have an object to provide a processor that can efficiently execute operations. [Means for solving the problem]

[0006] In order to achieve the above object, a processor according to an embodiment of the present invention comprises a plurality of processing cores and a controller for controlling the plurality of processing cores, wherein the controller determines, for each layer of a neural network having a plurality of layers, a processing core from the plurality of processing cores to be used for calculations, wherein one or more processing cores used for calculations of a first layer included in the neural network are at least partially different from one or more processing cores used for calculations of a second layer included in the neural network that is different from the first layer, each of the plurality of processing cores has one or more memories, wherein the one or more memories used for calculations of the first layer are at least partially different from one or more memories used for calculations of the second layer, each of the plurality of processing cores includes at least one output memory and at least one calculation unit, and the at least one output memory receives at least one calculation result from the at least one calculation unit. [Effects of the Invention]

[0007] It is possible to provide a processor that can efficiently execute calculations. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram illustrating an example of a processor according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram illustrating an example of a processing core of FIG. 1. [Figure 3] FIG. 1 is a block diagram showing an example of the configuration of a systolic array. [Figure 4] FIG. 3 is a circuit diagram illustrating an example of a processing element of FIG. 2. [Figure 5] FIG. 3 is a block diagram showing examples of various memories in FIG. 2. [Figure 6] 2 is an operational flow diagram when the processor of FIG. 1 executes convolution processing. FIG. [Figure 7] 1. FIG. 4 is another operational flowchart for when the processor of FIG. 1 executes convolution processing. [Figure 8] 1. FIG. 6 is yet another operational flow diagram when the processor of FIG. 1 executes convolution processing. [Figure 9] FIG. 9 is an explanatory diagram showing the flow of data transfer explained in FIGS. 6 to 8. [Figure 10] FIG. 10 is an explanatory diagram showing a continuation of FIG. 9. [Figure 11] FIG. 11 is an explanatory diagram showing a continuation of FIG. [Figure 12] FIG. 12 is an explanatory diagram showing a continuation of FIG. [Figure 13] FIG. 13 is an explanatory diagram showing a continuation of FIG. 12. [Figure 14] FIG. 14 is an explanatory diagram showing a continuation of FIG. 13. [Figure 15] FIG. 15 is an explanatory diagram showing a continuation of FIG. 14. [Figure 16] FIG. 16 is an explanatory diagram showing a continuation of FIG. [Figure 17] FIG. 10 is an explanatory diagram illustrating an example of convolution processing. [Figure 18] 18 is an explanatory diagram showing an example of executing the convolution processing shown in FIG. 17 by a systolic array reconfigured in the processor of FIG. 1. FIG. [Figure 19] FIG. 20 is an explanatory diagram showing an example of convolution processing executed by PE0 to PE8 shown in FIG. 18; [Figure 20] FIG. 20 is an explanatory diagram showing a continuation of FIG. 19. [Figure 21] FIG. 18 is an explanatory diagram (comparison example) showing an example in which the convolution processing shown in FIG. 17 is executed by a systolic array of another processor. [Figure 22] 1. FIG. 4 is an explanatory diagram showing another example of the convolution process performed by the processor of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] 1 is a block diagram showing an example of a processor according to an embodiment of the present invention. The processor 100 includes a plurality of processing cores 200 (TPC: Tensor Processing Core) interconnected via a network-on-chip NOC (hereinafter referred to as NOC), and a prefetch engine 300 that controls the operation of the processor 100. The plurality of processing cores 200 are arranged, for example, in a matrix, and each processing core 200 is connected to a corresponding one of a plurality of routers RT provided in the NOC. Note that the arrangement of the processing cores 200 is not limited to a matrix. For example, the arrangement direction of the plurality of processing cores 200 may be not only vertical and horizontal, but also diagonal (e.g., staggered).

[0011] In the example of FIG. 1, the processor 100 has nine processing cores 200 arranged in three rows and three columns, three each in the vertical and horizontal directions. However, the processor 100 may have processing cores 200 arranged in one row and n columns, or n rows and one column (n is an integer greater than or equal to 2). That is, the number of processing cores 200 may be two or more. For example, the processor 100 may have 256 processing cores 200 arranged in sixteen columns and sixteen in the horizontal directions. It is preferable that the number of processing cores 200 in the vertical and horizontal directions be the same, but the number of processing cores 200 in the vertical and horizontal directions may be different. Hereinafter, the processing core 200 is also referred to as a TPC 200.

[0012] Adjacent TPCs 200 are connected via dedicated transfer paths 110 (broken arrows) that are different from the NOC. The transfer paths 110 enable the transfer of information such as data (input data, weight data) and commands in directions to the right and downwards, starting from the TPC 200 in the upper left of Figure 1, for example.

[0013] The prefetch engine 300 has a function equivalent to, for example, an L2 cache, and includes a controller 310 that controls the transfer of information to the TPC 200 and the transfer of information between the TPCs 200. The controller 310 controls routing by a router RT included in the NOC to transfer information from a higher-level system to a predetermined TPC 200, receive information from the TPC 200, or transfer information between the TPCs 200.

[0014] Information such as input data, weight data, and commands output from the prefetch engine 300 to the TPC 200 is transferred to the TPC 200 via the routers RT arranged horizontally in Fig. 1, and is not transferred between the routers RT arranged vertically in Fig. 1. Similarly, information such as the convolution results (feature maps) output from the TPC 200 to the prefetch engine 300 is transferred to the prefetch engine 300 via the routers RT arranged horizontally in Fig. 1, and is not transferred between the routers RT arranged vertically in Fig. 1. This makes it possible to prevent information from crossing over within the router RT, and improves the efficiency of information transfer between the prefetch engine 300 and the TPC 200.

[0015] Fig. 2 is a block diagram showing an example of the processing core (TPC) 200 of Fig. 1. The TPC 200 includes a plurality of processing elements 400 (PEs) arranged in a matrix, an address generator 21, an instruction memory 22, a plurality of data memories 23, a data manager 24, a plurality of weight memories 25, and an output memory 26. The TPC 200 also includes multiplexers 27 (271, 272, 273, 274, 275), a reconfiguration controller 28 (281, 282, 283), and a plurality of latches LT.

[0016] One of the data memory 23 and the weight memory 25 is an example of a first memory capable of holding first data, which is either the input data DT or the weight W. The other of the data memory 23 and the weight memory 25 is an example of a second memory capable of holding second data, which is the other of the input data DT or the weight W. Hereinafter, the processing element 400 will also be referred to as a PE 400. Furthermore, although the following description will be given of the case where convolution processing is executed, processing (computation) other than convolution processing may also be executed.

[0017] 1 controls the router RT to control the transfer of input data DT to the data memory 23, the transfer of weight data W to the weight memory 25, and the transfer of instructions INST to the instruction memory 22. The controller 310 also controls the operations of the multiplexers 271-275. Hereinafter, the input data DT will also be referred to as data DT, and the weight data W will also be referred to as weight W. The input data DT and weight data W will also be referred to as data.

[0018] For example, the TPC 200 has four PEs 400, two arranged vertically and two arranged horizontally, but the number of PEs 400 is not limited to four. The TPC 200 may have three or more PEs 400 arranged vertically and three or more PEs 400 arranged horizontally. However, as will be described later, in this embodiment, a systolic array can be constructed by connecting a predetermined number of PEs 400 between multiple TPCs 200, so the TPC 200 only needs to have a minimum number of PEs 400 (i.e., four). The PEs 400 function as operation units, and a PE array, which is an example of an operation unit array, is constructed by the multiple PEs 400 included in each TPC 200.

[0019] The address generation unit 21 generates an address to be output to the data memory 23 and an address to be output to the weight memory 25 based on address generation information supplied from the controller 310 via the data management unit 24. The address generation unit 21 functions as a first address generator that generates an address to be output to the data memory 23 and a second address generator that generates an address to be output to the weight memory 25.

[0020] The address generation unit 21 generates addresses based on the address generation information and outputs the generated addresses to, for example, the data memory 23, so that the data DT can be output from the data memory 23 in an order according to the addresses. Therefore, the data DT held in the data memory 23 can be output in any order, and the stride value, etc. can be changed without re-storing the data DT in the data memory 23. The weights W can also be output from the weight memory 25 in any order according to the addresses. As a result, the load cycles of the data DT and the weights W can be reduced, and the efficiency of the convolution processing by the processor 100 can be improved.

[0021] The instruction memory 22 can hold an instruction INST supplied from the prefetch engine 300 via the router RT and the data management unit 24. In accordance with the execution of an operation by the PE 400, the instruction memory 22 outputs the instruction INST, which includes operation specifications such as the type of operation, to the PE 400 that executes the operation.

[0022] The data memory 23 has a plurality of storage areas capable of holding data DT used in convolution processing. The data memory 23 holds the data DT supplied from the prefetch engine 300 via the router RT and the data management unit 24, and outputs the held data DT according to an address from the address generation unit 21. Note that a data memory 23 is provided for each pair of PEs 400 arranged horizontally in FIG. 2. This makes it possible to supply different addresses to each data memory 23, and to cause each data memory 23 to output data DT according to the address.

[0023] The weight memory 25 has multiple storage areas capable of holding weights W used in convolution processing. The weight memory 25 holds weights W supplied from the prefetch engine 300 via the router RT and the data management unit 24, and outputs the held weights W according to addresses from the address generation unit 21. Note that a weight memory 25 is provided for each pair of PEs 400 aligned vertically in FIG. 2. This makes it possible to supply different addresses to each weight memory 25, and to cause each weight memory 25 to output a weight W according to the address.

[0024] The output memory 26 has multiple storage areas for storing calculation results such as feature maps by the PEs 400. The output memory 26 is an example of a result memory for storing calculation results by multiple PEs 400. The output memory 26 outputs the calculation results received from the PEs 400 of its own TPC 200 to the prefetch engine 300 via the data management unit 24. By providing an output memory 26 for each TPC 200, it is possible to eliminate the need to transfer calculation results between TPCs 200, thereby reducing the number of transfer cycles for calculation results. As a result, it is possible to improve the efficiency of convolution processing by the processor 100.

[0025] The instruction memory 22, the data memory 23, the weight memory 25, and the reconfiguration control unit 28 are connected to the data management unit 24 via a transfer bus TB. An example of the internal configuration of the instruction memory 22, the data memory 23, the weight memory 25, and the output memory 26 will be described with reference to FIG.

[0026] The data management unit 24 manages the transfer of information such as data between the prefetch engine 300 and the TPC 200. For example, the data management unit 24 outputs address information transferred from the prefetch engine 300 to the address generation unit 21, and outputs the instruction INST transferred from the prefetch engine 300 to the instruction memory 22. The data management unit 24 outputs data DT to be used for convolution transferred from the prefetch engine 300 to the data memory 23, and outputs weights W to be used for convolution transferred from the prefetch engine 300 to the weight memory 25. The data management unit 24 outputs selection control information for the multiplexer 27 transferred from the prefetch engine 300 to the reconfiguration control unit 28.

[0027] The data management unit 24 also transfers the calculation results such as feature maps output from the output memory 26 to the prefetch engine 300. The data management unit 24 transfers the data DT held in the data memory 23 to the data memory 23 of another TPC 200 via the router RT, and transfers the weights W held in the weight memory 25 to the weight memory 25 of another TPC 200 via the router RT.

[0028] For example, the data management unit 24 controls the transfer of data DT between the data memory 23 and the data memory 23 of another TPC 200 during convolution processing based on instructions from the prefetch engine 300. The data management unit 24 outputs identification information ID, which indicates the transfer destination of the information, to the transfer bus TB along with the information. When the identification information ID transferred to the transfer bus TB indicates the address generation unit 21, the instruction memory 22, the data memory 23, the weight memory 25, and the reconfiguration control unit 28, each receives the information transferred along with the identification information ID. In other words, the address generation unit 21, the instruction memory 22, the data memory 23, the weight memory 25, the output memory 26, and the reconfiguration control unit 28 are identified by the identification information ID transferred along with the data via the router RT.

[0029] Each reconfiguration control unit 28 (281, 282, 283) generates a selection signal for each multiplexer 27 (271, 272, 273, 274, 275) based on selection control information output from the controller 310 via the data management unit 24. Each multiplexer 27 selects one of a plurality of inputs in response to the selection signal from the reconfiguration control unit 28, and connects the selected input to the output.

[0030] The multiplexer 271 selects an instruction INST output from the instruction memory 22 of its own TPC 200, an instruction INST transferred from the instruction memory 22 of an upper (T) TPC 200, or an instruction INST transferred from the instruction memory 22 of a left (L) TPC 200. The instruction INST selected by the multiplexer 271 is output to the instruction input of the PE array of its own TPC 200, and transferred to the multiplexer 271 of the lower (B) TPC 200 via an instruction INST transfer path that includes the latch LT.

[0031] In the own TPC 200, the instruction INST supplied from the multiplexer 271 to the PE array is transferred to each PE 400 in the PE array. Also, the PE 400 in the upper right of Fig. 2 transfers the instruction INST received from the PE 400 on the left to the TPC 200 on the right (R). Note that the PE 400 in the lower right of Fig. 2 receives the instruction INST and executes various operations, but does not have the function of transferring the instruction INST to the TPC 200 on the right (R).

[0032] When the multiplexer 271 selects the output of the instruction memory 22 of its own TPC 200, the TPC 200 functions as a corner TPC 200 that serves as the starting point of the convolution process. In other words, the TPC 200 that serves as the starting point of the convolution process is located in the upper left of a systolic array constructed by the PEs 400 of a predetermined number of TPCs 200. On the other hand, when the multiplexer 271 selects an instruction INST transferred from the upper (T) TPC 200 or the left (L) TPC 200, the TPC 200 functions as an intermediate TPC 200 other than the starting point.

[0033] The multiplexers 272, 273 select either the data DT output from the data memory 23 of their own TPC 200 or the data DT transferred from the TPC 200 on the left side (L), and output the selected data DT to the data input of the PE array of their own TPC 200. In the PE array, the PE 400 connected to the outputs of the multiplexers 272, 273 functions as the left-most PE 400 that receives the data DT from the data memory 23, or as an intermediate PE 400 that receives the data DT from the TPC 200 on the left side (L). Here, the left-most PE 400 refers to the left-most PE 400 of a systolic array constructed by the PEs 400 of a predetermined number of TPCs 200, and the intermediate PE 400 refers to a PE 400 other than the left-most PE 400.

[0034] The multiplexers 274 and 275 select either the weight W output from the weight memory 25 of their own TPC 200 or the weight W output from the TPC 200 above (T), and output the selected weight W to the weight input of the PE array of their own TPC 200. In the PE array, the PE 400 connected to the output of the multiplexers 274 and 275 functions as the upper PE 400 that receives the weight W from the weight memory 25 of their own TPC 200, or as an intermediate PE 400 that receives the weight W from the TPC 200 above (T). Here, the upper PE 400 refers to the upper PE 400 of a systolic array constructed by the PEs 400 of a predetermined number of TPCs 200, and the intermediate PE 400 refers to a PE 400 other than the upper PE 400.

[0035] The multiplexers 272-275 enable the PEs 400 included in multiple TPCs 200 to be connected together, and the size of the systolic array used for convolution can be set arbitrarily. That is, the reconfiguration control units 281-283 and the multiplexers 271-275 enable the reconfiguration of a systolic array of an optimal size according to the amount of data DT used for convolution.

[0036] 3 is a block diagram showing an example of the configuration of a systolic array. In the example shown in FIG. 3, four TPCs 200 are used to reconfigure a systolic array including nine PEs 400, each enclosed by a bold frame. The processor 100 then uses the systolic array to perform, for example, deep learning (e.g., convolution processing) of multiple layers of a neural network. In FIG. 3, the instruction memory 22, data memory 23, weight memory 25, and output memory 26 are shown as instruction memory IM, data memory DM, weight memory WM, and output memory OM, respectively.

[0037] Shaded areas indicate elements used for convolution processing in the systolic array. The output of each data memory DM corresponding to the systolic array is connected in sequence to a predetermined number of PEs arranged horizontally in Figure 3. The output of each weight memory WM corresponding to the systolic array is connected in sequence to a predetermined number of PEs arranged vertically in Figure 3.

[0038] The instruction INST is stored only in the instruction memory IM of the top left TPC 200 in the systolic array. The data DT used for convolution is stored in the data memory DM of the TPC 200 on the left side in the systolic array. The weights W used for convolution are stored in the weight memory WM of the TPC 200 on the upper side in the systolic array. The feature map generated by the convolution process by each PE 400 is stored in the output memory OM of the TPC 200 to which each PE belongs.

[0039] When performing convolution processing using all 16 PEs 400 of the four TPCs 200, data is also stored in the data memory DM on the lower side of the TPC 200 in the lower left of Figure 3, and weights W are also stored in the weight memory WM on the right side of the TPC 200 in the upper right of Figure 3.

[0040] Fig. 4 is a circuit diagram showing an example of the processing element 400 of Fig. 2. The PE 400 has an instruction decoder 41, a register file 42, an ALU (Arithmetic and Logic Unit) 43, a MAC (Multiply and Accumulate) 44, multiplexers 45 (451, 452, 453), and multiple latches LT. That is, the PE 400 includes an arithmetic and logic unit (ALU) and a multiply-accumulate unit (MAC).

[0041] The instruction decoder 41 decodes the instruction INST and generates an output memory address OMAD, source register addresses SR1AD and SR2AD, a destination register address DRAD, and a multiplexer control signal MUXC. The multiplexer control signal MUXC is generated for each of the multiplexers 451, 452, and 453.

[0042] The output memory address OMAD indicates the storage destination of the output memory 26 that stores feature maps, etc. The source register addresses SR1AD and SR2AD indicate registers REG that store data such as weights W used in various calculations, and the destination register address DRAD indicates a register REG that stores the calculation results. The multiplexer control signal MUXC is output to the multiplexers 451, 452, and 453, respectively, and is used to switch between the multiplexers 451, 452, and 453.

[0043] The register file 42 has a plurality of registers REG, similar to a general-purpose processor such as a CPU (Central Processing Unit). The registers REG specified by an instruction may store, for example, weights W or data DT used in convolution, or may store the calculation result RSLT. The register file 42 may also have registers REG for fixed-point numbers and registers REG for floating-point numbers.

[0044] The ALU 43 can perform one or more of the following operations based on the instruction INST (decoded result) decoded by the instruction decoder 41: addition, subtraction, multiplication, left shift, right shift, minimum value calculation, maximum value calculation, logical AND, logical OR, and exclusive OR. Note that the types of operations that the ALU 43 can perform are not limited to those described above. For example, the ALU 43 can perform operations on 32-bit floating-point numbers. The MAC 44 performs multiply-and-accumulate operations on 16-bit or 32-bit floating-point numbers based on the instruction INST decoded by the instruction decoder 41. The number of bits that the MAC 44 can process is not limited to the above and may be 8 bits. Furthermore, multiple MACs 44 may be installed in each PE 400. In this way, the PE 400 has multiple types of arithmetic units and the register file 42, and therefore can perform operations similar to those of a CPU (Central Processing Unit).

[0045] In response to a multiplexer control signal MUXC, the multiplexer 451 outputs either the weight W held in the register REG or the weight W received from the upper side (T) to the ALU 43 or MAC 44. In response to a multiplexer control signal MUXC, the multiplexer 452 outputs either the data DT held in the register REG or the data DT received from the left side to the ALU 43 or MAC 44.

[0046] The multiplexer 453 outputs either the operation result RSLT output from the ALU 43 or MAC 44 to the register file 42 or the output memory 26 (FIG. 2) in response to the multiplexer control signal MUXC. The storage destination of the operation result RSLT in the output memory 26 is specified by the output memory address OMAD. The multiplexer 453 also stores the weight W bypassed from the multiplexer 451 in one of the registers REG of the register file 42 in response to the multiplexer control signal MUXC.

[0047] In PE400, the instruction INST and data DT can be transferred from the left side (L) to the right side (R) of Figure 4 via the latch LT, and the weight W can be transferred from the top side (T) to the bottom side (B) of Figure 4 via the latch LT.

[0048] The PE 400 shown in FIG. 4 does not have a multiplier, but has an ALU 43, a MAC 44, and a register file 42. Therefore, for example, the PE 400 can execute not only multiplication in convolution but also various arithmetic and logical operations. This makes it possible to use the PE 400 not only for operations in the convolution layer but also for operations in the pooling layer, activation layer, fully connected layer, etc. in deep learning of a neural network. Furthermore, the PE 400 can execute a convolution process in which the results of multiplying data DT and weight W are added together.

[0049] Furthermore, the PE 400 can execute other operations using the operation results stored in the register REG. Furthermore, since the weights W supplied from the outside can be bypassed to the register REG, the weights W stored in the register REG can be repeatedly used to execute convolution. As a result, for example, it is possible to reduce the number of times data DT is transferred to the data memory 23 and the number of times weights W are transferred to the weight memory 25 compared to the conventional method, thereby enabling efficient execution of convolution processing using a systolic array.

[0050] Fig. 5 is a block diagram showing an example of various memories in Fig. 2. The memory 500 shown in Fig. 5 is any one of the instruction memory 22, the data memory 23, the weight memory 25, or the output memory 26.

[0051] The memory 500 has a multiplexer 51, a register 52, a write enable logic 53, a write counter logic 54, a read counter logic 55, and a memory array 56 (ARY). When the identification information ID received via the transfer bus TB indicates its own memory 500 and the data includes information indicating the setting of the register 52, the multiplexer 51 stores the address included in the data in a predetermined register 52. The register 52 has areas for holding a start address STAD(W) for writing, an end address ENDAD(W) for writing, a start address STAD(R) for reading, and an end address ENDAD(R) for reading. The storage location in the register 52 is identified by the register address REGAD included in the data.

[0052] The write enable logic 53 operates when the identification information ID indicates its own memory 500 and the data valid signal DTVLD indicates that the write data is valid. The write enable logic 53 outputs an enable signal EN until the write address WRAD output from the write counter logic 54 reaches the end address ENDAD(W). The write counter logic 54 has a multiplexer 541 and an adder 542. The multiplexer 541 selects the start address STAD(W) when starting to write data to the memory array ARY, and then selects the write address WRAD output by the adder 542. The adder 542 operates while receiving the enable signal EN, and increments the write address output from the multiplexer 541 by "1" to generate the write address WRAD.

[0053] The read counter logic 55 sequentially generates read addresses RDAD from the start address STAD(R) to the end address ENDAD(R) set in the register 52. While the read counter logic 55 is generating valid read addresses RDAD, it generates a read data valid signal RDTVLD. The read data valid signal RDTVLD indicates that the read data RDT from the memory array ARY is valid.

[0054] The memory array ARY writes the write data WDT received together with the write address WRAD to a memory cell (not shown), reads data from the memory cell indicated by the read address RDAD, and outputs the data as read data RDT. The write data WDT and read data RDT are input data DT, weight data W, calculation result RSLT, or instruction INST. While not particularly limited, the memory array ARY has a memory capacity of several kilowords. For example, one word includes 5-bit identification information ID and 64-bit data. The number of bits in the identification information ID may be increased or decreased depending on the number of elements that need to be identified. The data may be 32 bits or 128 bits.

[0055] Fig. 6 is an operational flow diagram when convolution processing is executed by the processor 100 of Fig. 1. That is, Fig. 6 shows an example of a control method for the processor 100. The operational flow shown in Fig. 6 is started, for example, based on a convolution processing instruction from a higher-level system that controls the operation of the processor 100. Note that when multiple convolution processing instructions are received from the higher-level system, the processor 100 may repeatedly execute the operational flow of Fig. 6.

[0056] By repeating steps S100, S102, S104, and S106, the address generation information is transferred to the address generation unit 21, the data DT is transferred to the data memory 23, the weight W is transferred to the weight memory 25, and the instruction INST is transferred to the instruction memory 22 in sequence. Hereinafter, the address generation information, the instruction INST, the data DT, and the weight W are also referred to as data. The data transfer in steps S100, S102, S104, and S106 is performed for each TPC 200 arranged vertically in FIG. 1 (every three in the example of FIG. 1). In other words, data can be transferred in parallel to the TPCs 200 arranged vertically in FIG. 1.

[0057] In step S100, the controller 310 of the processor 100 reconfigures the router RT of the NOC to set a data transfer path. If the TPC 200 to which data is transferred is the same, step S100 may be omitted from the second time onwards, and the previously set transfer path may be used. Note that regardless of the data transfer path setting, the controller 310 can output a control signal to any TPC 200.

[0058] Next, in step S102, the controller 310 sets the memory address of the data transfer destination to the instruction memory 22, data memory 23, or weight memory 25 to which the data is to be transferred. Here, the memory address is an address assigned to the instruction memory 22, data memory 23, or weight memory 25 to which the data is to be transferred, and is the write address STAD(W) or ENDAD(W) shown in FIG. 5. The controller 310 outputs the memory address and register address REGAD to the NOC together with identification information ID indicating the data transfer destination. For example, the data memory 23 that receives the addresses STAD(W) and ENDAD(W) stores the received addresses STAD(W) and ENDAD(W) in the area indicated by the register address REGAD in the register 52.

[0059] Next, in step S104, the controller 310 sequentially outputs data to be used in the convolution process (data DT, weight W, or instruction INST) together with identification information ID to the NOC. The data management unit 24 of the TPC 200 to which data is to be transferred outputs the data received via the NOC to the transfer bus TB. The data memory 23, which has received the data DT from the controller 310 via the transfer bus TB, operates as follows: Each time the data memory 23 receives one word of data DT, it changes the storage destination from the start address STAD(W) to the end address ENDAD(W), and sequentially stores the data DT in the memory array ARY. Furthermore, when transferring address generation information to the address generation unit 21, the controller 310 outputs the address generation information together with identification information ID.

[0060] Next, in step S106, the processor 100 determines whether all data necessary for the convolution processing has been transferred, and repeats the operations of steps S100 to S104 until all data necessary for the convolution processing has been transferred. If the transfer of all data necessary for the convolution processing has been completed, in step S108, the processor 100 outputs selection control information for configuration together with identification information ID to the reconfiguration control unit 28 that is the target of configuration. The selection control information is output for each TPC 200 arranged vertically in FIG. 1. The reconfiguration control unit 28 that has received the selection control information switches the multiplexer 27, thereby configuring a systolic array of a predetermined size.

[0061] Next, in step S110, the controller 310 outputs a trigger signal to the TPC 200, which includes the instruction memory 22 storing the instruction INST, to start the convolution process. In response to the trigger signal, instructions are sequentially output from the instruction memory 22, and addresses to the data memory 23 and the weight memory 25 are sequentially output by the address generator 21, thereby starting the convolution process. A feature map, which is the result of the convolution, is stored in the output memory 26. In step S112, the controller 310 waits for the convolution process to be completed.

[0062] When the convolution process is completed, in step S114, the controller 310 controls the router RT to set a transfer path for transferring the feature map obtained by the convolution process. Next, in step S116, the controller 310 sets the memory address of the feature map transfer source in the output memory 26 where the feature map is stored. Here, the memory address is the read address STAD(R) or ENDAD(R) shown in FIG. 5. Then, in step S118, the controller 310 issues a transfer request to the output memory 26, sequentially receives the feature map data output from the output memory 26 one word at a time, and ends the convolution process using the systolic array.

[0063] Fig. 7 is another operational flow diagram when convolution processing is performed by the processor 100 of Fig. 1. That is, Fig. 7 shows another example of a control method for the processor 100. In Fig. 7, data DT and weights W to be used in the next convolution processing are stored in an empty data memory 23 and an empty weight memory 25 that are not used for convolution processing in the systolic array. Then, during the convolution processing, the data DT and weights W to be used in the next convolution processing are transferred from the data memory 23 and weight memory 25 that store the data DT and weights W to be used in the next convolution processing to the data memory 23 and weight memory 25 that correspond to the systolic array.

[0064] For example, there may be cases where the amount of data DT and weights W used in the convolution process is so large that the data DT and weights W cannot fit into the data memory 23 at the left end of the systolic array or the weight memory 25 at the top end of the systolic array. In this case, the data DT and weights W are transferred from the prefetch engine 300 into the systolic array before the convolution process begins, using the available data memory 23 and available weight memory 25 within the systolic array. By transferring the data DT and weights W to the available data memory 23 and available weight memory 25 in advance, respectively, the transfer of the data DT and weights W from outside and the convolution process can be performed without dividing them, even when the amount of data is large. As a result, the efficiency of the convolution process can be improved.

[0065] If only the data DT cannot fit into the data memory 23 at the left end of the systolic array, only the data DT is transferred from an empty data memory 23, and the weight W is not transferred from an empty weight memory 25. Similarly, if only the weight W cannot fit into the weight memory 25 at the top end of the systolic array, only the weight W is transferred from an empty weight memory 25, and the data DT is not transferred from an empty data memory 23.

[0066] Steps S200, S202, S204, and S206 are similar to steps S100, S102, S104, and S106 of Figure 6, except that they include the transfer of data to free data memory 23 and free weight memory 25. Steps S208 and S210 are similar to steps S108 and S110 of Figure 6.

[0067] After the convolution process is started, in steps S212 to S218, the controller 310 transfers data DT between data memories 23 and weights W between weight memories 25. That is, the data DT and weights W to be used in the next convolution process are transferred from the data memories 23 and weight memories 25, in which they are stored, to the data memories 23 and weight memories 25 corresponding to the systolic arrays.

[0068] First, in step S212, the controller 310 controls the router RT to set a data transfer path between the data source TPC 200 and the data destination TPC 200. Next, in step S214, the controller 310 sets read addresses STAD(R) and ENDAD(R) in the data source TPC 200, and sets write addresses STAD(W) and ENDAD(W) in the data destination TPC 200. Here, the data source and data destination are the data memory 23 or the weight memory 25.

[0069] Next, in step S216, the controller 310 instructs the TPCs 200, which are the data transfer source and destination, to transfer data. The data management unit 24 of the TPCs 200 that received the instruction manages the data transfer, for example, from the data memory 23 of the data transfer source to the data memory 23 of the data transfer destination. As shown in FIG. 1, the processor 100 has an NOC that interconnects multiple TPCs 200, and is capable of transferring data DT from the data memory 23 of one TPC 200 to the data memory 23 of another TPC 200. Also, it is capable of transferring weights W from the weight memories 25 of one TPC 200 to the weight memories 25 of another TPC 200. Therefore, regardless of the size of the systolic arrays constructed in step S208, data DT can be transferred between the data memories 23 included in the systolic arrays, and weights W can be transferred between the weight memories 25 included in the systolic arrays.

[0070] Next, in step S218, the processor 100 determines whether all data required for the next convolution process (data DT and weight W) has been transferred, and repeats the operations of steps S212 to S216 until all data required for the next convolution process has been transferred. If the transfer of all data required for the next convolution process has been completed, in step S220, the processor 100 waits for the convolution process to be completed. Note that since the execution time of the convolution process is sufficiently longer than the data transfer time, it is possible to transfer data to be used for the next convolution process while the convolution process is being executed.

[0071] Steps S222, S224, and S226 are the same as steps S114, S116, and S118 in Fig. 6. For example, after step S226, the process may proceed to step S110 in Fig. 6 to execute the next convolution process, or may proceed to step S210 to execute the next convolution process. In the operation shown in Fig. 7, by transferring data to be used in the next convolution process to data memory 23 and weight memory 25 during the convolution process, the next convolution process can be started immediately after the completion of the convolution process, thereby improving the efficiency of the convolution process.

[0072] 8 is a flowchart showing yet another operation when convolution processing is performed by the processor 100 of FIG. 1. That is, FIG. 8 shows yet another example of a method for controlling the processor 100. In FIG. 8, after steps S100 to S110 of FIG. 6 are executed, in step S300 during the convolution processing, data to be used in the next convolution processing is transferred from the controller 310 to the free data memory 23 and the free weight memory 25 via the NOC. The operation of step S300 is the same as the operation of steps S100 to S106 of FIG. 6, except that the data transfer destinations are the free data memory 23 and the free weight memory 25 that do not correspond to the systolic array.

[0073] After step S300, the feature maps are transferred in the same manner as in steps S112 to S118 in Fig. 6. Then, after step S118, data is transferred between TPCs 200 in the same manner as in steps S212 to S218 in Fig. 7, and data to be used in the next convolution process is transferred to the data memory 23 and weight memory 25 corresponding to the systolic array. After the data to be used in the next convolution process has been transferred, the process returns to step S110 to start the next convolution process, and the operations of steps S110, S300, S112 to S118, and S212 to S218 are repeated a predetermined number of times.

[0074] In FIG. 8 as well, by transferring data to be used in the next convolution process to the data memory 23 and the weight memory 25 during the convolution process, the next convolution process can be started immediately after the completion of the previous convolution process, thereby improving the efficiency of the convolution process.

[0075] 9 to 16 are explanatory diagrams showing the flow of data transfer described in FIG. 7. That is, FIGS. 9 to 16 show an example of a control method for the processor 100. As in FIG. 3, FIGS. 9 to 16 show an example in which nine PEs 400 included in two vertical and two horizontal TPCs 200 are used as a systolic array. In the following, the upper left TPC 200 will be referred to as TPC(00), the upper right TPC 200 will be referred to as TPC(01), the lower left TPC 200 will be referred to as TPC(10), and the lower right TPC 200 will be referred to as TPC(11).

[0076] As in Figure 3, the instruction memory 22, data memory 23, weight memory 25, and output memory 26 are shown as instruction memory IM, data memory DM, weight memory WM, and output memory OM, respectively. PE 400 is also referred to as PE. Shaded areas indicate the instruction memory IM, data memory DM, weight memory WM, and output memory OM used for convolution processing, while diagonal lines indicate the data memory DM, weight memory WM, and output memory OM that hold data to be used for the next convolution processing. PEs shown in bold frames are included in a systolic array, and routers RT shown in bold frames are the targets of transfer path reconfiguration.

[0077] 9, the controller 310 reconfigures the router RT to set the data transfer path from the prefetch engine 300 to TPC(00) and (10). The controller 310 also sets write addresses STAD(W) and ENDAD(W) in the data memory DM of TPC(00) and (10). The controller 310 then transfers data DT to be used for convolution processing to TPC(00) and (10).

[0078] 10, the controller 310 sets write addresses STAD(W) and ENDAD(W) in the weight memory 25 of TPC(00) while maintaining the transfer path, and transfers the weight W to be used in the convolution process. The controller 310 also sets write addresses STAD(W) and ENDAD(W) in the weight memory 25 of TPC(10), and transfers the weight W to be used in the next convolution process.

[0079] Furthermore, the controller 310 transfers the instruction INST to be used in the convolution process to the instruction memory 22 of TPC(00). The instruction INST is transferred only to the instruction memory 22 corresponding to the upper left of the systolic array. In this embodiment, the instruction INST does not need to be stored in all instruction memories 22 included in the TPC 200 that constitute the systolic array, thereby improving the efficiency of transferring the instruction INST from the controller 310 to the instruction memory 22. Below, a description of the setting of the write addresses STAD(W) and ENDAD(W) will be omitted. Note that the weight W to be used in the next convolution process may be transferred to the weight memory 25 of a TPC 200 other than the TPC 200 that includes the PE that constitutes the systolic array.

[0080] 10, the controller 310 outputs selection control information to the TPCs (00) and (10), causing the reconfiguration control units 281, 282, and 283 to reconfigure the multiplexers 271, 272, 273, 274, and 275. As a result, in the TPCs (00) and (10), the PEs enclosed by bold frames form part of the systolic array used for convolution processing. That is, the PE enclosed by the bold frame in the TPC (00) receives data DT, weights W, and instructions INST from the data memory DM, weight memory WM, and instruction memory IM of its own TPC (00) to perform convolution processing. The PE enclosed by the bold frame in the TPC (10) receives data DT from the data memory DM of its own TPC (10), and weights W and instructions INST from the weight memory WM and instruction memory IM of the TPC (00), to perform convolution processing.

[0081] 11, the controller 310 reconfigures the router RT to set a data transfer path from the prefetch engine 300 to TPC(01) and (11). Then, the controller 310 transfers data DT to be used for the next convolution process to the data memory DM of TPC(01) and (11). Note that the data DT to be used for the next convolution process may be transferred to the data memory 23 of a TPC 200 other than the TPC 200 including the PE that constitutes the systolic array.

[0082] Next, in FIG. 12, the controller 310 transfers the weight W to be used for the convolution process to the weight memory 25 of TPC(01) while maintaining the transfer path, and transfers the weight W to be used for the next convolution process to the weight memory 25 of TPC(11). Also in FIG. 12, the controller 310 outputs selection control information to TPC(01) and (11), causing the reconfiguration control units 281, 282, and 283 to reconfigure the multiplexers 271, 272, 273, 274, and 275. As a result, in TPC(01) and (11), a part of the systolic array to be used for the convolution process is constructed by the PEs indicated by bold frames. Then, as in FIG. 3, a systolic array including nine PEs to be used for the convolution process is constructed.

[0083] The PE in the bold frame of TPC(01) receives data DT and instruction INST from the data memory DM and instruction memory IM of TPC(00), and receives weight W from its own weight memory WM of TPC(01), and performs convolution processing. The PE in the bold frame of TPC(11) receives data DT from the data memory DM of TPC(10), instruction INST from the instruction memory IM of TPC(00), and weight W from the weight memory WM of TPC(01), and performs convolution processing. After this, TPC(00), (01), (10), and (11) start convolution processing based on instructions from controller 310.

[0084] 13, during the convolution process, the controller 310 reconfigures the router RT to set the data transfer path to a path from the prefetch engine 300 to TPC(00) and (10). The controller 310 also sets write addresses STAD(W) and ENDAD(W) in the data memory DM of TPC(00) and (10). Next, although not shown, the controller 310 reconfigures the router RT to set the data transfer path to a path from the prefetch engine 300 to TPC(01) and (11). The controller 310 then sets read addresses STAD(R) and ENDAD(R) in the data memory DM of TPC(01) and (11). Note that feature map information obtained by the convolution process is sequentially stored in the output memory OM.

[0085] 14, the controller 310 changes the data transfer path to transfer the data DT held by TPC(01), (11) to be used in the next convolution process to TPC(00), (10). That is, the controller 310 reconfigures the router RT to set the data transfer path to a path from TPC(01), (11) to TPC(00), (10). The controller 310 then outputs a trigger signal to instruct TPC(01), (11) to transfer the data DT, and transfers the data DT (shaded) to be used in the next convolution process from TPC(01), (11) to TPC(00), (10).

[0086] As a result, as explained in FIG. 7, even when the amount of data used in the convolution process is large, the data DT or weight W that has been transferred to the TPC in advance can be transferred between TPCs during the convolution process, thereby improving the efficiency of the convolution process.

[0087] 15, similarly to Figures 13 and 14, the controller 310 sets write addresses STAD(W) and ENDAD(W) in the weight memories WM of TPC(00) and (01). Also, the controller 310 sets read addresses STAD(R) and ENDAD(R) in the weight memories WM of TPC(10) and (11).

[0088] Then, the controller 310 changes the data transfer path to transfer the weights W to be used in the next convolution process held by TPC(10) and (11) to TPC(00) and (01). That is, the controller 310 reconfigures the router RT to set the data transfer path to a path from TPC(10) and (11) to TPC(00) and (01). The controller 310 then outputs a trigger signal to instruct TPC(10) and (11) to transfer the weights W (shaded) to be used in the next convolution process from the weight memories WM of TPC(10) and (11) to the weight memories WM of TPC(00) and (01).

[0089] After the convolution process is completed, the controller 310 reconfigures the router RT to set the data transfer path to a path from the prefetch engine 300 to TPC(00), (10). Then, the controller 310 sets read addresses STAD(R) and ENDAD(R) in the output memory 26 of TPC(00), (10).

[0090] 16, in order to transfer the feature map, which is the result of the convolution, to the prefetch engine 300, the controller 310 reconfigures the router RT and sets the data transfer path to a path from TPC(00) and (10) to the prefetch engine 300. The controller 310 then outputs a trigger signal to TPC(00) and (10) to instruct them to transfer the feature map, and the feature map is transferred from TPC(00) and (10) to the prefetch engine 300.

[0091] After this, in order to transfer the feature maps obtained by TPC(01) and (11) to the prefetch engine 300, the router RT is reconfigured, and the feature maps are transferred from TPC(01) and (11) to the prefetch engine 300, completing the first convolution process. Then, the next convolution process is performed using the data DT and weights W transferred in Figures 14 and 15.

[0092] FIG. 17 is an explanatory diagram showing an example of convolution processing. In FIG. 17, convolution is performed using the input in the second row and second column of the input data DT (X0-X8) of three rows and three columns and three kernels of two rows and two columns (Y0-Y3, Y4-Y7, Y8-Y11). Then, nine outputs Z (Z0-Z8, Z9-Z17, or Z18-Z26) are output for each kernel. Note that the input data DT of three rows and three columns is padded to four rows and four columns for convolution. The sign "*" in the arithmetic expression for the output Z indicates multiplication.

[0093] Fig. 18 is an explanatory diagram showing an example of executing the convolution processing shown in Fig. 17 using a systolic array reconfigured in the processor 100 of Fig. 1. In Fig. 18, the convolution processing is executed using a systolic array including the nine PEs (PE0-PE8 in Fig. 18) shown in Fig. 3. Data X0-X8 are input data held in the data memory DM, and weights Y0-Y11 are weights held in the weight memory WM.

[0094] Data X0, X1, X3, X4, X1, X2, X4, and X5 are input sequentially to PE0-PE2. Data X3, X4, X6, X7, X4, X5, X7, and X8 are input sequentially to PE3-PE5. Data X6, X7, "0", "0", X7, X8, "0", and "0" are input sequentially to PE6-PE8. Weights Y0-Y3 are input sequentially to PE0, PE3, and PE6. Weights Y4-Y7 are input sequentially to PE1, PE4, and PE7. Weights Y8-Y11 are input sequentially to PE2, PE5, and PE8. Then, each of PE0-PE8 performs convolution using the input data X0-X8 and weights Y0-Y11 to generate outputs Z0-Z26 (feature maps).

[0095] 19 and 20 are explanatory diagrams showing an example of convolution processing executed by PE0-PE8 shown in FIG. 18. The MAC 44 (FIG. 4) of each PE (PE0-PE8) calculates the product of data X (X0-X8) and weight Y (Y0-Y11) for each cycle CYC (CYC1-CYC16) and performs convolution by accumulating the calculated products. In this case, the MAC 44 multiplies the data X supplied from the data memory DM or the data X held in the register REG by the weight Y supplied from the weight memory WM or the weight Y held in the register REG.

[0096] PE0 uses cycles CYC1-CYC12 to generate outputs Z0-Z2, PE1 uses cycles CYC2-CYC13 to generate outputs Z9-Z11, and PE2 uses cycles CYC3-CYC14 to generate outputs Z18-Z20.

[0097] PE3 uses cycles CYC2-CYC13 to generate outputs Z3-Z5, PE4 uses cycles CYC3-CYC14 to generate outputs Z12-Z14, and PE5 uses cycles CYC4-CYC15 to generate outputs Z21-Z23.

[0098] PE6 uses cycles CYC3-CYC14 to generate outputs Z6-Z8, PE7 uses cycles CYC4-CYC15 to generate outputs Z15-Z17, and PE8 uses cycles CYC5-CYC16 to generate outputs Z24-Z26.

[0099] Then, the convolution is performed in 16 cycles, and the outputs Z0 to Z26 are stored in the output memory 26 in 3 cycles. As a result, the convolution process shown in FIG. 17 is completed in 19 cycles.

[0100] In this embodiment, convolution is performed using not only the data X and weight Y output from the data memory DM and weight memory WM, but also the data X and weight Y held in the register REG. This allows convolution for three kernels to be performed in parallel without retransferring the data X and weight Y to the data memory DM and weight memory WM, thereby reducing the number of cycles required for the convolution process shown in FIG. 17.

[0101] Fig. 21 is an explanatory diagram (comparison example) showing an example of executing the convolution processing shown in Fig. 17 by a systolic array of another processor. Fig. 21 shows an example of convolution processing by one kernel (Y0-Y3) out of three kernels (Y0-Y3, Y4-Y7, Y8-Y11).

[0102] In FIG. 21, each set of nine inputs (e.g., X0, X1, X3, X4 or X1, X2, X4, X5) extracted from input data DT (X0-X8) is convolved with a kernel (Y0-Y3) using PE0-PE8 of the systolic array.

[0103] Each set of input data DT is loaded into the systolic array in three cycles, and then the weights W are transferred sequentially into the systolic array over five cycles to perform convolution. The products of each input data DT and each weight W are then added for each column of the systolic array to output the sums PSUM0-PSUM2, and the sum of the sums PSUM0-PSUM2 is output as the feature map Z.

[0104] In the convolution process in Figure 21, loading the input data requires 3 cycles, calculating the convolution requires 5 cycles, and outputting the feature map Z requires 1 cycle. As a result, convolution of one set of input data DT with one weight kernel requires 9 cycles, and convolution of each of the nine sets of input data with three weight kernels requires 243 cycles.

[0105] Fig. 22 is an explanatory diagram showing another example of convolution processing executed by the processor 100 of Fig. 1. In Fig. 22, 16 TPCs 200 (64 PEs 400) arranged in four columns and four columns are used to sequentially construct a systolic array including the 16 PEs 400 while shifting the PEs 400 in use. The hatched area indicates the data memory 23 in which data DT used for the convolution processing is stored and the weight memory 25 in which weights W used for the convolution processing are stored, and the black rectangle indicates the instruction memory 22 in which instructions INST for the convolution processing are stored.

[0106] For example, first, systolic array A, which includes 16 PEs 400 located at the upper left of FIG. 22, is reconfigured, and data DT is transferred to the data memory 23 on the left side of systolic array A, and weights W are transferred to the weight memory 25 on the upper side of systolic array A. Also, instructions INST are transferred to the instruction memory 22 on the upper left of systolic array A. Then, convolution processing is started by systolic array A.

[0107] During the convolution process, systolic array B is reconfigured by shifting PE 400 one position to the right and down relative to systolic array A currently performing the convolution, and data DT, weight W, and instruction INST are transferred to systolic array B. After the convolution process by systolic array A is completed, convolution process by systolic array B begins. During the convolution process by systolic array B, the feature map obtained by the convolution process by systolic array A is transferred, and systolic array C is reconfigured by shifting PE 400 one position to the right and down. Then, data DT, weight W, and instruction INST are transferred to systolic array C.

[0108] After this, while the systolic array is performing the convolution process, the feature map is transferred from the systolic array that has completed the convolution process, and the other systolic arrays are reconfigured, and the data DT, weights W, and instructions INST are transferred. By repeating the above, the transfer of the feature map obtained by the convolution process, the reconfiguration of the systolic array, and the transfer of the data DT, weights W, and instructions INST can be performed in the background of the convolution process. As a result, processing efficiency can be improved compared to when the reconfiguration of the systolic array, the data transfer, and the convolution process are performed separately. Furthermore, in the operation shown in FIG. 22, the transfer of the data DT and weights W to the systolic array is required only once, making it possible to eliminate the need to transfer the data DT and weights W within the systolic array.

[0109] As described above, in the embodiment of the present invention, the multiplexers 272-275 can connect the PEs 400 included in multiple TPCs 200, allowing the size of the systolic array used for convolution processing to be set arbitrarily. Because the size of the systolic array is variable, convolution processing of various data amounts can be performed without dividing the data. This makes it possible to apply, for example, convolution processing of new matrix sizes that are not supported. In other words, the reconfiguration control units 281-283 and the multiplexers 272-275 can reconfigure a systolic array of an optimal size according to the amount of data DT used for convolution processing.

[0110] Since a data memory 23 and a weight memory 25 are provided for each TPC 200, data DT to be used in the next convolution process can be transferred in advance to an empty data memory 23 that is not used for the convolution process. Also, weights W to be used in the next convolution process can be transferred in advance to an empty weight memory 25 that is not used for the convolution process. Because the empty data memory 23 and empty weight memory 25 are unrelated to the convolution process, data DT or weights W can be transferred during the convolution process.

[0111] Furthermore, during a convolution process, data to be used in the next convolution process can be transferred to an empty data memory 23 and an empty weight memory 25. Alternatively, during a convolution process, data DT can be transferred between data memories 23, and weights W can be transferred between weight memories 25. As a result, after the completion of a convolution process, the next convolution process can be started without transferring data DT and weights W from the controller 310, thereby improving processing efficiency. Furthermore, the number of times data DT and weights W are transferred to the systolic array can be reduced, thereby improving processing efficiency.

[0112] By having the address generator 21 generate addresses based on the address generation information, the data DT held in the data memory 23 can be output in any order, and the stride value, etc. can be changed without re-storing the data DT in the data memory 23. As a result, the number of load cycles of the data DT can be reduced, and the efficiency of the convolution process by the processor 100 can be improved.

[0113] By providing the PE 400 with multiple types of arithmetic units 43 and 44 and the register file 42, it is possible to execute not only the multiplication in convolution but also various arithmetic and logical operations similar to a CPU. Therefore, the processor 100 can be used not only for convolution but also for operations in various layers in deep learning. For example, following the convolution process, operations using a pooling layer (such as extracting the maximum value and averaging) can be executed within the PE 400. Furthermore, the PE 400 can execute a convolution process that accumulates the multiplication results.

[0114] The results stored in the register REG can be used to execute processing (calculation) other than convolution processing. Furthermore, because the weights W supplied from the outside can be bypassed to the register REG, the weights W held in the register REG can be repeatedly used to execute convolution. As a result, for example, it is possible to reduce the number of times data DT is transferred to the data memory 23 and the number of times weights W are transferred to the weight memory 25 compared to the conventional method, and convolution processing using a systolic array can be executed efficiently.

[0115] The instruction INST used for the convolution process is transferred only to the instruction memory 22 corresponding to the upper left of the systolic array, and the instruction INST held in one instruction memory 22 is sequentially transferred to the PE 400 that executes the convolution process via the multiplexer 271. Since it is not necessary to store the instruction INST in all the instruction memories 22 included in the TPC 200 that configures the systolic array, the efficiency of transferring the instruction INST from the controller 310 to the instruction memory 22 can be improved.

[0116] Since an output memory 26 is provided for each TPC 200, the feature map can be output directly to the NOC without passing through other TPCs 200, thereby improving the efficiency of transferring the feature map to the prefetch engine 300.

[0117] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications and variations are possible without departing from the scope of the claims. [Explanation of symbols]

[0118] 21 Address generation unit 22 Instruction Memory 23 Data Memory 24 Data Management Department 25 weight memory 26 Output Memory 27(271, 272, 273, 274, 275) Multiplexer 28(281, 282, 283) Reconfiguration control unit 41 Instruction Decoder 42 Register File 43 ALU 44 MAC44 45(451, 452, 453) Multiplexer 100 processors 110 Transfer Route 200 Processing Cores (TPC) 300 Prefetch Engine 310 Controller 400 Processing Elements (PE) DT data INST instruction RT Router W weight

Claims

1. Multiple processing cores and a controller that controls the plurality of processing cores; the controller determines, for each layer of a neural network having a plurality of layers, a processing core to be used for calculation among the plurality of processing cores; one or more processing cores used for the calculation of a first layer included in the neural network are at least partially different from one or more processing cores used for the calculation of a second layer included in the neural network that is different from the first layer; each of the plurality of processing cores has one or more memories; At least one memory used for the first layer calculation and at least one memory used for the second layer calculation are different from each other in at least part; each of the plurality of processing cores includes at least one output memory and at least one arithmetic unit; the at least one output memory receives at least one operation result of the at least one operation unit; Processor.

2. Multiple processing cores and a controller that controls the plurality of processing cores; the controller determines, for at least one layer of a neural network having a plurality of layers, a processing core to be used for calculation among the plurality of processing cores; each of the plurality of processing cores includes at least one output memory and at least one arithmetic unit; the at least one output memory receives at least one operation result of the at least one operation unit; Processor.

3. The processor according to claim 1 or 2, wherein each of the plurality of processing cores includes a plurality of arithmetic units.

4. The processor according to claim 1 , wherein each of the plurality of processing cores has a path for communication with the other processing cores.

5. The processor of claim 1 , wherein the layers of the neural network include at least a convolutional layer, a pooling layer, an activation layer, or a fully connected layer.

6. The processor according to claim 1 , wherein each of the plurality of processing cores includes a weight memory from which a weight is supplied to the processing core itself.

7. The processor according to claim 6 , wherein the weights are transferred to the weight memory in a processing core currently performing an operation among the plurality of processing cores.

8. The processor according to claim 1 , wherein the plurality of processing cores are arranged in a matrix of at least two rows and two or more columns.

9. The processor of claim 1 , wherein each of the plurality of processing cores includes an instruction memory.

10. A method for controlling a processor, comprising: the processor includes a plurality of processing cores and a controller that controls the plurality of processing cores; the controller determines, for each layer of a neural network having a plurality of layers, a processing core to be used for calculation among the plurality of processing cores; one or more processing cores used for the calculation of a first layer included in the neural network are at least partially different from one or more processing cores used for the calculation of a second layer included in the neural network that is different from the first layer; each of the plurality of processing cores has one or more memories; At least one memory used for the first layer calculation and at least one memory used for the second layer calculation are different from each other in at least part; each of the plurality of processing cores includes at least one output memory and at least one arithmetic unit; the at least one output memory receives at least one operation result of the at least one operation unit; How the processor is controlled.

11. A method for controlling a processor, comprising: the processor includes a plurality of processing cores and a controller that controls the plurality of processing cores; the controller determines, for at least one layer of a neural network having a plurality of layers, a processing core to be used for calculation among the plurality of processing cores; each of the plurality of processing cores includes at least one output memory and at least one arithmetic unit; the at least one output memory receives at least one operation result of the at least one operation unit; How the processor is controlled.

12. 12. The processor control method according to claim 10, wherein each of the plurality of processing cores includes a plurality of arithmetic units.

13. 13. The processor control method according to claim 10, wherein each of the plurality of processing cores has a path for communication with the other processing cores.

14. The processor control method according to claim 10 , wherein the layers of the neural network include at least a convolution layer, a pooling layer, an activation layer, or a fully connected layer.

15. 15. The processor control method according to claim 10, wherein each of the plurality of processing cores includes a weight memory from which a weight is supplied to the processing core itself.

16. The processor control method according to claim 15 , wherein the weights are transferred to the weight memory in a processing core currently performing an operation among the plurality of processing cores.

17. 17. The processor control method according to claim 10, wherein the plurality of processing cores are arranged in a matrix of at least two rows and two or more columns.

18. 18. The method of claim 10, wherein each of the plurality of processing cores includes an instruction memory.

Images