Neural network arithmetic apparatus

Abstract

To provide a neural network arithmetic apparatus which can suppress rapid power fluctuation.SOLUTION: A neural network arithmetic apparatus 10 has a plurality of processing elements (PE) 104 arranged in an array fashion, an activation memory 103 for storing input activation data supplied to the PE, and a main controller 101 for controlling an operation of each PE. Each of the PEs arranged in an array fashion has a function for transferring a command from the main controller between the adjacent PEs, and a clock line of a clock generator 11 is connected to each of logically grouped PEs. The main controller supplies a clock to PEs in each group in synchronization with supply to PE of data for use in neural network calculation, to increase the number of groups supplied with a clock as time passes from activation of the arithmetic processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an arithmetic device, and more particularly to a neural network arithmetic device that performs arithmetic operations using a plurality of processor elements. [Background technology]

[0002] AI accelerators typically use a PE array, which is a two-dimensional array of many PEs (Processing Elements), for parallel processing. If an AI accelerator with many of these PE arrays is implemented in a way that occupies a dominant area of the entire SOC, starting and stopping the AI accelerator can cause large instantaneous fluctuations in power consumption, resulting in malfunctions such as voltage drops.

[0003] Conventional techniques for solving this problem include (1) placing a large-capacity capacitor outside the chip that can withstand momentary fluctuations in power consumption, and (2) gradually turning power on and off throughout the SOC to maintain an operating state in which fluctuations do not occur during operation. Furthermore, a technique for controlling power fluctuations during processing is known, as described in Patent Document 1. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-119213 Summary of the Invention [Problem to be solved by the invention]

[0005] However, method (1) requires an external capacitor, which not only increases costs but also poses new challenges in reliability and durability, especially in harsh system requirements such as automotive applications.Meanwhile, method (2) requires a certain amount of time for startup and shutdown, which results in unnecessary power consumption during that time and can impair the power efficiency of the entire system.

[0006] The method described in Patent Document 1 is effective in controlling power fluctuations during processing, but does not provide a solution for gradually increasing or decreasing power consumption at startup.

[0007] In view of the above background, it is an object of the present invention to provide a neural network operation device that can suppress abrupt power fluctuations. [Means for solving the problem]

[0008] The present invention employs the following technical solutions to solve the above problems. The reference symbols in parentheses in the claims and this section are merely examples showing the correspondence with the specific solutions described in the embodiments below as one aspect, and do not limit the technical scope of the present invention.

[0009] The neural network calculation device of the present invention is a neural network calculation device (10) that performs neural network processing, and comprises a plurality of processor elements (104) arranged in an array, an activation memory (103) that stores input activation data to be supplied to the processor elements, and a main controller (101) that controls the operation of each of the processor elements, and the main controller (101) is configured to control the timing at which a clock supplied from a clock generator (11) is supplied to each processor element.

[0010] With this configuration, the main controller controls the timing of clock supply to the processor elements, thereby making effective use of power and suppressing sudden power fluctuations. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an LSI including a neural network calculation device according to a first embodiment. [Figure 2] FIG. 3 is a diagram showing a signal that instructs the timing of clock input in the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating power consumption of the neural network calculation device according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing a signal that instructs the timing of clock input in the second embodiment. [Figure 5] FIG. 10 is a diagram illustrating power consumption of the neural network calculation device according to the second embodiment. [Figure 6] FIG. 10 is a diagram illustrating an LSI including a neural network calculation device according to a third embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of the physical configuration of an LSI including a neural network calculation device according to a fourth embodiment. [Figure 8] FIG. 10 is a diagram showing the logical structure of an LSI of a neural network calculation device according to a fourth embodiment. [Figure 9] FIG. 10 is a diagram illustrating an LSI including a neural network calculation device according to a fifth embodiment. [Figure 10] FIG. 11 is a diagram illustrating processing by a neural network calculation device according to a fifth embodiment. [Figure 11] FIG. 13 is a diagram for explaining processing by a neural network calculation device according to a sixth embodiment. [Figure 12] FIG. 13 is a diagram illustrating a configuration of a neural network calculation device according to a seventh embodiment. [Figure 13] FIG. 13 is a diagram illustrating a configuration of a neural network calculation device according to a ninth embodiment. [Figure 14] FIG. 13 is a diagram for explaining processing by a neural network calculation device according to the ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] A neural network calculation device according to an embodiment will be described below with reference to the drawings. (First embodiment) FIG. 1 is a diagram showing an LSI including a neural network calculation device 10 according to the first embodiment.

[0013] The reference numeral 1 denotes the entire SOC. In many cases, the SOC 1 includes a CPU and a bus system (not shown) that control the entire system, and functions as a system. Reference numeral 10 denotes a neural network calculation device of this embodiment. Reference numeral 11 denotes a clock generator that supplies a clock to the entire SOC 1. This clock generator also supplies a clock to the neural network calculation device 10. Depending on the design, this clock may supply one type of frequency or may supply clocks of multiple types of frequencies. In this embodiment, for simplicity, a case in which one type of clock is supplied will be described, but the present invention can also be realized with multiple types of clocks.

[0014] Depending on its configuration, the neural network computing device 10 may occupy 50% or more of the overall size of the SOC 1. Accordingly, the power consumption of the neural network computing device 10 may also become a major part of the overall SOC 1.

[0015] Next, the internal configuration of the neural network calculation device 10 will be described. Reference numeral 101 denotes a main controller. The main controller 101 is a module that controls the processing sequence of the entire neural network calculation device 10, manages input and output data, and controls operation requests to processor elements (described later). In addition, the main controller 101 also controls power based on information on the operating state of the neural network calculation device 10.

[0016] Reference numeral 102 denotes a clock controller. The clock controller 102 supplies clocks to processor elements 104 (described later) and all other internal modules of the neural network calculation device 10. The clock controller 102 is equipped with a mechanism for individually supplying and stopping clocks to the multiple processor elements 104. In addition, the clock controller 102 may be provided with a mechanism for controlling the clock frequency for each individual processor element PEij.

[0017] Reference numeral 103 denotes an activation memory. The activation memory 103 is a module that stores activation data to be supplied to a processor element 104 (described later). The main controller 101 transfers the necessary activation data to the activation memory 103 in advance using a DMA controller (not shown) or the like, and instructs the processor element 104 to supply the data when processing starts. Based on this instruction, the activation memory 103 supplies the necessary activation data to the processor element 104.

[0018] The implementation method for this part may be such that the main controller 101 issues an instruction to the activation memory 103, which then actively supplies data to the processor element, or the main controller 101 issues an instruction to the processor element 104, which then reads the required data from the activation memory 103.

[0019] Reference numeral 104 denotes a processor element. The processor element 104 is equipped with one or more convolution operation units consisting of a multiplier and an adder that perform convolution operations using input activations and weight values from a weight input memory (not shown). In many cases, the processor element 104 is equipped with multiple convolution multipliers, allowing multiple convolution operations to be performed simultaneously in parallel.

[0020] The term processor element 104 is used to refer to the processor elements collectively, and when referring to an individual processor element, it is referred to as processor element "PEij," where ij is a number that identifies the processor element PE by its position, i (i=0,...,m) identifies the row, and j (j=0,...,n) identifies the column.

[0021] The output activation, which is the calculation output of the processor element 104, is either stored again in the activation memory 103 or output to the outside of the module.

[0022] These processor elements 104 are logically arranged in a two-dimensional array inside the neural network calculation device 10, and adjacent processor elements on the top, bottom, left, and right sides transfer activation data to each other during the convolution calculation process, thereby realizing convolution calculations of kernel sizes greater than or equal to 1.

[0023] For example, when PE11 in Figure 1 performs a convolution operation with a kernel size of 2x2, PE11 can obtain the activation data of the left neighbor of the target to be processed by obtaining the activation data input to PE10 from the activation memory 103. Similarly, PE11 also receives the data of the neighbor above from PE01 and the activation data once transmitted from PE00 to PE01, and uses them for the operation. Alternatively, activation data of the adjacent 2x2 region may be obtained from the activation memory 103.

[0024] Next, a clock processing flow of an LSI including the neural network calculation device 10 will be described. Fig. 2 is a diagram showing signals with which the main controller 101 instructs the clock controller 102 on the timing to input a clock. Reference numerals 201 to 204 in Fig. 2 are Enable signals output by the main controller 101 to the clock controller 102. Reference numerals 205 to 208 are clock signals output by the clock controller 102 to the processor element 104.

[0025] Of the processor elements 104 arranged two-dimensionally, one clock is supplied to the processor elements 104 in the same column. For example, 205 is a clock signal on a clock line connected to a processor element in the first column (PEx0 in FIG. 1). Similarly, 206 is a signal on a clock line connected to a processor element in the second column (PEx1 in FIG. 1). In this embodiment, a group is formed by the processor elements 104 in the same column.

[0026] The main controller 101 sequentially adjusts the timing of applying the clock for each column. More specifically, the clock for the next column is started with a delay of the number of cycles required for completing the multiply-and-accumulate operation of one element in the convolution operation process. Figure 2 is based on the assumption that the multiply-and-accumulate operation of one element is completed in one cycle, and therefore the clock is applied with a delay of one cycle for each column. This allows the next column to be started immediately after the activation is used in the processing of the adjacent cycle.

[0027] By controlling the clock in this way, the processor elements PEij are started one column at a time without causing data delays between adjacent processor elements PEij, which results in a more gradual increase in power consumption than if the clock supply were started simultaneously to all processor elements PEij, as shown in Figure 3.

[0028] (Second embodiment) The basic configuration of the neural network calculation device of the second embodiment is the same as that of the neural network calculation device of the first embodiment. In the second embodiment, the increase in power consumption is made more gradual than in the first embodiment.

[0029] 4 is a diagram showing signals instructing the timing of clock input in an LSI including a neural network calculation device according to the second embodiment, that is, a diagram showing signals used by the main controller 101 to instruct the clock controller 102 as to the timing of clock input.

[0030] 4 indicate enable signals that the main controller 101 outputs to the clock controller 102. Reference numerals 405 to 408 indicate clock signals that the clock controller 102 outputs to the processor elements 104.

[0031] At this time, one clock is supplied to the processor elements 104 in the same column among the two-dimensionally arranged processor elements 104. For example, 405 is the clock signal of the clock line connected to the processor element in the first column (PEx0 in FIG. 1). Similarly, 406 is the clock signal of the clock line connected to the processor element in the second column (PEx1 in FIG. 1).

[0032] At this time, the main controller 101 sequentially adjusts the timing of applying the clock for each column. More specifically, the clock for the next column is activated by shifting it by the number of cycles it takes to complete the multiplication and accumulation operation for one element during the convolution operation process. FIG. 4 is based on the assumption that the multiplication and accumulation operation for one element is completed in one cycle, and therefore the clock is applied by shifting it by one cycle for each column. This allows the next column to be activated immediately after the activation is used in the processing of the adjacent cycle. Furthermore, in this embodiment, the activation interval for each column is thinned out by the number of cycles of the number of columns in the array, and the thinning interval is gradually reduced.

[0033] Specifically, if there are four columns, the first clock is applied, followed by a four-cycle wait before applying the next clock. After that, two cycles are left before applying the clock, and then the normal clock is supplied. By controlling the clock in this way, only 1 / 4 of the processor elements are in operation while the first calculation process is being executed. After that, only 1 / 4 of the processors are in operation for two cycles, and then 1 / 2 of the processors are in operation in the next cycle. For the next two cycles, 3 / 4 of the processor elements are in operation, and from then on all processor elements are in operation.

[0034] By controlling in this manner, the processor elements are started up one row at a time without causing data delays between adjacent processor elements, and the overall processor element operating rate can be increased gradually. As a result, as shown in Figure 5, power consumption increases more slowly than with the clock supply method shown in the first embodiment.

[0035] (Third embodiment) FIG. 6 is a diagram showing an LSI including a neural network calculation device according to the third embodiment.

[0036] The reference numeral 6 denotes the entire SOC. In many cases, the SOC 6 includes a CPU and a bus system (not shown) that control the entire system, and functions as a system. 60 is a neural network calculation device of this embodiment. Reference numeral 61 denotes a clock generator that supplies a clock to the entire neural network calculation device 60. The clock generator 61 also supplies a clock to the neural network calculation device 60. Depending on the design, this clock may supply one type of frequency or may supply clocks of multiple types of frequencies. For simplicity, this embodiment will be described assuming that one type of clock is supplied, but the present invention can also be realized with multiple types of clocks.

[0037] Depending on its configuration, the neural network computing device 60 may occupy 50% or more of the overall size of the SOC 6. Accordingly, the power consumption of the neural network computing device 60 may also become a major part of the overall SOC.

[0038] Next, the internal configuration of the neural network calculation device 60 will be described. Reference numeral 601 denotes a main controller. The main controller 601 is a module that controls the processing sequence of the entire neural network calculation device 60, manages input and output data, and controls operation requests to processor elements (described later). In addition, the main controller 601 also controls power based on information about the operating state of the neural network calculation device 60.

[0039] Reference numeral 602 denotes a clock controller. The clock controller 602 supplies clocks to processor elements 604 (described later) and all other internal modules of the neural network calculation device 60. The clock controller 602 is equipped with a mechanism for individually supplying and stopping clocks to the multiple processor elements 604. In addition, the clock controller 602 may be provided with a mechanism for controlling the clock frequency for each individual processor element PEij.

[0040] Reference numeral 603 denotes an activation memory. The activation memory 603 is a module that stores activation data to be supplied to a processor element 604 (described later). The main controller 601 transfers the necessary activation data to the activation memory 603 in advance using a DMA controller (not shown) or the like, and instructs the processor element to supply the data when processing starts. Based on this instruction, the activation memory 603 supplies the necessary activation data to the processor element 604.

[0041] This part can be implemented in such a way that the main controller 601 issues an instruction to the activation memory 603, which then actively supplies data to the processor element, or the main controller 601 issues an instruction to the processor element 604, which then reads the necessary data from the activation memory 603.

[0042] Reference numeral 604 denotes a processor element. The processor element 604 is equipped with one or more convolution operation units consisting of a multiplier and an adder that perform convolution operations using input activations and weight values from a weight input memory (not shown). In many cases, a processor element is equipped with multiple convolution multipliers, allowing multiple convolution operations to be performed simultaneously in parallel.

[0043] The term processor element 604 is used to refer to the processor elements collectively, and when referring to an individual processor element, it is referred to as processor element "PEij," where ij is a number that identifies the processor element PE by its position, i (i=0,...,m) identifies the row, and j (j=0,...,n) identifies the column.

[0044] The output activation, which is the calculation output of the processor element 604, is either stored again in the activation memory 603 or output to the outside of the module.

[0045] These processor elements 604 are logically arranged in a two-dimensional array inside the neural network calculation device 60, and adjacent processor elements on the top, bottom, left, and right sides transfer activation data to each other during the convolution calculation process, thereby realizing convolution calculations of kernel sizes greater than or equal to 1.

[0046] For example, when PE11 in Figure 6 performs a convolution operation with a kernel size of 2x2, PE11 can obtain the activation data of the left neighbor of the target to be processed by obtaining the activation data input to PE10 from the activation memory 603. Similarly, PE11 also receives the data of the neighbor above from PE01 and the activation data once transmitted from PE00 to PE01, and uses them for the operation. Alternatively, activation data of the adjacent 2x2 region may be obtained from the activation memory 603.

[0047] Next, the clock processing flow of an LSI including the neural network calculation device 60 will be described. The timing of clock processing in the third embodiment is the same as in the first embodiment, so will be described using Figure 2. In the third embodiment, the same clock is supplied to different groups of processor elements 604. Figure 2 is a diagram showing signals that the main controller 601 uses to instruct the clock controller 602 on the timing to input the clock.

[0048] 2, 201 to 204 denote enable signals that the main controller 601 outputs to the clock controller 602. 205 to 208 denote clock signals that the clock controller 602 outputs to the processor element 604.

[0049] Of the two-dimensionally arranged processor elements 604, as shown in Figure 6, clock groups are formed radially from PE00 according to the vertical and horizontal separation distances, and one clock is supplied to each clock group. For example, 605 is a clock group connected to the processor element at the top left (PE00 in Figure 6). 606 is a clock group connected to clock groups formed by processor element groups adjacent to PE00 in both the vertical and horizontal directions. Figure 6 shows groups 605 to 609, but the number of groups is not limited to this.

[0050] The main controller 601 sequentially adjusts the timing of applying the clock for each clock group. More specifically, the clock for the next group is started with a delay of the number of cycles required to complete the multiply-and-accumulate operation for one element of the convolution operation process. Figure 2 is based on the assumption that the multiply-and-accumulate operation for one element is completed in one cycle, so the clock is applied with a delay of one cycle for each column.

[0051] This allows the next row and column to be started immediately after the activation has been used in the processing of the adjacent cycle. By controlling the clock in this way, the processor elements are started sequentially without delaying data between adjacent processor elements PEij, which results in a more gradual increase in power consumption than if the supply of clocks to all processor elements was started simultaneously.

[0052] (Fourth embodiment) Fig. 7 is a diagram showing an example of the physical configuration of an LSI including a neural network operation device according to the fourth embodiment. On the other hand, Fig. 8 is a diagram showing the corresponding logical structure of the LSI. Each internal module will be explained using Fig. 8.

[0053] 8 indicates the entire SOC. In many cases, the SOC 8 includes a CPU and a bus system (not shown) that control the entire system, and functions as a system. 80 is a neural network calculation device according to the fourth embodiment. Reference numeral 81 denotes a clock generator that supplies a clock to the entire SOC 8. This clock generator also supplies a clock to the neural network calculation device 80. Depending on the design, this clock may supply one type of frequency or may supply clocks of multiple types of frequencies. For simplicity, this embodiment will be described assuming that one type of clock is supplied, but the present invention can also be realized with multiple types of clocks.

[0054] Next, the internal configuration of the neural network calculation device 80 will be described. Reference numeral 801 denotes a main controller. The main controller 801 is a module that controls the processing sequence of the entire neural network calculation device 80, manages input and output data, and controls operation requests to processor elements (described later). In addition, the main controller 801 also controls power based on information on the operating state of the neural network calculation device 80.

[0055] Reference numeral 803 denotes an activation memory. The activation memory 803 is a module that stores activation data to be supplied to a processor element 804 (described later). The main controller 801 transfers the necessary activation data to the activation memory 803 in advance using a DMA controller (not shown) or the like, and instructs the processor element 804 to supply the data when processing starts. Based on this instruction, the activation memory 803 supplies the necessary activation data to the processor element 804. Figure 7 shows an example in which the activation memories 803 are arranged side by side on the left side of the layout.

[0056] Reference numeral 804 denotes a processor element. The processor element 804 is equipped with one or more convolution operation units consisting of a multiplier and an adder that perform convolution operations using input activations and weight values from a weight input memory (not shown). In many cases, the processor element 804 is equipped with multiple convolution multipliers, allowing multiple convolution operations to be performed simultaneously in parallel.

[0057] The term processor element 804 is used to refer to the processor elements collectively, and when referring to an individual processor element, it is referred to as processor element "PEij," where ij is a number that identifies the processor element PE by its position, i (i=0,...,m) identifies the row, and j (j=0,...,n) identifies the column.

[0058] The output activation, which is the calculation output, is either stored again in the activation memory 803 or output to the outside of the module.

[0059] These processor elements 804 are logically arranged in a two-dimensional array inside the neural network calculation device 80, and adjacent processor elements on the top, bottom, left, and right sides transfer activation data to each other during the convolution calculation process, thereby realizing convolution calculations of kernel sizes greater than or equal to 1.

[0060] For example, when PE11 in Figure 8 performs a convolution operation with a kernel size of 2x2, PE11 can obtain the activation data of the left neighbor of the processing target by obtaining the activation data input to PE10 from the activation memory 803. Similarly, PE11 also receives the data of the neighbor above from PE01 and the activation data once transmitted from PE00 to PE01, and uses them for the operation. Alternatively, activation data of the adjacent 2x2 region may be obtained from the activation memory 803.

[0061] At this time, when the placement and wiring of the two-dimensionally arranged processor elements 804 is performed as shown in FIG. 7, a clock group is formed for adjacent PEs, and a clock is supplied to each clock group.

[0062] 805 to 807 in Figure 8 logically represent clock groups. For example, 805 is the clock group connected to the processor element at the top left. These constitute a single clock group because they are in close proximity in the physical placement and routing of Figure 7. 806 is the clock group made up of processor elements 804 from the second column onwards in the first row. In the example of Figure 7, PE1x is arranged in a strip in the placement and routing, and in this case the clock groups are separated.

[0063] As described above, clock groups are set in accordance with the placement and wiring state shown in Fig. 7. At this time, it would be sufficient to connect one clock line to the clock group and configure a physically fixed clock group, but because placement and wiring is often determined in a later stage of the design process and may change depending on the process, it is also possible to implement an implementation that achieves the same effect by connecting clock lines independently to each processor element and adjusting the clock supply timing by clock controller 802 as if they were logically the same group.

[0064] Next, the clock processing flow of the LSI including the neural network calculation device 80 will be explained. The timing of clock processing in the fourth embodiment is the same as in the first embodiment, so it will be explained using Fig. 2. Fig. 2 is a diagram showing signals that the main controller 601 sends to the clock controller 602 to instruct the timing of inputting the clock.

[0065] The main controller 801 sequentially adjusts the timing of applying the clock for each group. More specifically, the clock for the next column is started with a delay of the number of cycles required to complete the multiply-and-accumulate operation for one element of the convolution operation. Figure 2 is based on the assumption that the multiply-and-accumulate operation for one element is completed in one cycle, so the clock is applied with a delay of one cycle for each column.

[0066] This allows the next row and column to be activated immediately after the activation has been used in the processing of the adjacent cycle. By controlling the clock in this way, the processor elements are activated sequentially without delaying the data between adjacent processor elements, resulting in a more gradual increase in power consumption than if the clock were supplied to all processor elements.

[0067] In this embodiment, an example in which the clock timing is the same as in the first embodiment has been described, but the clock timing may also be determined by a method in which the activation interval of each column is thinned out by the number of cycles of the number of columns in the array, and the thinning interval is gradually reduced (see FIG. 4), as in the second embodiment.

[0068] (Fifth embodiment) 9 is a diagram showing an LSI including a neural network calculation device according to a fifth embodiment. In the above-described embodiments, a configuration for delaying the timing for starting clock supply when calculation is started has been described, but in this embodiment, an example will be described in which fluctuations in power consumption of a neural network calculation device are suppressed by utilizing power consumption in a diagnostic circuit.

[0069] That is, the neural network calculation device of this embodiment is a neural network calculation device that performs neural network processing, and comprises a plurality of processor elements arranged in an array, an activation memory that stores input activation data to be supplied to the processor elements, and a main controller that controls the operation of each of the processor elements, wherein either or both of the processor elements and the main controller have a calculation check processing function, and the main controller is configured to instruct the processor elements to selectively execute calculation processing and calculation check processing. Hereinafter, a detailed description will be given with reference to the drawings.

[0070] The reference numeral 9 denotes the entire SOC. In many cases, the SOC 9 includes a CPU and a bus system (not shown) that control the entire system, and functions as a system. 90 is a neural network calculation device of this embodiment. Reference numeral 91 denotes a clock generator that supplies a clock to the entire SOC 9. The clock generator 91 also supplies a clock to the neural network calculation device 90. Depending on the design, this clock may supply one type of frequency or may supply clocks of multiple types of frequencies. For simplicity, this embodiment will be described assuming that one type of clock is supplied, but the present invention can also be realized with multiple types of clocks.

[0071] Depending on its configuration, the neural network computing device 90 may occupy 50% or more of the overall size of the SOC, and therefore the power consumption of the neural network computing device 90 may also be a major portion of the overall SOC 9.

[0072] Next, the internal configuration of the neural network calculation device 90 will be described. Reference numeral 901 denotes a main controller. The main controller 901 is a module that controls the processing sequence of the entire neural network calculation device 90, manages input and output data, and controls operation requests to processor elements (described later). In addition, the main controller 901 also controls power based on information on the operating state of the neural network calculation device 90.

[0073] Reference numeral 902 denotes a clock controller. The clock controller 902 supplies clocks to processor elements 904 (described later) and all other internal modules of the neural network calculation device 90. The clock controller 902 is equipped with a mechanism for individually supplying and stopping clocks to the multiple processor elements 904. In addition, the clock controller 902 may be provided with a mechanism for controlling the clock frequency for each individual processor element PEij.

[0074] 903 is an activation memory. The activation memory 903 is a module that stores activation data to be supplied to a processor element 904 (described later). The main controller 901 transfers the necessary activation data to this activation memory 903 in advance using a DMA controller (not shown) or the like, and instructs the processor element 904 to supply the data when processing starts. Based on this instruction, the activation memory 903 supplies the necessary activation data to the processor element 904.

[0075] The implementation method for this part may be such that the main controller 901 issues an instruction to the activation memory 903, which then actively supplies data to the processor element, or such that the main controller 901 issues an instruction to the processor element 904, which then reads the required data from the activation memory 903.

[0076] Reference numeral 904 denotes a processor element. The processor element 904 is equipped with one or more convolution operation units each consisting of a multiplier and an adder that executes a convolution operation using an input activation and a weight value from a weight input memory (not shown) as input. In many cases, a processor element is equipped with multiple convolution multipliers, allowing multiple convolution operations to be executed simultaneously in parallel.

[0077] The term processor element 904 is used to refer to the processor elements collectively, and when referring to an individual processor element, it is referred to as processor element "PEij," where ij is a number that identifies the processor element PE by its position, i (i=0,...,m) identifies the row, and j (j=0,...,n) identifies the column.

[0078] The output activation, which is the calculation output, is either stored again in the activation memory 903 or output to the outside of the module.

[0079] These processor elements 904 are logically arranged in a two-dimensional array inside the neural network calculation device 90, and adjacent processor elements on the top, bottom, left, and right sides transfer activation data to each other during the convolution calculation process, thereby realizing convolution calculations of kernel sizes greater than or equal to 1.

[0080] For example, when PE11 in Figure 9 performs a convolution operation with a kernel size of 2x2, PE11 can obtain the activation data of the left neighbor of the target to be processed by obtaining the activation data input to PE10 from the activation memory 903. Similarly, PE11 also receives the data of the neighbor above from PE01 and the activation data once transmitted from PE00 to PE01, and uses them for the operation. Alternatively, activation data of the adjacent 2x2 region may be obtained from the activation memory 903.

[0081] 905 is a diagnostic circuit. In FIG. 9, the diagnostic circuit is written as "BIST," which stands for "Build in soft test." The diagnostic circuit 905 generates or stores an expected value corresponding to an input pattern when performing diagnostic processing, and compares the calculation result of the processor element 904 for the input pattern with the expected value. One such circuit is provided for each processor element 904. By periodically executing this function, it becomes possible to detect hardware failures in the processor element 904.

[0082] The diagnostic circuits 905 are each connected to the main controller 901, and the main controller 901 can select whether to supply normal activation data to each processor element 904 and execute an operation, or to supply pattern data from the diagnostic circuits 905 and execute an operation.

[0083] In the figure, the diagnostic circuit 905 is arranged adjacent to each processor element, and when a pattern from the diagnostic circuit 905 is selected by a selection signal from the main controller 901, a diagnostic data pattern can be supplied to the corresponding processor element 904.

[0084] Alternatively, the diagnostic circuit 905 may be implemented inside the main controller 901, and configured to supply diagnostic patterns to each processor element 904. In this configuration, the diagnostic circuit 905 generates a diagnostic pattern when at least one processor element 904 executes diagnostic processing, and sends the diagnostic pattern to the target processor element 904 together with a selection signal generated by the main controller 901.

[0085] Compared to the case of calculating normal activation data, the diagnostic processing executed by the diagnostic circuit 905 does not require data transfer between processor elements 904, and the processing is completed for each processor element 904. In addition, the pattern used for the diagnosis only needs to achieve a predetermined toggle rate, and the power consumed by each processor element 904 can be controlled to some extent in advance of the toggle rate.

[0086] Next, we will explain the processing flow of clock and diagnostic processing using this configuration. In this processing, the main controller 901 instructs the clock controller 902 to input a clock. Of the processor elements 904 arranged two-dimensionally, one clock is supplied to the processor elements 904 in the same column.

[0087] With the above configuration, the main controller 901 enables the diagnostic circuit 905 and adjusts the timing of applying the clock for each column in turn. In this embodiment, the timing of applying the clock may be simultaneous, may be shifted by one cycle for each PE column, or may be shifted by more than one cycle. In this case, in the case of one cycle, a toggle pattern is set that is likely to make the power consumed by the diagnostic circuit smaller than the power consumed by the actual arithmetic processing.

[0088] In the case of multiple cycles, it is preferable that the diagnostic pattern generated by the diagnostic circuit 905 be a pattern in which the toggle rate gradually increases. Furthermore, if the processor element 904 is made up of multiple arithmetic units, it is not necessary to activate all of them in a way that can realize a series of convolution operation processes, and it is sufficient to confirm the validity of each arithmetic unit, so that each arithmetic unit may be activated individually.

[0089] Thereafter, the main controller 901 switches to actual product-sum operation processing in order, starting with the processor elements 904 that have completed processing of the diagnostic circuit 905. This allows the power consumption of the module to be gradually increased before the normal convolution operation processing, without relying on the data flow of the convolution operation and without wasting power consumption.

[0090] The above will be explained using Fig. 10 as an example. Fig. 10 is a diagram in which the horizontal axis represents time and the vertical axis represents operation of processor elements 904 in the column direction. In Fig. 10, for example, the number of columns of processor elements 904 (i.e., the number of groups) is 8. In this example, for simplicity, the main controller 901 supplies all clocks to the clock controller 902 simultaneously.

[0091] Based on the above assumptions, the main controller 901 first activates the diagnostic circuit in the first column. The toggle rate of the pattern of this diagnostic circuit is adjusted to, for example, around 25%. When the processing of the diagnostic circuit in the first column is completed, the main controller 902 activates the diagnostic circuit in the second column. The toggle rate of the pattern of this diagnostic circuit is adjusted to, for example, around 50%. Similarly, the diagnostic circuit in the third column with a toggle rate of 75% and the diagnostic circuit in the fourth column with a toggle rate of 100% are activated in sequence.

[0092] 10, the actual convolution calculation is then started sequentially for those whose diagnosis has been completed. As described above, data transfer occurs between processor elements 904 during this convolution calculation process, and therefore there is a start-up delay that matches the latency of the data transfer between processor elements. On the other hand, since the diagnosis process does not depend on this data transfer, the start-up timing can be set more freely.

[0093] In the example of Figure 10, the diagnostic process and the convolution operation process for columns 4 to 8 overlap. In this case, the toggle rate of the diagnostic pattern for columns 4 to 8 can be set to a lower value. Also, if the column patterns are always fixed, there is a possibility that the fault detection rate will differ between diagnostic circuits with low toggle rates and diagnostic circuits with high toggle rates. Therefore, the activation rate can be averaged by appropriately changing the order in which the diagnostic circuits are activated.

[0094] In this way, the neural network operation device 80 of this embodiment can suppress power fluctuations by controlling the clock supply to the processor element 904 and the diagnostic circuit 905 in combination.

[0095] (Sixth embodiment) The basic configuration of the neural network calculation device of the sixth embodiment is the same as that of the neural network calculation device 50 of the fifth embodiment. The neural network calculation device of the sixth embodiment sequentially increases power by changing the number of activated diagnostic circuits in the neural network calculation device 90 of the fifth embodiment.

[0096] 11 is a diagram for explaining the processing by the neural network calculation device of the sixth embodiment. In FIG. 11, the main controller 901 performs diagnostic processing on the processor elements 904 in the 0th and 4th columns using the diagnostic circuit 905. Therefore, the processor element column operating at this time is 2. At the same time as this processing is completed, the main controller 901 performs similar processing on the 2nd and 6th columns. Therefore, the processor element column operating at this time is also 2.

[0097] The main controller 901 then starts diagnostic processing for the remaining four processor element columns. The main controller 901 then executes convolution processing on all processor elements. This allows the operating rate of the processor elements to increase stepwise, making it possible to gradually increase power consumption. In this embodiment, the diagnostic circuit is started in the above pattern, but the pattern is not limited to the above, as long as the number of operating processor elements 904 increases gradually.

[0098] In the fifth and sixth embodiments described above, examples have been given in which clock signals are supplied to processor elements 904 by forming groups in units of columns, but it goes without saying that it is also possible to switch between rows and columns and form groups in units of rows to supply clock signals. Furthermore, these embodiments can also be implemented in combination with the clock supply delay control described in the first to fourth embodiments.

[0099] (Seventh embodiment) The basic configuration of the neural network calculation device of the seventh embodiment is the same as that of the neural network calculation device 90 of the fifth embodiment (see FIG. 9). In the seventh embodiment, the number of activations of the diagnostic circuit is changed according to the state of the neural network processing, thereby stabilizing the power consumption.

[0100] That is, the neural network calculation device of this embodiment is a neural network calculation device that performs neural network processing, and includes a plurality of processor elements arranged in an array, an activation memory (103) that stores input activation data to be supplied to the processor elements, and a main controller (101) that controls the operation of each of the processor elements, and the main controller is configured to detect the type of neural network processing and instruct some of the processor elements to perform calculation check processing based on power consumption that is predetermined according to the type of neural network processing. This will be described in detail below with reference to the drawings.

[0101] The main controller instructs the processor elements to perform arithmetic operations. At this time, depending on the neural network processing, some processes do not use all of the processor elements. For example, there is a process that simply connects two networks in the channel direction without performing a convolution operation. In such a process, there is a possibility that power consumption will drop sharply compared to when a convolution operation is performed, since the arithmetic unit is not operating.

[0102] As another example, in the case of an edge of an image, not all processor elements may be active, and only a portion of them may be active. This may also reduce power consumption.

[0103] In view of the above background, in this embodiment, the main controller instructs the processor elements or the arithmetic processing units of the processor elements to execute diagnostic processing in response to the processing of the neural network.

[0104] 12 is a diagram for explaining the processing of the neural network processing device of the seventh embodiment. In the neural network processing device of the seventh embodiment, as in the fifth embodiment, clocks are supplied to processor elements in groups of columns.

[0105] 12, in the first processing, only the processor elements in the even-numbered rows perform processing, while the processor elements in the odd-numbered rows perform only transfer. At the next processing timing, only the odd-numbered rows perform processing, while the even-numbered rows perform transfer processing, and at this timing the main controller can start diagnostic processing for the processor elements in the even-numbered rows. At this time, if there is a time difference between the actual calculation processing and the diagnostic processing, the diagnostic processing may be executed at any time.

[0106] The above-described processing can suppress sudden fluctuations in power consumption caused by the processing contents of the processor elements.

[0107] (Eighth embodiment) The basic configuration of the neural network calculation device of the eighth embodiment is the same as that of the neural network calculation device 50 of the fifth embodiment. In the above-described embodiment, the explanation has been given with a focus on suppressing power fluctuations that accompany the simultaneous start of clock supply when the calculation of the neural network calculation device is started, but power fluctuations can also occur when the clock supply is stopped.

[0108] In the neural network calculation device of the eighth embodiment, at the end of processing, the main controller stops the clocks to the processor elements in stages in the reverse order of Fig. 5. Also, at the end of processing, the main controller stops the clocks to the processor elements and the diagnostic circuit in stages in the reverse order of Fig. 10 or 11.

[0109] Furthermore, when the ongoing process is completed, the main controller determines whether the current process is the final layer of the neural network process, and transitions to gradual clock stopping only if the neural network process is the final layer.On the other hand, if the neural network process is not the final layer, the main controller may be configured to cancel the procedure for gradual clock stopping and continue to supply the clock.

[0110] This function allows layer processing to continue without performance degradation when layer processing continues, while avoiding sudden fluctuations in power consumption when layer processing ends.

[0111] (Ninth embodiment) 13 is a diagram showing the configuration of a neural network calculation device according to the ninth embodiment. In the above-described embodiments, the unit of clock supply has been explained as a processor element, but in this embodiment, more detailed control for suppressing fluctuations in power consumption will be explained.

[0112] The neural network calculation device of this embodiment is a neural network calculation device (10) that performs neural network processing, and includes a plurality of processor elements (104) arranged in an array, an activation memory (103) that stores input activation data to be supplied to the processor elements, and a main controller (101) that controls the operation of each of the processor elements, wherein the processor elements each have a plurality of processing units that perform different functions and are configured to be able to supply a clock to each processing unit independently, and the main controller is configured to individually control the start timing of the plurality of processing units of each processor element when starting calculation processing, and start the processor elements in stages. Hereinafter, a detailed description will be given with reference to the drawings.

[0113] The basic configuration of the neural network calculation device of the ninth embodiment is the same as that of the first embodiment, but in the neural network calculation device 130 of this embodiment, the processor element 1304 is internally divided into a calculation unit 13041 and a data transfer unit 13042. In this embodiment, the processor element is divided and controlled by the calculation unit 13041 and the data transfer unit 13042, and power is gradually increased by a combination of data calculation and calculation processing.

[0114] 13041 is a calculation unit that performs a multiply-and-accumulate operation based on input activations and weight values. The calculation unit 13041 is composed of at least one multiply-and-accumulate unit, and typically has multiple units implemented depending on the required performance. Depending on the layer processing to be supported, the calculation unit 13041 may be capable of performing operations other than those performed by the multiply-and-accumulate unit.

[0115] A data transfer unit 13042 transfers activation data to be shared between adjacent processor elements when performing convolution operations using a kernel of 2x2 or more.

[0116] The arithmetic unit 13041 and the data transfer unit 13042 are configured so that clocks can be supplied independently from outside the processor element 1304. Furthermore, they may be configured so that resets are controlled independently. Alternatively, although this embodiment is based on the premise that clock supply is controlled from outside the processor element, the clock may be controlled within the processor element 1304.

[0117] During normal convolution calculation processing, both the calculation unit 13041 and the data transfer unit 13042 are executed simultaneously. If only one of the calculation unit 13041 or the data transfer unit 13042 is executed, power consumption will be lower than during normal processing. Furthermore, when comparing the circuits of the calculation unit 13041 and the data transfer unit 13042, the calculation unit 13041 generally has a larger logic scale and therefore consumes more power. Utilizing the above features, in this embodiment, processing is executed in the following sequence.

[0118] FIG. 14 is a diagram showing the processing sequence of this embodiment. For simplicity, FIG. 14 assumes that eight columns of processor elements are arranged, but the present invention is not limited to eight columns. FIG. 14 shows the operating state of each processor element when multiple convolutional layers are processed consecutively. The horizontal axis represents time, and the vertical axis represents the processor element column. FIG. 14 shows a sequence when three layers are executed consecutively, but this sequence is valid as long as at least two or more layers can be processed.

[0119] When performing layer processing for Conv0, the main controller 1301 first instructs both the calculation unit 13041 and the data transfer unit 13042 to operate for the even-numbered columns (0th, 2nd, 4th, and 6th columns), and at the same time instructs the data transfer unit 13042 to operate only for the odd-numbered columns (1st, 3rd, 5th, and 7th columns), without operating the calculation unit 13041. At this time, the data transfer unit 13042 is executed by all processor elements 1304, so that data required by all processor elements 1304 is supplied as usual, but calculation processing is only performed on the even-numbered columns, and no calculation is performed on the odd-numbered columns.

[0120] Next, the main controller 1301 instructs both the calculation unit 13041 and the data transfer unit 13042 to operate for the odd-numbered columns (1st, 3rd, 5th, and 7th columns), and at the same time instructs the data transfer unit 13042 to operate only for the even-numbered columns (0th, 2nd, 4th, and 6th columns), without operating the calculation unit 13041. With this combination, calculation processing for the odd-numbered columns is executed, and together with the previous processing, Conv0th layer processing is completed.

[0121] The processor element 1304 continues to execute the Conv1 layer, but the main controller 1301 issues an instruction to all processor elements 1304 to activate both the arithmetic unit 13041 and the data transfer unit 13042 from the Conv1 layer onwards. This allows all of the processor elements to operate. As a result, the utilization rate of the arithmetic unit during Conv0 layer processing is limited to 50% of the normal level, so power consumption can be controlled to be smaller than the power consumed during layer processing from Conv1 onwards. [Explanation of symbols]

[0122] 1 SOC, 10 neural network calculation unit, 11 clock generator, 101 main controller, 102 clock controller, 103 activation memory, 104 processing element, 201~204 Enable signals, 205~208 Clock signals, 401~404 Enable signals, 405~408 Clock signals, 6 SOC, 60 Neural network computing unit, 61 Clock generator, 601 main controller, 602 clock controller, 603 activation memory, 604 processing element, 605~609 group, 8 SOC, 80 Neural network computing device, 81 Clock generator, 801 Main controller, 802 Clock controller, 803 activation memory, 804 processing element, 805~807 group, 9 SOC, 90 Neural network computing device, 91 Clock generator, 901 Main controller, 902 Clock controller, 903 activation memory, 904 processing element, 905 diagnostic circuits, 13 SOC, 130 neural network computing device, 131 clock generator, 1301 Main controller, 1302 Clock controller, 1303 activation memory, 1304 processing element, 13041 Calculation unit, 13042 Data transfer unit.

Claims

1. A neural network calculation device (10) for performing neural network processing, a plurality of processor elements (104) arranged in an array; an activation memory (103) for storing input activation data to be supplied to the processor element; a main controller (101) that controls the timing at which a clock supplied from a clock generator (11) is supplied to each processor element and also controls the operation of each of the processor elements; Equipped with the main controller has a function of detecting a progress state of neural network processing, and the function of detecting a progress state of neural network processing determines whether or not there is a subsequent neural network processing when the neural network processing is a final layer processing; The main controller continues to supply clocks when there is subsequent neural network processing, and instructs the main controller to perform an operation check process when there is no subsequent neural network processing, thereby gradually reducing power consumption.

2. The arrayed processor elements have the function of transferring instructions from the main controller between adjacent processor elements, the clock lines of the clock generator are connected to each group of the processor elements logically grouped together, 2. The neural network calculation device according to claim 1, wherein the main controller supplies a clock to the processor elements for each group in synchronization with the supply of data for neural network calculation to the processor elements, and increases the number of groups to which the clock is supplied as time passes from the start of calculation processing.

3. The arrayed processor elements have the function of transferring instructions from the main controller between adjacent processor elements, the clock lines of the clock generator are connected to each column of processor elements; 2. The neural network calculation device according to claim 1, wherein the main controller starts calculation processing for each column, instructs a clock generator to supply a clock to the corresponding column, and sequentially supplies clocks from one end side of the column of processor elements in synchronization with the supply of data for neural network calculation.

4. The clock lines of the clock generator are connected to groups (605-609) of logically grouped processor elements (604) radially from the end points of the arrayed processor elements, and the clock lines are connected independently to each of the groups.

2. The neural network calculation device according to claim 1, wherein the main controller (601) starts calculation processing for each of the radially connected groups, instructs the clock generator (61) to supply a clock to the group of processor elements, and sequentially supplies clocks from the end points of the processor elements in synchronization with the supply of data for neural network calculation.

5. The clock lines of the clock generator are connected independently to each group of processor elements (804) that are grouped together in accordance with the physical layout of the processor elements (804) that are arranged and wired on semiconductor silicon, 2. The neural network calculation device according to claim 1, wherein the main controller starts calculation processing for each group and instructs a clock generator to supply clocks to the corresponding processor elements, supplying clocks sequentially so as to equalize power consumption on semiconductor silicon.

6. 6. The neural network operation device according to claim 1, wherein the main controller instructs the processor elements to selectively execute an operation process and an operation check process.

7. 6. The neural network operation device according to claim 1, wherein the main controller issues an instruction to gradually increase the number of executions of the operation check process before starting the neural network process.

8. The neural network calculation device according to any one of claims 1 to 7, wherein the main controller detects the type of neural network processing and instructs some of the processor elements to perform calculation check processing based on power consumption predetermined according to the type of neural network processing.

9. The processor element has a plurality of processing units each performing a different function, and is configured to be able to supply a clock to each processing unit independently; 2. The neural network computing device according to claim 1, wherein said main controller controls the start timing of each of the processor elements individually at the start of the computing process, and starts said processor elements in stages.

Images