Semiconductor Devices

Abstract

To provide a semiconductor device with which it is possible to reduce power consumption.SOLUTION: A group controller 22a detects a zero weight parameter in which a value is zero from among n×m weight parameters transferred to a weight parameter buffer WBF. Then, the group controller 22a, by input of the zero weight parameter, replaces n×m weight parameters transferred to the weight parameter buffer WBF, so that multiplication results of n multipliers MUL included in a target multiplier group which is one of m multiplier groups MULG become zero together. The group controller 22a controls a target multiplier group MULG [2] to be disabled, and replaces n×m pixel data transferred to a data input buffer IBF, on the basis of replacement of the n×m weight parameters.SELECTED DRAWING: Figure 3A

Description

[Technical Field]

[0001] The present invention relates to a semiconductor device, for example, a semiconductor device that executes neural network processing. [Background technology]

[0002] Patent Document 1 discloses a data processing device that determines whether binarized input data is "0" or "1," and reads out the corresponding coefficient from a storage unit only when the input data is "1," and sequentially accumulates and adds up the read coefficients. The data processing device also discloses a method that counts the frequency of each coefficient that is read out, and adjusts all coefficients so that the coefficient with the highest frequency is offset to 0, thereby omitting the calculation of accumulation and addition associated with the coefficient with the highest frequency.

[0003] Patent Document 2 discloses a semiconductor device that generates a single integrated coefficient table by integrating input coefficient tables for multiple channels, multiplies each coefficient included in the integrated coefficient table by each pixel value of an input image, and accumulates the multiplication results by dividing them by channel number. Patent Document 2 also discloses an integrated coefficient table in which the largest coefficient is extracted from coefficients at the same matrix location for multiple channels, or in which the matrix size is expanded to accommodate coefficients for multiple channels. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-76900 [Patent Document 2] Japanese Patent Application Publication No. 2019-40403 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, in neural network processing such as a convolutional neural network (CNN), a massive amount of calculation processing is performed using multiple multiply-accumulate (MAC) circuits mounted on a semiconductor device. Specifically, a MAC circuit mainly performs a multiply-accumulate operation between multiple pixel data included in image data and multiple weight parameters included in a filter. In recent years, in order to perform a massive amount of multiply-accumulate operations in a short period of time, a large number of MAC circuits operating in parallel are sometimes mounted on a semiconductor device. As a result, the increase in power consumption, in particular, has become non-negligible.

[0006] The embodiments described below have been made in consideration of the above, and other problems and novel features will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0007] A semiconductor device according to one embodiment performs neural network processing and includes one or more memories, m multiplier groups each including n multipliers, where n and m are integers equal to or greater than 2, a weight parameter buffer, a data input buffer, first and second DMA controllers, and a group controller. The memory stores weight parameters and pixel data. The weight parameter buffer outputs the n×m weight parameters it holds to each of the n×m multipliers. The data input buffer outputs the n×m pixel data it holds to each of the n×m multipliers. The first DMA controller transfers the n×m weight parameters from the memory to the weight parameter buffer. The second DMA controller transfers the n×m pixel data from the memory to the data input buffer. The group controller controls enable / disable for each of the m multiplier groups. More specifically, the group controller detects a zero weight parameter, whose value is zero, from among the n×m weight parameters transferred to the weight parameter buffer. Then, the group controller replaces the n×m weight parameters transferred to the weight parameter buffer so that, upon receiving the zero weight parameter, the multiplication results of the n multipliers included in a target multiplier group, which is one of the m multiplier groups, are all zero. The group controller controls the target multiplier group to be disabled, and replaces the n×m pixel data transferred to the data input buffer based on the replacement of the n×m weight parameters. [Effects of the Invention]

[0008] By using the semiconductor device of one embodiment, power consumption can be reduced. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of a main part of a semiconductor device according to a first embodiment. [Figure 2]FIG. 2 is a schematic diagram showing an example of the general operation of the main part of the MAC unit in FIG. [Figure 3A] FIG. 3A is a diagram showing a detailed configuration example and an operation example of the main part of the neural network engine in FIG. [Figure 3B] FIG. 3B is a diagram showing an example of a configuration and an example of an operation that is a comparative example to FIG. 3A. [Figure 4] FIG. 4 is a flowchart showing an example of the processing contents of the group controller in FIG. 3A. [Figure 5] FIG. 5 is a timing chart showing an example of a schematic operation of the entire neural network engine in FIG. 3A. [Figure 6] FIG. 6 is a diagram showing a detailed configuration example and an operation example of a main part of the neural network engine in FIG. 1 in the semiconductor device according to the second embodiment. [Figure 7] FIG. 7 is a diagram showing a detailed configuration example and an operation example of a main part of the neural network engine in FIG. 1 in the semiconductor device according to the third embodiment. [Figure 8] FIG. 8 is a diagram for explaining an example of the processing contents of the decompressor in FIG. [Figure 9] FIG. 9 is a diagram showing a specific example of the processing content in FIG. [Figure 10] FIG. 10 is a diagram showing a detailed configuration example and an operation example of a main part of the neural network engine in FIG. 1 in the semiconductor device according to the fourth embodiment. [Figure 11] FIG. 11 is a schematic diagram showing a specific example of the operation of the group controller in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the following embodiments, the description will be divided into multiple sections or embodiments for convenience, as necessary. However, unless otherwise specified, they are not unrelated to one another, and one is a partial or complete modification, detail, supplementary explanation, etc., of the other. Furthermore, in the following embodiments, when the number of elements (including the number, numerical value, amount, range, etc.) is mentioned, it is not limited to the specific number, and may be more or less than the specific number, unless otherwise specified or clearly limited to a specific number in principle. Furthermore, in the following embodiments, it goes without saying that the components (including element steps, etc.) are not necessarily essential, unless otherwise specified or clearly considered essential in principle. Similarly, in the following embodiments, when the shape, positional relationship, etc. of components, etc. are mentioned, it is intended to include those that are substantially similar to or similar to the shape, etc., unless otherwise specified or clearly considered not to be essential in principle. The same applies to the above numerical values and ranges.

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings. In all drawings for explaining the embodiments, components having the same functions are designated by the same reference numerals, and repeated explanations thereof will be omitted. In the following embodiments, explanations of the same or similar parts will not be repeated unless particularly necessary.

[0012] (Embodiment 1) <Outline of semiconductor device> Fig. 1 is a schematic diagram showing an example of the configuration of a main part of a semiconductor device according to embodiment 1. The semiconductor device 10 shown in Fig. 1 is, for example, an SoC (System on Chip) configured on a single semiconductor chip. The semiconductor device 10 is typically mounted in an ECU (Electronic Control Unit) of a vehicle, and provides the functions of an ADAS (Advanced Driver Assistance System).

[0013] The semiconductor device 10 shown in FIG. 1 includes a neural network engine (NNE) 15, a processor 17 such as a CPU (Central Processing Unit), one or more memories MEM1 and MEM2, and a system bus 16. The system bus 16 interconnects the neural network engine 15, the memories MEM1 and MEM2, and the processor 17. The neural network engine 15 executes neural network processing, such as CNN. The processor 17 executes a predetermined program stored in the memory MEM1, causing the semiconductor device 10 to perform predetermined functions, including controlling the neural network engine 15.

[0014] The memory MEM1 is, for example, a dynamic random access memory (DRAM). The memory MEM1 stores image data DT consisting of a plurality of pixel data, parameters PR, and a header HD added to the parameters PR. The parameters PR include weight parameters WP and bias parameters BP. The header HD includes various information for controlling the sequence operation of the neural network engine 15, including setting information for a parameter switch circuit SWP (described later).

[0015] The neural network engine 15 includes multiple DMA controllers DMAC1 and DMAC2, a MAC unit 20, a sequence controller 21, a group controller 22, a decompressor 23, a register REG, a data switch circuit SWD, a parameter switch circuit SWP, and various buffers, including a weight parameter buffer WBF, a data input buffer IBF, and a data output buffer OBF.

[0016] The MAC unit 20 includes m multiplier groups MULG[1] to MULG[m], where m is an integer greater than or equal to 2, and k adders ADD[1] to ADD[k], where k is an integer greater than or equal to 2. Each of the m multiplier groups MULG[1] to MULG[m] includes n multipliers MUL1 to MULn, where n is an integer greater than or equal to 2. In this specification, the m multiplier groups MULG[1] to MULG[m] are collectively referred to as the multiplier group MULG, the n multipliers MUL1 to MULn are collectively referred to as the multiplier MUL, and the k adders ADD[1] to ADD[k] are collectively referred to as the adder ADD. The MAC unit 20 performs product-accumulation operations using the n×m multipliers MUL and the k adders ADD.

[0017] The weight parameter buffer WBF is configured, for example, by a latch circuit or the like, and outputs the n×m weight parameters W that it holds to the n×m multipliers MUL in the MAC unit 20. The DMA controller DMAC1 transfers the n×m weight parameters W from the memory MEM1 to the weight parameter buffer WBF via the system bus 16.

[0018] More specifically, the memory MEM1 stores, for example, compressed weight parameters WP. The DMA controller DMAC1 reads the header HD and the compressed weight parameters WP from the memory MEM1, stores the header HD in a register REG, and transfers the compressed weight parameters WP to the weight parameter buffer WBF via a decompressor 23 and a parameter switch circuit SWP.

[0019] At this time, the decompressor 23 restores the compressed weight parameters WP to n×m weight parameters W. The parameter switch circuit SWP is configured, for example, by a crossbar switch or the like, and stores the restored weight parameters W in each storage area of the weight parameter buffer WBF by performing one-to-one connection, one-to-N connection, N-to-1 connection, or the like based on the settings. Note that the header HD stores, for example, setting information and the like for the switch circuit SWP.

[0020] The memory MEM2 is, for example, an SRAM (Static Random Access Memory) or the like, and is used as a high-speed cache memory for the neural network engine 15. For example, image data DT in the memory MEM1, in other words, pixel data, is copied in advance to the memory MEM2 and then used by the neural network engine 15. The data input buffer IBF is, for example, composed of a latch circuit or the like, and outputs the n×m pieces of pixel data Di that it holds to the n×m multipliers MUL in the MAC unit 20. The DMA controller DMAC2 transfers the n×m pieces of pixel data Di from the memory MEM2 to the data input buffer IBF.

[0021] As a result, the MAC unit 20 performs a multiply-and-accumulate operation between the n×m weight parameters W from the weight parameter buffer WBF and the n×m pixel data Di from the data input buffer IBF, i.e., convolutional layer processing. Although details are omitted, the MAC unit 20 may also perform various processes required for CNN, such as adding the value of a bias parameter BP to the result of the multiply-and-accumulate operation, calculating an activation function, and processing a pooling layer. The MAC unit 20 stores the pixel data Do obtained by such CNN processing in the data output buffer OBF.

[0022] The DMA controller DMAC2 transfers pixel data Do from the data output buffer OBF to the memory MEM2. The pixel data Do transferred to the memory MEM2 is used as pixel data Di to be input to the next convolutional layer. More specifically, the transfer of pixel data between the DMA controller DMAC2 and the data input buffer IBF or the data output buffer OBF is performed via a data switch circuit SWD. The switch circuit SWD is configured, for example, by a crossbar switch, and performs one-to-one connection, one-to-N connection, N-to-1 connection, or the like based on the settings.

[0023] The sequence controller 21 controls the overall operation sequence of the neural network engine (NNE) 15. As one of its functions, the sequence controller 21 performs connection settings for the parameter switch circuit SWP based on information in the header HD held in the register REG. The sequence controller 21 also performs transfer settings for the DMA controller DMAC2, connection settings for the data switch circuit SWD, and operation settings for the MAC unit 20 based on setting information (not shown) from the processor 17, for example.

[0024] The transfer settings of the DMA controller DMAC2 determine the address range when transferring pixel data Di from memory MEM2, the address range when transferring pixel data Do to memory MEM2, etc. The connection settings of the data switch circuit SWD determine the detailed correspondence between the read addresses of memory MEM2 and the multiple storage areas included in the data input buffer IBF, the detailed correspondence between the multiple storage areas included in the data output buffer OBF and the write addresses of memory MEM2, etc.

[0025] In the operation setting of the MAC unit 20, the connection relationship between the n×m multipliers MUL and the k adders ADD is determined based on the size of the filter used in the processing of the convolution layer, the number of input channels, the number of output channels, etc. Note that although the sequence controller 21 is provided here, the processor 17 may control the operation sequence of the entire neural network engine (NNE) 15 instead of the sequence controller 21.

[0026] The group controller 22, which will be described in detail later, controls the enable / disable of each of the m multiplier groups MULG[1] to MULG[m] using enable signals EN[1] to EN[m]. A multiplier group MULG controlled to be enabled performs a multiplication operation using n multipliers MUL. On the other hand, a multiplier group MULG controlled to be disabled does not perform a multiplication operation and outputs, for example, zero, which is a default value, as the multiplication result of the n multipliers MUL.

[0027] Fig. 2 is a schematic diagram showing an example of the general operation of the main parts of the MAC unit in Fig. 1. Fig. 2 shows input image data DTi consisting of a plurality of pixel data Di1, Di2, Di3, Di4, ..., and filters FLTa, FLTb, ... for a plurality of output channels A, B, .... For simplicity of description, filter FLTa is composed of X×Y (=2×2) weighting parameters Wa1 to Wa4, and filter FLTb is also composed of X×Y (=2×2) weighting parameters Wb1 to Wb4. In practice, filters usually have a size of X×Y×Z, where Z is the number of input channels, and X×Y is often 3×3 or 5×5, etc.

[0028] The MAC unit 20 generates output image data DTo for output channel A, in other words, certain pixel data Do1A in the feature map, by performing a product-sum operation on pixel data Di1 to Di4 included in a certain pixel range and weight parameters Wa1 to Wa4 for output channel A. The MAC unit 20 also generates certain pixel data Do1B in the feature map of output channel B by performing a product-sum operation on pixel data Di1 to Di4 included in the same pixel range and weight parameters Wb1 to Wb4 for output channel B. Furthermore, the MAC unit 20 generates other pixel data in the feature map by performing similar product-sum operations while shifting the pixel range in the input image data DTi.

[0029] Through such operations, the MAC unit 20 executes processing of a certain convolutional layer in the CNN. The output image data DTo for the multiple output channels A, B, ... is used as input image data for the multiple input channels in the next convolutional layer. Here, the multiple weight parameters Wa1 to Wa4, Wb1 to Wb4, ... may generally include many zero weight parameters whose values are zero. In the example shown in FIG. 2, one weight parameter Wa4 in the filter FLTa and one weight parameter Wb1 in the filter FLTb are zero weight parameters. In the multiplication using zero weight parameters as input, the multiplication result is also zero, so there is no need to operate the multiplier MUL.

[0030] <Neural network engine details> FIG. 3A is a diagram showing a detailed configuration example and an operation example of the main part of the neural network engine in FIG. 1. FIG. 3B is a diagram showing a configuration example and an operation example that are comparative examples to FIG. 3A. First, in the comparative example shown in FIG. 3B, the MAC unit 30 includes n×m multipliers MUL, k adders ADD[1], ADD[2], ..., and an operation switch circuit SWM. However, unlike the case of FIG. 1, the n×m multipliers MUL are not grouped into a multiplier group MULG.

[0031] The operation switch circuit SWM is configured, for example, by a crossbar switch. The switch circuit SWM determines the connection relationship between the n×m multipliers MUL and the k adders ADD[1], ADD[2], ... based on a setting signal from the sequence controller 21. In this example, to perform the processing of the convolution layer shown in FIG. 2, the switch circuit SWM connects four multipliers MUL to each of the k adders ADD[1], ADD[2], ...

[0032] The DMA controller DMAC1 transfers weight parameters Wa1 to Wa4 in filter FLTa to storage areas #1 to #4 of the weight parameter buffer WBF, respectively, and transfers weight parameters Wb1 to Wb4 in filter FLTb to storage areas #5 to #8, respectively, via the decompressor 23 and the parameter switch circuit SWP. At this time, the switch circuit SWP determines the connection relationship between the output from the decompressor 23 and each storage area #1 to #8, ... in the weight parameter buffer WBF based on a setting signal from the sequence controller 21. Note that each weight parameter is represented by, for example, 8-bit data, and each storage area is, for example, an 8-bit area.

[0033] On the other hand, the pixel data DMA controller DMAC2 shown in Fig. 1 has a data input DMA controller DMAC2i and a data output DMA controller DMAC2o, as shown in detail in Fig. 3B. Similarly, the data switch circuit SWD shown in Fig. 1 has a data input switch circuit SWDi and a data output switch circuit SWDo, as shown in detail in Fig. 3B.

[0034] The data input DMA controller DMAC2i transfers pixel data Di1 to Di4 from a memory MEM2 (not shown) to a data input buffer IBF via a data input switch circuit SWDi based on a setting signal from the sequence controller 21. Specifically, the pixel data Di1 to Di4 are transferred to storage areas #1 to #4 of the data input buffer IBF, respectively, and the pixel data Di1 to Di4 are also transferred to storage areas #5 to #8, respectively. At this time, the switch circuit SWDi stores the same pixel data Di1 in multiple storage areas #1, #5, ... by, for example, establishing a one-to-N connection based on a setting signal from the sequence controller 21. Note that each pixel data Di is represented by, for example, 8-bit data, and each storage area is, for example, an 8-bit area.

[0035] Using four multipliers MUL and an adder ADD[1], the MAC unit 20 performs a product-sum operation on the four pixel data Di1 to Di4 held in storage areas #1 to #4 of the data input buffer IBF and the four weight parameters Wa1 to Wa4 held in storage areas #1 to #4 of the weight parameter buffer WBF.The MAC unit 20 then stores the product-sum operation result as pixel data Do1A in storage area #1 of the data output buffer OBF.

[0036] Furthermore, using four multipliers MUL and adder ADD[2], MAC unit 20 performs a multiply-and-accumulate operation on four pixel data Di1-Di4 held in storage areas #5-#8 of data input buffer IBF and four weight parameters Wb1-Wb4 held in storage areas #5-#8 of weight parameter buffer WBF. MAC unit 20 then stores the result of the multiply-and-accumulate operation as pixel data Do1B in storage area #2 of data output buffer OBF. Each of storage areas #1, #2, ... in data output buffer OBF is, for example, an 8-bit area.

[0037] The data output DMA controller DMAC2o transfers pixel data Do1A and Do1B from the data output buffer OBF to a memory MEM2 (not shown) via a data output switch circuit SWDo based on a setting signal from the sequence controller 21. At this time, the switch circuit SWDo determines appropriate mapping of the pixel data Do1A and Do1B in the memory MEM2 based on, for example, a setting signal from the sequence controller 21.

[0038] In this configuration and operation, the weight parameters Wa4 and Wb1 stored in storage areas #4 and #5 of the weight parameter buffer WBF are zero weight parameters, as described in Figure 2. Therefore, the two multipliers MUL that receive the weight parameters Wa4 and Wb1 as input do not necessarily need to perform multiplication operations. If the two multipliers MUL do not perform multiplication operations, power consumption can be reduced, and particularly when the number of multipliers MUL is several thousand or more and the number of zero weight parameters is large, power consumption can be reduced more significantly.

[0039] Therefore, for example, it is conceivable to provide a circuit for each multiplier MUL that outputs a disable signal when, for example, an 8-bit weight parameter W stored in a storage area is all zero. However, for example, when several thousand or more multipliers MUL are provided, providing such a circuit for each multiplier MUL may increase the circuit area. In particular, from the perspective of layout design, it may be difficult to secure space for placing such a circuit and wiring around each multiplier MUL. Therefore, it is advantageous to use a configuration such as that shown in FIG. 3A.

[0040] 3A, unlike the case of FIG. 3B, in the MAC unit 20, n×m multipliers MUL are grouped into m multiplier groups MULG in units of n multipliers, or in units of two multipliers in this example. That is, each of the m multiplier groups MULG has two multipliers MUL1 and MUL2. Enable signals EN[1], EN[2], EN[3], EN[4], ... that control enable / disable are input to the m multiplier groups MULG[1], MULG[2], MULG[3], MULG[4], ..., respectively.

[0041] Multiplier group MULG[1] is assigned to storage areas #1 and #2 of weight parameter buffer WBF and storage areas #1 and #2 of data input buffer IBF. Similarly, multiplier group MULG[2] is assigned to storage areas #3 and #4 of each buffer, multiplier group MULG[3] is assigned to storage areas #5 and #6 of each buffer, and multiplier group MULG[4] is assigned to storage areas #7 and #8 of each buffer. Furthermore, in FIG. 3A, unlike in FIG. 3B, a group controller 22a is provided.

[0042] [Group Controller Details] Fig. 4 is a flow diagram showing an example of the processing contents of the group controller in Fig. 3A. In Fig. 4, the group controller 22a receives n × m weight parameters W output from the DMA controller DMAC1 via the decompressor 23, that is, weight parameters Wa1 to Wa4 and Wb1 to Wb4 in the example shown in Fig. 3A (step S101).

[0043] Next, the group controller 22a detects zero weight parameters whose values are zero, such as weight parameters Wa4 and Wb1 in the example shown in FIG. 3A (step S102). At this time, the group controller 22a may detect the zero weight parameters by providing a zero determination circuit or the like that performs an OR operation on multiple bits, for example, 8 bits, that make up the weight parameter W. Furthermore, the group controller 22a does not necessarily need to detect zero weight parameters simultaneously for n×m weight parameters W, but may detect zero weight parameters in (n×m) / M iterations, with a predetermined number M as a unit. In this case, M zero determination circuits may be provided.

[0044] Next, the group controller 22a determines whether the number of zero weight parameters detected in step S102 is n or more (step S103). If the number of zero weight parameters is less than n (step S103: No), the group controller 22a ends the process, and if the number is n or more (step S103: Yes), the group controller 22a proceeds to step S104. In the example of FIG. 3A, the number n of multipliers MUL included in each multiplier group MULG is 2, and the number of zero weight parameters Wa4 and Wb1 is 2, so the process of step S104 is performed.

[0045] In step S104, the group controller 22a rearranges the n×m weight parameters W transferred to the weight parameter buffer WBF so that, upon receiving the zero weight parameters, the multiplication results of the n multipliers MUL included in the target multiplier group MULG, which is one of the m multiplier groups MULG, are all zero. That is, the group controller 22a rearranges the weight parameters W that would normally be stored in the weight parameter buffer WBF in the arrangement shown in Fig. 3B, as shown by reference numeral 25a in Fig. 3A.

[0046] In detail, the group controller 22a outputs a swap setting signal SSW to the parameter switch circuit SWP, for example, instructing the swap as indicated by reference numeral 25a in FIG. 3A. As a result, the group controller 22a swaps, via the switch circuit SWP for the multiplier group MULG[2], the weight parameter Wa3 that would normally be transferred to storage area #3 of the weight parameter buffer WBF with the weight parameter Wb1 that would normally be transferred to storage area #5. As a result, the multiplication results of the two multipliers MUL1 and MUL2 included in the target multiplier group MULG[2] are both zero.

[0047] Regarding the specific replacement procedure, in this example, the group controller 22a sequentially detects the positions of zero weight parameters based on the normal arrangement of the weight parameters Wa1 to Wa4 and Wb1 to Wb4 transferred to the weight parameter buffer WBF, i.e., the arrangement shown in Fig. 3B. Then, the group controller 22a determines the first detected position, i.e., the multiplier group MULG[2] assigned to memory area #4 in the example of Fig. 3B, as the target.

[0048] Then, the group controller 22a swaps the weight parameter Wa3, which is a weight parameter of another storage area #3 assigned to the target multiplier group MULG[2] and which is a non-zero weight parameter that would normally be transferred to storage area #3, with the second detected zero weight parameter, i.e., the weight parameter Wb1 that would normally be transferred to storage area #5. After that, if there are third and fourth detected zero weight parameters, a similar swap is performed on the multiplier group MULG assigned to the third detected position.

[0049] That is, in this procedure, the replacement is performed on the multiplier groups MULG assigned to the 1st, n+1th, 2n+1th, ... detected positions. Note that a similar procedure may be performed on the multiplier groups MULG assigned to the nth, 2nth, 3nth, ... detected positions. For example, by performing the replacement using such a procedure, the number of replacements can be reduced to some extent. However, the replacement procedure is not limited to this particular procedure, and various procedures may be used, such as a procedure in which zero-weight parameters are filled in order starting from memory area #1.

[0050] 4, after step S104, the group controller 22a disables the target multiplier group MULG, i.e., multiplier group MULG[2] in the example of FIG. 3A, using the enable signal EN[2] (step S105). Furthermore, based on the replacement of the weight parameters Wa1 to Wa4 and Wb1 to Wb4 in step S104, the group controller 22a replaces the n×m pixel data transferred to the data input buffer IBF, as shown by reference numeral 25b in FIG. 3A (step S106).

[0051] 3A, the group controller 22a swaps pixel data Di3, which would normally be transferred to storage area #3 of the data input buffer IBF, with pixel data Di1, which would normally be transferred to storage area #5. Specifically, the group controller 22a outputs a swap setting signal SSW, which instructs such a swap, to the data input switch circuit SWDi.

[0052] Furthermore, similar to the case of step S106, the group controller 22a changes the setting of the calculation switch circuit SWM using the exchange setting signal SSW based on the exchange of the weight parameters Wa1 to Wa4 and Wb1 to Wb4 in step S104 (step S107). Specifically, as shown by reference numeral 25c in Fig. 3A, the group controller 22a exchanges the connection destination of the multiplier MUL1 in the multiplier group MULG[2] assigned to the memory area #3 with the connection destination of the multiplier MUL1 in the multiplier group MULG[3] assigned to the memory area #5.

[0053] 3A, the connection destination of the multiplier MUL1 in the multiplier group MULG[2] is changed from the adder ADD[1] to the adder ADD[2], and the connection destination of the multiplier MUL1 in the multiplier group MULG[3] is changed from the adder ADD[2] to the adder ADD[1]. As a result, it is possible to obtain the same product-sum operation result as in the case of FIG. 3B with the multiplier group MULG[2] disabled.

[0054] In the example of FIG. 3A, for the sake of simplicity, the number n of multipliers MUL in the multiplier group MULG is 2, but it may be, for example, a value of several tens or several hundreds. Also, in this example, as shown in FIG. 2, a case is taken as an example in which a multiply-and-accumulate operation using a plurality of filters FLTa, FLTb, ... is executed in parallel in a certain control cycle. However, if the size of one filter is large, for example, X×Y×Z=3×3×1000, a multiply-and-accumulate operation using one filter may be executed in a certain control cycle. In this case, it is not necessarily required to perform the switching within the operation switch circuit SWM shown in FIG. 3A.

[0055] [Overall operation of the neural network engine] Figure 5 is a timing chart showing an example of the overall operation of the neural network engine in Figure 3A. From time t1 to time t3 shown in Figure 5, the data input DMA controller DMAC2i transfers pixel data Di from memory MEM2 to the data input buffer IBF via the data input switch circuit SWDi. More specifically, the DMA controller DMAC2i has multiple channels, and transfers pixel data Di in parallel using the multiple channels.

[0056] In this example, at time t0, which is earlier than time t1, the parameter DMA controller DMAC1 starts transferring weight parameters W from the memory MEM1 to the weight parameter buffer WBF. The group controller 22a receives the weight parameters W transferred to the weight parameter buffer WBF as input and detects zero weight parameters as described with reference to FIG. 4 and other figures.

[0057] Then, based on the detection result of the zero weight parameter, the group controller 22a outputs a replacement setting signal SSW to the parameter switch circuit SWP and the data input switch circuit SWDi at time t2, which is before time t3. The group controller 22a also outputs the replacement setting signal SSW to the operation switch circuit SWM. Furthermore, in conjunction with the replacement of the weight parameter W, the group controller 22a disables the target multiplier group MULG[2] using the enable signal EN[2].

[0058] From time t2 to time t3, the parameter switch circuit SWP stores the weight parameter W in a storage area in the weight parameter buffer WBF based on a setting signal from the sequence controller 21, and switches the storage destination based on a switching setting signal SSW from the group controller 22a. Similarly, the data input switch circuit SWDi stores the pixel data Di in a storage area in the data input buffer IBF based on a setting signal from the sequence controller 21, and switches the storage destination based on a switching setting signal SSW from the group controller 22a.

[0059] From time t3 to time t4, m multiplier groups MULG[1] to MULG[4] multiply pixel data Di stored in the data input buffer IBF by weight parameters W stored in the weight parameter buffer WBF. However, at this time, the target multiplier group MULG[2] is disabled by an enable signal EN[2] from the group controller 22a.

[0060] From time t4 to time t5, the operation switch circuit SWM connects the output of each multiplier MUL in the m multiplier groups MULG[1] to MULG[4] to one of the k adders ADD[1], ADD[2], ... based on a setting signal from the sequence controller 21. At this time, the operation switch circuit SWM switches the connection destination based on the sequence controller 21 based on a switching setting signal SSW from the group controller 22a. Then, the product-sum operation results from the k adders ADD[1], ADD[2], ..., i.e., the output pixel data Do, are stored in each of the storage areas #1, #2, ... in the data output buffer OBF.

[0061] From time t4 to time t6, the data output DMA controller DMAC2o transfers pixel data Do stored in each storage area #1, #2, ... in the data output buffer OBF to the memory MEM2 via the data output switch circuit SWDo. More specifically, the DMA controller DMAC2o has multiple channels, and transfers the pixel data Do in parallel using the multiple channels. At this time, the data output switch circuit SWDo transfers the pixel data Do stored in each storage area #1, #2, ... in the data output buffer OBF to a predetermined channel of the DMA controller DMAC2o based on a setting signal from the sequence controller 21.

[0062] In conjunction with processing in the convolutional layer, the neural network engine 15 repeatedly executes the operations shown from time t1 to time t6 in Fig. 5. When changing the filter, in addition, new operations are performed by the DMA controller DMAC1 and the group controller 22a for the parameters shown from time t0 to time t2.

[0063] <Major Effects of the First Embodiment> As described above, in the system of the first embodiment, by providing the group controller 22a, the weight parameters W transferred to the weight parameter buffer WBF are swapped so that the multiplication results of the n multipliers MUL included in a certain target multiplier group MULG are all zero. This allows the target multiplier group MULG to be disabled, thereby reducing power consumption. Furthermore, by providing an enable signal for each multiplier group MULG rather than for each multiplier MUL, it is possible to suppress an increase in the number of wirings in the MAC unit 20, thereby simplifying layout design, etc. These effects become more pronounced, particularly as the number of multipliers MUL included in the MAC unit 20 increases, i.e., when a large number of multiply-accumulate operations need to be performed at high speed.

[0064] (Embodiment 2) <Neural network engine details> 6 is a diagram showing a detailed configuration example and an operation example of the main parts of the neural network engine in FIG. 1 in a semiconductor device according to the second embodiment. The configuration example shown in FIG. 6 differs from the configuration example shown in FIG. 3A in the following two points. The first difference is that MAC unit 20 includes registers R1 to R8, ... that hold the multiplication results of each multiplier MUL, and further includes a zero register RZ that always holds a zero value. The second difference is that operation switch circuit SWM performs a different switching operation from that in FIG. 3A based on a switching setting signal SSW from group controller 22b.

[0065] Although not shown in FIG. 3A, the multiplication results of each multiplier MUL are generally stored in registers R1 to R8, ... as shown in FIG. 6, and then added by adders ADD[1], ADD[2], .... However, if the target multiplier group MULG[2] is disabled as in the case of FIG. 3A, the multiplication results of multipliers MUL1 and MUL2 in that group are indeterminate in principle. As a result, zero is not necessarily stored in registers R3 and R4 assigned to those multipliers MUL1 and MUL2.

[0066] Therefore, the group controller 22b outputs, as part of the exchange setting signal SSW, position information of the registers that will result in a multiplication result of zero, in this example, information on registers R3 and R4, to the operation switch circuit SWM. Based on this information, the operation switch circuit SWM changes both inputs from registers R3 and R4 to inputs from the zero register RZ, as can be seen from a comparison of Figures 3A and 6.

[0067] <Major Effects of the Second Embodiment> As described above, the method of the second embodiment can also provide the same effects as those described in the first embodiment. Furthermore, the multiplication results from the disabled multiplier group MULG can be reliably set to zero before the adders ADD[1], ADD[2], ... perform addition. As a result, it is possible to reliably prevent erroneous product-sum operations.

[0068] (Embodiment 3) <Neural network engine details> Fig. 7 is a diagram showing a detailed configuration example and an operation example of the main part of the neural network engine in Fig. 1 in a semiconductor device according to embodiment 3. The configuration example shown in Fig. 7 differs from the configuration example shown in Fig. 3A in the following respect: In other words, group controller 22c determines whether or not each of n x m weight parameters Wa1 to Wa4, Wb1 to Wb4, ... is zero in step S102 of Fig. 4 by referring to the compressed weight parameters WP input to decompressor 23, rather than the output of decompressor 23.

[0069] Fig. 8 is a diagram for explaining an example of the processing contents of the decompressor in Fig. 7. Fig. 9 is a diagram showing a specific example of the processing contents in Fig. 8. First, as described in Fig. 1, the memory MEM1 stores compressed weight parameters WP as shown in Fig. 8 in advance. Then, the parameter DMA controller DMAC1 reads the compressed weight parameters WP and the header HD added thereto from the memory MEM1 as shown in Fig. 8, and outputs the compressed weight parameters WP therein to the decompressor 23.

[0070] As shown in Fig. 1, the header HD is output to the sequence controller 21 via a register REG. As shown in Fig. 8, the header HD includes, for example, a transfer source identifier ID1 and a transfer destination identifier ID2 used in the parameter switch circuit SWP. The sequence controller 21 determines the connection relationships within the switch circuit SWP based on the information in this header HD.

[0071] 8, the compressed weight parameters WP are composed of map data MPD of j bits, where j is an integer equal to or greater than 2, or 28 bits in this example, and a set of i weight parameters W1, W2, ..., Wi, where i is an integer equal to or greater than 2, or 11 weight parameters in this example. Each bit in the 28-bit map data MPD indicates whether a weight parameter is zero or non-zero. The 11 weight parameters W1, W2, ..., Wi are assigned in order to bits in the map data MPD that indicate non-zero. As a result, the decompressor 23 restores a minimum of 11 and a maximum of 28 weight parameters W1, W2, ..., Wx from the compressed weight parameters WP, which include the 11 weight parameters WP1, WP2, ..., WPi.

[0072] As a specific example, in the example of FIG. 9, the 28-bit map data MPD includes "00011000...", and the 11 weight parameters W include, in order, Wa1, Wa2, Wa3, Wb2, Wb3, Wb4.... The 28 bits in the map data MPD correspond to the 28 weight parameters W, respectively, and each of the 28 weight parameters W indicates whether it is zero, i.e., a zero weight parameter, or a non-zero weight parameter. In this example, the weight parameter W corresponding to a bit where the map data MPD is "1" is zero. The 11 weight parameters W are then assigned in order as weight parameters W corresponding to bits where the map data MPD is "0".

[0073] As a result, the decompressor 23 outputs the restored weight parameters Wa1, Wa2, Wa3, 0, 0, Wb2, Wb3, Wb4, ... in the order in which they are stored in the weight parameter buffer WBF in Fig. 3B, as shown in Fig. 9. By using this method, if all 28 bits of map data MPD in Fig. 8 are "1", the decompressor 23 outputs 28 zero weight parameters. On the other hand, if the 1st to 11th bits in map data MPD are all "0", the decompressor 23 outputs 11 non-zero weight parameters W1, W2, ..., W11.

[0074] 7, the group controller 22c determines whether or not each of the n×m weight parameters Wa1 to Wa4, Wb1 to Wb4, ... is zero by referring to the map data MPD in the compressed weight parameters WP input to the decompressor 23, as shown in Figures 8 and 9. In the example of Figure 9, the group controller 22c detects the positions of "1" bits and the number of "1" bits in the map data MPD.

[0075] <Major Effects of the Third Embodiment> As described above, the method of embodiment 3 also provides the same effects as those described in embodiment 1. Furthermore, unlike the method of embodiment 1, the use of map data MPD eliminates the need to provide a zero determination circuit that performs, for example, an 8-bit OR operation for each weight parameter W. This makes it possible to reduce the circuit area and shorten the time required for zero determination.

[0076] (Fourth embodiment) <Neural network engine details> 10 is a diagram showing a detailed configuration example and an operation example of the main part of the neural network engine in FIG. 1 in a semiconductor device according to a fourth embodiment. FIG. 11 is a schematic diagram showing a specific operation example of the group controller in FIG. 10. The configuration example shown in FIG. 10 differs from the configuration example shown in FIG. 3A in the following two points. The first difference is that a zero data detection circuit 32 that detects zero pixel data whose value is zero is provided between the data input DMA controller DMAC2i and the data input switch circuit SWDi. The second difference is that the group controller 22d performs replacement reflecting the detection result of the zero data detection circuit 32 in addition to the detection result of the zero weight parameter.

[0077] That is, the group controller 22d detects the position of zero pixel data from among the n×m pieces of pixel data Di transferred to the data input buffer IBF using the zero data detection circuit 32. Then, when the detected zero pixel data or the detected zero weight parameter is input, the group controller 22d performs rearrangement so that the multiplication results of the n multipliers MUL included in the target multiplier group MULG all become zero.

[0078] Specifically, the group controller 22d uses the data input switch circuit SWDi to rearrange the n×m pieces of pixel data Di transferred to the data input buffer IBF, and also uses the parameter switch circuit SWP to rearrange the n×m pieces of weight parameters W transferred to the weight parameter buffer WBF. Furthermore, the group controller 22d also rearranges the connections in the calculation switch circuit SWM based on these rearrangements.

[0079] As a specific example, in FIG. 11, the weight parameters Wa4 and Wb1 are both zero weight parameters, as in the first embodiment. Furthermore, in FIG. 11, the pixel data Di2 is zero pixel data. The zero data detection circuit 32 detects the position of the zero pixel data and outputs the position information to the group controller 22d. The group controller 22d detects the position of the multiplier MUL where at least one of the pixel data Di or the weight parameter W is zero, assuming that each multiplier MUL multiplies the same pixel data Di1 to Di4 by the weight parameters Wa1 to Wa4 and by the weight parameters Wb1 to Wb4.

[0080] 11, the position of the multiplier MUL detected in this manner is shown as a zero position detection result 33. Based on this zero position detection result 33, the group controller 22d performs rearrangement so that the multiplication results of the n multipliers MUL included in a certain target multiplier group MULG all become zero. In the example shown in FIG. 11, the group controller 22d interchanges the weight parameter Wa1 and the weight parameter Wa4 as shown by the reference numeral 35.

[0081] 10 shows the state after such a swap. That is, as indicated by reference numeral 35b, the group controller 22d swaps the weight parameter Wa1, which would normally be stored in memory area #1 in the weight parameter buffer WBF, with the weight parameter Wa4, which would normally be stored in memory area #4. Also, as indicated by reference numeral 35a, the group controller 22d swaps the pixel data Di1, which would normally be stored in memory area #1 in the data input buffer IBF, with the pixel data Di4, which would normally be stored in memory area #4.

[0082] Furthermore, as shown by reference numeral 35c, the group controller 22d switches the connection destination of the multiplier MUL1 in the multiplier group MULG[1] with the connection destination of the multiplier MUL2 in the multiplier group MULG[2]. However, in this example, the connection destination remains the same before and after the switching. The group controller 22d then controls both the multiplier groups MULG[1] and MULG[3] to be disabled using the enable signals EN[1] and EN[3].

[0083] <Major Effects of the Fourth Embodiment> As described above, the method of embodiment 4 can also achieve the same effects as those described in embodiment 1. Furthermore, by inputting not only zero weight parameters but also zero pixel data, and by performing rearrangement so that the multiplication result from a certain target multiplier group MULG becomes zero, it becomes possible to further increase the number of multiplier groups MULG that can be controlled to be disabled compared to embodiment 1. As a result, it becomes possible to further reduce power consumption.

[0084] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention. [Explanation of symbols]

[0085] 10 Semiconductor devices 15 Neural Network Engine (NNE) 16 System Bus 17 processors 20 MAC units 22 Group Controller 23 Decompressor 32 Zero data detection circuit ADD adder DMAC1, DMAC2 DMA controllers Di pixel data EN Enable signal IBF Data Input Buffer MPD map data MUL multiplier MULG Multiplier Group SWD, SWM, SWP switch circuit W weight parameter WBF Weight Parameter Buffer WP compressed weight parameters

Claims

1. A semiconductor device that executes neural network processing, one or more memories for storing weight parameters and pixel data; m multiplier groups, each including n multipliers, where n and m are integers equal to or greater than 2; a weight parameter buffer that outputs the n×m weight parameters stored therein to the n×m multipliers, respectively; a data input buffer that outputs the n×m pieces of pixel data held therein to the n×m pieces of multipliers, respectively; a first DMA (Direct Memory Access) controller for transferring the n×m weight parameters from the memory to the weight parameter buffer; a second DMA controller for transferring the n×m pixel data from the memory to the data input buffer; a group controller for controlling enable / disable for each of the m multiplier groups; Equipped with The group controller detecting a zero weight parameter having a value of zero from among the n×m weight parameters transferred to the weight parameter buffer, and swapping the n×m weight parameters transferred to the weight parameter buffer so that, upon input of the zero weight parameter, multiplication results of the n multipliers included in a target multiplier group that is one of the m multiplier groups all become zero; Disable the target multiplier group; permuting the n×m pieces of pixel data to be transferred to the data input buffer based on the permutation of the n×m weight parameters; Semiconductor device.

2. 2. The semiconductor device according to claim 1, further comprising: k adders, where k is an integer equal to or greater than 2, that add multiplication results output from the n×m multipliers; a switch circuit that determines a connection relationship between the n×m multipliers and the k adders based on a setting; Equipped with the group controller changes the setting of the switch circuit based on the replacement of the n×m weight parameters. Semiconductor device.

3. 2. The semiconductor device according to claim 1, the group controller receives the n×m weight parameters output from the first DMA controller and determines whether or not each of the n×m weight parameters received is zero; Semiconductor device.

4. 2. The semiconductor device according to claim 1, the memory stores, as compressed weight parameters, a set of j-bit map data, where j is an integer equal to or greater than 2, each bit of which represents whether a weight parameter is zero or non-zero, and a plurality of weight parameters which are assigned in order to the bits representing non-zero; the semiconductor device further includes a decompressor that restores the compressed weight parameters transferred by the first DMA controller to the n×m weight parameters; the group controller determines whether or not each of the n×m weight parameters is zero by referring to the map data in the compressed weight parameters input to the decompressor; Semiconductor device.

5. 2. The semiconductor device according to claim 1, The group controller further, detecting a position of zero pixel data having a value of zero from among the n×m pieces of pixel data transferred to the data input buffer, and rearranging the n×m pieces of pixel data transferred to the data input buffer and rearranging the n×m pieces of weight parameters transferred to the weight parameter buffer so that the multiplication results of the n multipliers included in the target multiplier group all become zero when the detected zero pixel data or the detected zero weight parameter is input; Disabling the target multiplier group; Semiconductor device.

6. A semiconductor device composed of one semiconductor chip, a neural network engine that executes neural network processing; one or more memories for storing weight parameters and pixel data; a processor; a bus connecting the neural network engine, the memory, and the processor to one another; Equipped with The neural network engine m multiplier groups, each including n multipliers, where n and m are integers equal to or greater than 2; a weight parameter buffer that outputs the n×m weight parameters stored therein to the n×m multipliers, respectively; a data input buffer that outputs the n×m pieces of pixel data held therein to the n×m pieces of multipliers, respectively; a first DMA (Direct Memory Access) controller for transferring the n×m weight parameters from the memory to the weight parameter buffer; a second DMA controller for transferring the n×m pixel data from the memory to the data input buffer; a group controller for controlling enable / disable for each of the m multiplier groups; Equipped with The group controller detecting a zero weight parameter having a value of zero from among the n×m weight parameters transferred to the weight parameter buffer, and swapping the n×m weight parameters transferred to the weight parameter buffer so that, upon input of the zero weight parameter, multiplication results of the n multipliers included in a target multiplier group that is one of the m multiplier groups all become zero; Disable the target multiplier group; permuting the n×m pieces of pixel data to be transferred to the data input buffer based on the permutation of the n×m weight parameters; Semiconductor device.

7. 7. The semiconductor device according to claim 6, further comprising: k adders, where k is an integer equal to or greater than 2, that add multiplication results output from the n×m multipliers; a switch circuit that determines a connection relationship between the n×m multipliers and the k adders based on a setting; Equipped with the group controller changes the setting of the switch circuit based on the replacement of the n×m weight parameters. Semiconductor device.

8. 7. The semiconductor device according to claim 6, the group controller receives the n×m weight parameters output from the first DMA controller and determines whether or not each of the n×m weight parameters received is zero; Semiconductor device.

9. 7. The semiconductor device according to claim 6, the memory stores, as compressed weight parameters, a set of j-bit map data, where j is an integer equal to or greater than 2, each bit of which represents whether a weight parameter is zero or non-zero, and a plurality of weight parameters which are assigned in order to the bits representing non-zero; the neural network engine further includes a decompressor that restores the compressed weight parameters transferred by the first DMA controller to the n×m weight parameters; the group controller determines whether or not each of the n×m weight parameters is zero by referring to the map data in the compressed weight parameters input to the decompressor; Semiconductor device.

10. 7. The semiconductor device according to claim 6, The group controller further, detecting a position of zero pixel data having a value of zero from among the n×m pieces of pixel data transferred to the data input buffer, and rearranging the n×m pieces of pixel data transferred to the data input buffer and rearranging the n×m pieces of weight parameters transferred to the weight parameter buffer so that the multiplication results of the n multipliers included in the target multiplier group all become zero when the detected zero pixel data or the detected zero weight parameter is input; Disabling the target multiplier group; Semiconductor device.

Images