SEMICONDUCTOR DEVICE

Abstract

A semiconductor device comprises a scratchpad memory, a memory controller, and a MAC (multiplication-accumulation) unit. The scratchpad memory is configured to store image data from N channels and includes M memory locations, each individually accessible, where M is an integer of at least 2 and N is an integer of at least 2. The memory controller manages access to the scratchpad memory such that pixel data from the N channels located at the same position in the image data of the N channels are stored in differential memories within the M memory locations. The MAC unit comprises a plurality of computational units to calculate pixel data from the N channels read from the scratchpad memory, using the memory controller and a weighting parameter.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The disclosure of Japanese Patent Application No. 2023-097830, filed on June 14, 2023, including the specification, drawings, and abstract, is incorporated herein by reference in its entirety. BACKGROUND

[0002] The present invention relates to a semiconductor device and, for example, to a semiconductor device for executing a neural network process.

[0003] Patent Document 1 discloses an image recognition device including an integration coefficient table generation device and an input pattern generation circuit for efficiently performing convolutional arithmetic operations in CNN (Convolutional Neural Network). The integration coefficient table generation device integrates two types of 3×3 input coefficient tables into one type of 5×5 integration coefficient table and outputs them to a 5×5 convolutional arithmetic operation circuit. The input pattern generation circuit generates 5×5 pixel values ​​from 3×3 pixel values ​​stored in the line buffer based on the rule set in the input pattern register and outputs the pixel values ​​to the 5×5 convolutional arithmetic operation circuit.

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2019-40403 SUMMARY

[0005] In a semiconductor device responsible for image processing such as CNN, for example, the computation of a plurality of channels in a convolutional layer is performed in parallel using a plurality of multiply-accumulate calculators included in a MAC (multiply-accumulate) unit, thereby realizing an improvement in performance, particularly a reduction in processing time. In this case, to further improve effective performance, it is desirable to reduce the input / output latency between the scratchpad memory (also referred to as SPM in the specification), in which the image data of a plurality of channels is stored, and the MAC unit.

[0006] Here, for example, to reduce the input / output latency between the SPM and MAC units, a method of using a dedicated data format in which image data of a plurality of channels is integrated for data transmission between the SPM and MAC units is conceivable. However, the image data of a plurality of channels stored in the SPM can be processed in a series of image processing by a general-purpose signal processing circuit such as a DSP (Digital Signal Processor) instead of a MAC unit. A general-purpose signal processing circuit cannot support the dedicated data format. Therefore, even if a dedicated data format is used to transmit data between the SPM and MAC units, there is a possibility that the processing times of the image processing cannot be sufficiently shortened.

[0007] In view of the foregoing, embodiments described below have been presented, and other problems and novel features will become apparent from the description of the present application and the accompanying drawings.

[0008] A semiconductor device according to one aspect includes a scratchpad memory, a memory controller, and a MAC (multiply-accumulate) unit. The scratchpad memory is configured to store image data of N channels and includes M individually accessible memories, where M is an integer of at least 2 and N is an integer of at least 2. The memory controller controls access to the scratchpad memory such that pixel data of the N channels located at the same position in image data of the N channels are respectively stored in difference memories in the M memories. The MAC unit includes a plurality of calculators for calculating pixel data of the N channels read from the scratchpad memory using the memory controller and a weighting parameter.

[0009] A semiconductor device according to another aspect includes a scratchpad memory, a memory controller, and a CPU (Central Processing Unit) and a MAC (Multiply Accumulate Unit). The scratchpad memory stores image data of N channels and includes M individually accessible memories, where M is an integer of 2 or more and N is an integer of 2 or more. The memory controller is configured to control access to the scratchpad memory based on a setting value of a register. The CPU is configured to determine the setting value of the register for the memory controller. The MAC unit includes a plurality of calculators.The CPU determines the setting value of the register such that pixel data of N channels located at a same pixel position in image data of the N channels are respectively stored in different memories of the M memories, and each of the calculators performs a multiplication-accumulation operation on the pixel data of the N channels read from the scratchpad memory using the memory control and a weighting parameter.

[0010] A semiconductor device according to yet another aspect comprises a scratchpad memory that stores D-dimensional data and includes M individually accessible memories, wherein the D-dimensional data is configured such that each data differs in one dimension by an index value, where D is an integer of 2 or more and M is an integer of 2 or more, and a memory controller is configured to control access to the scratchpad memory such that N data elements having the same index value in the first to (D-1)-th dimensions are each stored in different memories in the M memories, where the number of the index value in the D dimension is N.

[0011] Using a semiconductor device according to one or more embodiments, image processing times can be shortened. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram illustrating a schematic configuration of a semiconductor device according to a first embodiment. Fig. Figure 2 is a schematic diagram illustrating a configuration example of a neural network. Fig. 3 is a schematic diagram showing an example process for an intermediate layer in CNN in the Fig. 1 illustrated semiconductor device. Fig. 4 is a diagram for explaining an operation example of the memory controller in Fig. 1 and is a diagram illustrating an example of a memory map of image data of respective channels stored in a scratchpad memory (SPM). Fig. 5A shows a timing diagram showing a schematic operation example of the neural network engine (NNE) in Fig. 1 is compared with a case where the system of the first comparative example is used and a case where the system of the embodiment is used. Fig. 5B is a diagram illustrating an example of a case where the number of clock cycles required for the process of the convolution layer of one layer is compared between the case where the method of the first comparative example is used and the case where the method of the embodiment is used. Fig. 6 is a schematic diagram for explaining the influence on the input / output latency that occurs in the entire image processing using the neural network, under the premise that the Fig. 4 shown memory map is used. Fig. Figure 7 is a diagram showing various exemplary configurations of the scratchpad memory (SPM) in Fig. 1 illustrates. Fig. Figure 8 is a diagram illustrating an example of a memory map different from that shown in Fig. 4 illustrated, which differs from Example 2-2 in Fig. 7 corresponds. Fig. 9 is a block diagram showing a detailed configuration of a main part of the Fig. 1 illustrated semiconductor device. Fig. 10A is a schematic diagram showing a configuration example and a part of an operation example of the address router in Fig. 9 shows. Fig. 10B is a diagram showing an operating example of the Fig. 10A shown address router. Fig. 10C is a diagram showing another operating example than Fig. 10B illustrates. Fig. 10D is a diagram showing another operating example than Fig. 10B illustrates. Fig. 10E is a diagram showing another operating example than Fig. 10B illustrates. Fig. 11 is a schematic diagram showing a configuration example and a partial operation example of the data router for writing to Fig. 9 illustrates. Fig. 12 is a schematic diagram showing a configuration example of a data router for reading in Fig. 9 illustrates. Fig. 13 is a flowchart showing an example of the processing of the channel stride correction unit in Fig. 9 illustrates. Fig. 14 is a diagram illustrating a detailed configuration of a main part of the semiconductor device according to the second embodiment. Fig. 15 is a diagram illustrating a detailed configuration of a main part of the semiconductor device according to the third embodiment. Fig. 16 is a block diagram illustrating a detailed configuration example of a main part different from that of the Fig. 15 differs. Fig. 17 is a schematic diagram illustrating an example of a four-dimensional data format used in the semiconductor device according to the fourth embodiment. Fig. 18A is a diagram illustrating a specific embodiment of a D-dimensional format in the semiconductor device according to the fourth embodiment. Fig. Figure 18B is a schematic diagram illustrating an arrangement configuration of data accessed in parallel in a four-dimensional format based on the Fig. 18A shown format is to be accessed. Fig. 18C is a diagram showing the start address and end address of the respective Fig. 18B shows the data elements shown. Fig. 18D is a diagram illustrating an example of neural network software executed by the CPU based on the Fig. 18A is executed. Fig. 18E is a diagram illustrating an example of a memory map of the respective data stored in the scratchpad memory (SPM) after the stride correction is performed based on the Fig. 18C shown format. Fig. 19 is a diagram illustrating an example of a memory map of image data of respective channels stored in a scratch pad memory (SPM) in the semiconductor device as the first comparative example. Fig. Figure 20 is a schematic diagram for explaining the influence on the input / output latency that occurs in the entire image processing using the neural network, under the premise that the Fig. 19 shown memory map is used. Fig. 21 is a schematic diagram for explaining an influence on the input / output latency occurring in the entire image processing using the neural network in a semiconductor device as a second comparative example. Fig. 22 is a diagram showing an example of a memory map as a comparative example of the Fig. 8 illustrates. DETAILED DESCRIPTION

[0012] In the following embodiments, when necessary for convenience, description is made by dividing it into a plurality of sections or embodiments, but unless specifically stated, they are not independent of each other, and one refers to the modified example, detail, supplementary description, or the like of part or all of the other. In the following embodiments, the number of elements, etc. (including the number of elements, numerical values, quantities, ranges, etc.) is not limited to the specific number, but cannot be less than or equal to the specific number, except for cases where the number is specifically stated and is clearly limited to the specific number in principle.Furthermore, in the following embodiments, it is needless to say that the constituent elements (including element stages and the like) are not necessarily essential except for the case where they are specifically stated and the case where they are considered to be essential in principle. Similarly, in the following embodiments, with reference to the shapes, positional relationships, and the like of components and the like, it is assumed that the shapes and the like are substantially approximate or similar to the shapes and the like except for the case where they are specifically stated and the case where they are considered to be obvious in principle, and the like. The same applies to the above numerical values ​​and ranges.

[0013] In the following, embodiments will be described in detail with reference to the drawings. In all the drawings illustrating the embodiments, elements having the same functions are denoted by the same reference numerals, and repetitive descriptions thereof will be omitted. In the following embodiments, descriptions of the same or similar parts will not be repeated in principle unless particularly necessary. First embodiment

[0014] Fig. 1 is a schematic diagram illustrating an example of a configuration of a semiconductor device according to a first embodiment; Fig. The semiconductor device 10 illustrated in Figure 1 is, for example, a SoC (System on a Chip) or a microcontroller realized by a single semiconductor chip. The semiconductor device 10 includes a neural network engine (herein also referred to as NNE) 15, a scratchpad memory (SPM) 16, a DMA controller (DMAC) 17 (Direct Memory Access Controller), a DSP 18, a main memory 19, a CPU (Central Processing Unit) 20, and a system bus 21. The NNE 15, the DMAC 17, the DSP 18, the main memory 19, and the CPU 20 are connected to the system bus 21.

[0015] The NNE 15 executes a neural network process represented by a CNN. The SPM 16 includes M memories, MR[0] to MR[M-1], which can be accessed in parallel, where M is an integer of 2 or more. In the description, M memories, MR[0] to MR[M-1], are collectively referred to as memory MR. The memory MR is, for example, an SRAM. The SPM 16 is used as a high-speed cache memory of the NNE 15 and stores image data input to and output from the NNE 15. The SPM 16 is also accessible from the DSP 18. The DSP 18 is a general-purpose signal processing circuit and performs part of the neural network process on the image data DT stored, for example, in the SPM 16.

[0016] The main memory 19 is, for example, a DRAM. The main memory 19 stores image data DT, parameters PM, and the like used in the neural network process. The image data DT includes, for example, a camera image CIMG obtained from a camera and a feature map FM generated by the neural network process. The parameter PM includes a weighting parameter set WTS, which includes a plurality of weighting parameters WT according to a kernel size, and a bias parameter BS. The main memory 19 may be provided externally to the semiconductor device 10.

[0017] The DMAC 17 is the other of the general-purpose signal processing circuits and controls data transfer between the SPM 16 and the main memory 19 via the system bus 21. For example, the CPU 20 executes software (not shown) stored in the main memory 19 to cause the entire semiconductor device 10 to perform desired functions. As one of them, the CPU 20 constructs the neural network software system 40 by executing the neural network software. The neural network software system 40 performs, for example, various settings, startup control, and the like on the NNE 15, the DMAC 17, and the DSP 18 to control the operation sequence of the entire image processing including the neural network process.

[0018] Specifically, the NNE 15 includes a MAC unit 25, a post-processor 26, a line buffer 27, a write buffer 28, and a memory controller 29. The memory controller 29 includes a read access controller 30 and a write access controller 31. The read access controller 30 reads each pixel data PDi constituting the image data DT from the SPM 16 and stores pixel data PDi in the line buffer 27. The write access controller 31 writes pixel data PDo stored in the write buffer 28 to the SPM 16.

[0019] The MAC unit 25 includes i multiplication-accumulation calculators MAC[0] to MAC[i], where i is an integer of 2 or more. In the description, the i multiplication-accumulation calculators MAC[0] to MAC[i] are collectively referred to as multiplication-accumulation calculators MAC. The multiplication-accumulation calculator MAC performs multiplication-accumulation operations on each pixel data PDi stored in the line buffer 27 and each weight parameter WT input in advance. At this time, the MAC unit 25 reads the weight parameter set WTS from the main memory 19 in advance using a controller (not shown).

[0020] Further, the multiply-accumulate calculator MAC obtains the pixel data PDo through the multiply-accumulate operation of the pixel data PDi and the weight parameter WT and the like, and stores the pixel data PDo in the write buffer 28 via the post-processor 26. The post-processor 26 generates the pixel data PDo by performing an addition of the bias parameter BS, an operation of the activation function, a pooling process, or the like as needed to the multiply-accumulate operation performed by the multiply-accumulate calculator MAC.

[0021] Fig. Figure 2 is a schematic diagram illustrating a configuration example of a neural network. The neural network generally includes an input layer 45, a plurality of intermediate layers 46[1] to 46[j], and an output layer 47. The input layer 45 stores, for example, image data DT of three channels including R (red), G (green), and B (blue), i.e., camera image CIMG and the like.

[0022] In the intermediate layers 46[1] to 46[j], operation results obtained by multiplying and accumulating the previous layer and the weight parameter sets WTS1 to WTSj or the like are stored as the image data DT, that is, the feature maps FM1 to FMj. Each feature map FM, for example, FMj, has a size of Wj × Hj × Cj, where the size in the width direction or X direction is Wj, the size in the height direction or Y direction is Hj, and the number of channels is Cj.

[0023] The output layer 47 stores, as the feature map FMo, operation results obtained, for example, by a multiplication-accumulation operation of the last intermediate layer 46[j] and the weighting parameter set WTSo. The feature map FMo is, for example, 1 × 1 × Co in size, where the number of channels is Co. The feature map FMo is an image processing result using the neural network. The image processing result is typically stored in the main memory 19.

[0024] The Fig. 1 performs a neural network process in the following manner for the semiconductor device 10 illustrated in Fig. 2. (A) First, a camera input interface (not shown) in the semiconductor device 10 stores image data DT from an external camera, i.e., camera image CIMG, in the main memory 19. (B) Subsequently, the DMAC 17 stores the camera image CIMG in the SPM 16 by transferring the camera image CIMG stored in the main memory 19 to the SPM 16. As a result, the input layer 45 is formed on the SPM 16.

[0025] (C) Next, the NNE 15 or the DSP 18 performs an operation using the input layers 45 formed in the SPM 16 and the weight parameter set WTS1 stored in the main memory 19 as inputs, and stores the feature map FM1 in the SPM 16 as the operation result. As a result, the intermediate layer 46[1] is formed on the SPM 16. Whether the NNE 15 or the DSP 18 should be used to perform the operation is determined by the neural network software system 40. Such a determination applies similarly to the other layers.

[0026] (D) Subsequently, the NNE 15 or the DSP 18 performs an operation using the intermediate layer 46[1] formed on the SPM 16 and the weight parameter set WTS2 stored in the main memory 19 as inputs, and stores the feature map FM2 in the SPM 16 as an operation result. As a result, the intermediate layer 46[2] is formed on the SPM 16. By performing the same process thereafter, the final intermediate layers 46[j] are formed on the SPM 16.

[0027] (E) Next, the NNE 15 or the DSP 18 performs an operation using the last-stage intermediate layer 46[j] formed in the SPM 16, i.e., the feature map FMj and the weighting parameter set WTSo stored in the main memory 19, as inputs, and stores the operation result in the SPM 16 as the feature map FMo. As a result, the output layer 47 is formed on the SPM 16. (F) Finally, the DMAC 17 transfers the output layer 47 formed on the SPM 16, i.e., the feature map FMo as the image processing result, to the main memory 19.

[0028] Fig. 3 is a schematic diagram showing an example process for an intermediate layer in CNN in the Fig. 1. The intermediate layer to be processed, ie the convolutional layer, is assigned the feature maps FMi[0] to FMi[N i -1] of N iInput channels CHi [0] to CHi[N i -1] from the convolutional layer of the previous stage and the weighting parameter sets WTS[0] to WTS[N o -1] of N o Output channels assigned to the convolutional layer to be processed.

[0029] The feature mappings FMi[0] to FMi[N i -1] are stored in the SPM 16 and the weighting parameter sets WTS[0] to WTS[N o -1] are stored in the main memory 19. The feature map FM of each channel has a size of W × H, where W is the size in the width direction and H is the size in the height direction. Each of the weighting parameter sets WTS [0] to WTS[N o -1] includes N kw × N kh × N i Weighting parameters WTs, where N kw the number in the width direction is N kh is the number in the height direction and N i is the number of channels entered. N kw × Nkh is a kernel size, typically 3 × 3 or the like.

[0030] Here, in the embodiment of Fig. 3 the multiplication-accumulation calculator MAC[0] performs a multiplication-accumulation operation on the pixel data set PDS of a predetermined size based on a specific pixel position contained in the feature map FMi[0] to FMi[N i -1], and the weighting parameter set WTS[0] of the output channel CHo[0]. The addition of the bias parameter BS and the operation of the activation function are performed on the result of the multiplication-accumulation operation, so that the pixel data PDo of the pixel position, which serves as a reference in the feature map FMo[0] of the output channel CHo[0], is generated. Note that the pixel data set PDS N kw × N kh × N i Pixel data PDi in the same way as the weighting parameter set WTS.

[0031] In addition, the multiplication-accumulation calculator MAC[N o -1] parallel to the operation in the multiplication-accumulation calculator MAC[0], a multiplication-accumulation operation is performed on the same pixel data set PDS used by the multiplication-accumulation calculator MAC[0] and the weighting parameter set WTS[N o -1] of the output channel CHo[N o -1], which is different from the case of the multiplication-accumulation calculator MAC[0]. The addition of the bias parameter BS and the operation of the activation function are performed on the result of the multiplication-accumulation operation, so that the pixel data PDo of the pixel position used as a reference in the feature map FMo[N o -1] of the output channel CHo[N o -1] are generated.

[0032] Then, the process described above is performed while sequentially shifting the reference pixel position in the width direction or the height direction, so that all pixel data PDo containing the feature maps FMo[0] to FMo[N o -1] of N o output channels. The feature mappings FMo[0] to FMo[N o -1] of N o Output channels are stored in the SPM 16 as image data DT. The image data DT thus stored in the SPM 16 are input to, for example, the convolutional layer of the next stage and are stored as the feature maps FMi[0] to FMi[N i -1] the N i = N o Input channels are used. It should be noted that the convolution process as described in Fig. 3 is represented by expression (1). [Expression 1] [:1:]FMo: Feature mapping (output) FMi: Feature mapping (input) WT: weighting parameter [x]: Pixel position in X direction (horizontal) [y]: Line position in Y direction (vertical) [kw]: Core position in X-direction (horizontal) [kh]: Core position in Y direction (vertical) No: Number of output channels N i : Number of input channels N kw : Kernel size in X direction (horizontal) N kh : Kernel size in Y direction (vertical) W: Pixel size in X direction (horizontal) H: Row size in Y direction (vertical) Comparison examples

[0033] Fig. 19 is a diagram illustrating an example of a memory map of image data of respective channels stored in a scratchpad memory (SPM) in a semiconductor device as a first comparative example. In this case, the SPM 16 includes eight memories MR[0] to MR[7], and each memory MR has a bit width of 16 bytes (= 128 bits). That is, the memory bus width of the SPM 16 is 128 bytes (= 16-byte × 8).

[0034] Here, the size of the image data of one channel is 640 bytes, consisting of 64 bytes / line × 10 lines in a raster structure. In this case, if the image data DT of eight channels CH[0] to CH[7] are stored in the SPM 16 in raster order using a format such as one of the general-purpose formats, a memory map as shown in Fig. 19. For example, the plurality of pixel group data PGD0 to PGD3 in the respective channels corresponds to the data of the first row in the raster structure. The plurality of pixel group data PGD4 to PGD7 corresponds to the data of the second row in the raster structure. In this specification, the plurality of pixel group data PGD0 to PGD7 are collectively referred to as pixel group data PGD.

[0035] The image data DT of the eight channels CH[0] to CH[7] in Fig. 19 correspond, for example, to the feature maps FMi[0] to FMi[7] of the eight channels serving as the input data, or to the feature maps FMo[0] to FMo[7] of the eight channels serving as the output data in Fig. 3. The pixel group data PGD0 included in the image data DT of the channel CH[0] corresponds, for example, to a data group of 16 bytes in the width direction, for example, 16 pixels in the feature map FMi[0] included in Fig. 3. The subsequent pixel group data PGD1 corresponds to a data group of 16 bytes following the pixel group data PGDO in the width direction. As described above, the pixel group data PGD includes a plurality or single pixel data PDi and is the minimum data unit when the memory controller 29 accesses the SPM 16.

[0036] The pixel data set PDS, which is Fig. 3 corresponds to the data group sent from the line buffer 27 to the MAC unit 25 in Fig. 1 is output. Specifically, the line buffer 27 outputs, for example, a plurality of pixel data sets PDSs for a width-direction convolution operation, i.e., the shifted pixel data sets PDSs, to the MAC unit 25 in one clock cycle. The MAC unit 25 can also perform operations on the plurality of pixel data sets PDSs in parallel in one clock cycle.

[0037] The line buffer 27 sequentially switches the positions of the plurality of pixel data sets PDS output to the MAC unit 25 in the width direction or the height direction every clock cycle. When the positions are switched, a plurality of new pixel data PDi is required in addition to the pixel data PDi already acquired in the line buffer 27, in other words, in addition to the pixel data PDi that can be repeatedly used according to the convolution operation.

[0038] The read access controller 30 transfers the newly required plurality of pixel data PDi, for example, pixel group data PGD of eight channels CH[0] to CH[7], from the SPM 16 to the line buffer 27. With such a process, in the steady state, the data transfer from the SPM 16 to the line buffer 27, the data transfer from the line buffer 27 to the MAC unit 25, and the MAC operation in the MAC unit 25 are processed in a pipeline.

[0039] Here the logic address Laddr of the SPM 16 is Fig. 19 is expressed by expression (2) using the physical addresses of the M memories MR[0] to MR[M-1], for example, the word address WDaddr, and the index (idx) having a range from 0 to M-1 to identify the M memories MR[0] to MR[M-1]. Laddr=WDaddr×128+(idx×16)

[0040] Thus, in the memory map, which is shown in Fig. 19, N is an integer of 2 or more, and the pixel data PD of the N channels arranged at the same pixel position in the image data DT of the N channels or the pixel group data PGD are stored in the same memory MR in the M memories MR.

[0041] Therefore, as in Fig. As described in Figure 3, when the pixel group data PGD of N channels stored in the SPM 16 are input to the line buffer 27, they are read to the same memory MR in a time-division manner. Furthermore, when the pixel data PDo of N channels arranged at the same pixel position and obtained by a plurality of multiplication-accumulation calculators MAC or the like are output to the SPM 16, they must be written to the same memory MR in a time-division manner. Therefore, the input / output latency between the SPM 16 and the NNE 15, particularly the MAC unit 25, is increased, and there is a possibility that the processing times of the image processing cannot be sufficiently shortened.

[0042] For example, Resnet50, which is a well-known neural network model, has 50 layers. In the upstream intermediate layers, N i (Number of input channels) = 64, No (Number of output channels) = 256, W (X size) = 112, H (Y size) = 112, N kw (X-direction kernel size) = 1 and N kh (Y-direction kernel size) = 1. When applied to expression (1) above, 205,520,896 (= 64 × 256 × 112 × 112 × 1 × 1) multiply-accumulate operations are required. Therefore, it is desirable to increase the degree of parallelism of the multiply-accumulate calculators (MAC).

[0043] For example, the degree of parallelism in the input channel, output channel, X-direction pixel, Y-direction pixel, X-direction kernel, or Y-direction kernel depends on the architecture. However, considering that raster processing is a general hardware processing, increasing the degree of parallelism in the channel direction is an important hardware requirement. Especially when increasing the degree of parallelism in the channel direction, since the input / output latency between the SPM 16 and the NNE 15 greatly affects the effective performance, a technique for reducing the input / output latency is required.

[0044] Fig. Figure 20 is a schematic diagram for explaining the influence of input / output latency occurring in the entire image processing using the neural network, under the premise that the Fig. 19 shown memory map is used. Fig. 20 shows a schematic process of the semiconductor device in which the Fig. 2, the neural network with five intermediate layers is a process target. In this case, the semiconductor device sequentially executes DMAC process[1], NNE process[1], NNE process[2], DSP process, NNE process[3] to NNE process[5], and DMAC process[2]. DMAC process[1] is a process related to the input layer, NNE process[1] to NNE process[4] and the DSP process are processes related to the intermediate layers, and NNE process[5] and DMAC process[2] are processes related to the output layer.

[0045] In the DMAC process[1], the DMAC 17 transmits the camera image CIMG stored in the main memory 19 to the SPM 16. In the NNE process[1], the NNE 15 receives the camera image CIMG stored in the SPM 16 and performs signal processing, thereby generating the feature map FM1 and outputting the feature map FM1 to the SPM 16. In the NNE process[2], the NNE 15 receives the feature map FM1 stored in the SPM 16 and performs signal processing, thereby generating the feature map FM2 and outputting the feature map FM2 to the SPM 16.

[0046] In the DSP process, the DSP 18 receives the feature map FM2 stored in the SPM 16 and performs signal processing, thereby generating the feature map FM3, and outputs the feature map FM3 to the SPM 16. From the NNE process [3] to the NNE process [5], the NNE 15 receives the feature map FM of the intermediate layer in the previous stage stored in the SPM 16 and performs signal processing, thereby generating the feature map FM of the intermediate layer in the subsequent stage, and outputs the generated feature map FM to the SPM 16. In the DMAC process [2], the DMAC 17 transfers the output layer feature map FMo stored in the SPM 16 to the main memory 19.

[0047] If such processes are carried out which are Fig. When the memory map shown in Figure 19 is used, the input / output latency increases with respect to the input / output from the NNE process [1] to the NNE process [5]. Furthermore, in the DSP process, the input / output latency increases depending on the content of the process. As a result, the effective performance relative to the theoretical performance is greatly reduced, and the processing time of image processing using the neural network may be increased.

[0048] Fig. Fig. 21 is a schematic diagram for explaining an influence of input / output latency occurring in an entire image processing using a neural network in a semiconductor device as a second comparative example. Fig. 20, it is conceivable to apply a dedicated data format, for example by integrating a plurality of channels, for the data transmission between the SPM 16 and the NNE 15. In particular, as in Fig. 21, the dedicated data format is applied to the feature maps FM1, FM4 and FM5 stored in the SPM 16.

[0049] However, general-purpose signal processing circuits such as the DSP 18 and the DMAC 17 cannot support the dedicated data format and must use a general-purpose format such as the one in Fig. 19. Therefore, as shown in Fig. As shown in Figure 21, even when the dedicated data format is used, the input / output latency is still increased at the time of input to the NNE process [1] and the NNE process [3] and at the time of output to the NNE process [2] and the NNE process [5]. Consequently, there is a possibility that the processing time of image processing cannot be sufficiently shortened. Memory control

[0050] Accordingly, the Fig. 1 shown memory controller 29 as follows. Fig. 4 is a diagram for explaining an operation example of the memory controller in Fig. 1 is a diagram illustrating an example of a memory map of image data of respective channels stored in the scratchpad memory (SPM). The memory controller 29 controls access to the SPM 16 such that the pixel group data PGD of the N channels located at the same pixel position, and thus the pixel data PD in the image data DT of the N channels, are respectively stored in different memories MR among the M memories MR[0] to MR[M-1].

[0051] In the case of Fig. 4, as in the case of Fig. 19, the SPM 16 has eight memories MR[0] to MR[7], and each memory MR has a bit width of 16 bytes (= 128 bits). In addition, as shown in Fig. 19, the size of the image data DT of one channel is 640 bytes (= 64 bytes / line × 10 lines). The SPM 16 stores image data DT of eight channels CH[0] to CH[7] according to the planar format.

[0052] In the case of Fig. 4 is, however, unlike in the case of Fig. 19, a blank area BLNK is provided between the storage area of ​​the last pixel group data PGD39 in one channel and the storage area of ​​the first pixel group data PGDO in another channel in two adjacent channels. In this case, the size of the blank area BLNK is 16 bytes. The memory controller 29 defines, for example, a start address for each of the eight channels CH[0] to CH[7], that is, an address of a storage area of ​​the first pixel group data PGDO, so that such a blank area BLNK is provided.

[0053] Thus, for example, the first pixel group data PGDO in the eight channels CH[0] to CH[7] is stored in the eight memories MR[0] to MR[7]. The same applies to the remaining pixel group data PGD; for example, the last pixel group data PGD39 in the eight channels CH[0], CH[1] to CH[7] is stored in the memories MR[7], MR[0] to MR[6], respectively.

[0054] Fig. 5A shows a timing diagram showing a schematic operation example of the neural network engine (NNE) in Fig. 1 is compared with a case where the system of the first comparative example is used and a case where the system of the embodiment is used. Fig. 5B is a diagram illustrating an example of a case where the number of clock cycles required for the process of the convolution layer of one layer is compared between the case where the method of the first comparative example is used and the case where the method of the embodiment is used.

[0055] The neural network engine (NNE) 15 repeatedly executes a process cycle Tcyc as shown in Fig. 5A, for example, to process a layer of convolutional layers. In this case, if the first comparative example, that is, the memory map as shown in Fig. 19, the pixel group data PGD at the same pixel position in each channel must be read from the same memory MR and written to the same memory MR after the multiplication-accumulation operation by the multiplication-accumulation calculators MAC. Therefore, the SPM 16 must perform a time-division multiplexed read operation and a time-division multiplexed write operation while providing a waiting time Tw each time a read target or write target channel is switched.

[0056] On the other hand, if the embodiment, that is, the memory map as shown in Fig. 4, the pixel group data PGD at the same pixel position in each channel can be read from different memories MR and written to different memories MR after the multiplication-accumulation operation by the multiplication-accumulation calculators MAC. Therefore, unlike the first comparative example, there is no need to provide the waiting time Tw. In other words, the pixel group data PGD at the same pixel position can be simultaneously read from and written to the memories MR that are different from each other. Consequently, the input / output latency can be shortened.

[0057] Here, as a specific example, it is assumed that MAC unit 25 can process 32 input channels and 32 output channels in one clock cycle and can process two pixels in the X direction within the clock cycle, that is, it can perform 2048 (= 32 × 32 × 2) convolution operations in parallel in one clock cycle. The latency excluding the input and output of the channel is assumed to be 50 clock cycles. Fig. 5B, the number of clock cycles required for the process of one convolutional layer is compared between the first comparative example and the embodiment under the premise that the MAC unit 25 is used.

[0058] For example, in a typical CNN network model such as Resnet50, the X / Y size of the image data DT is large in the upstream convolutional layers. As processing proceeds in the downstream layers, the number of channels of the image data DT increases, while the X / Y size of the image data DT decreases. Fig. Example 1 shown in Figure 5B, which is assumed to be the case of the upstream layer, W (X-size) = 112, H (Y-size) = 112, N i (Number of input channels) = 64, N o (Number of output channels) = 256, N kw (Kernel X size) = 1 and N kh (Kernel Y-size) = 1. In Example 2, which is assumed to be the case of the downstream layer, W = 7, H = 7 and N i = 512, N o = 2048, N kw = 1, N kh = 1 is used.

[0059] Furthermore, Fig. 5B, the theoretical power TP is calculated by Expression (3). The effective power AP_C according to the method of the first comparative example is calculated by Expression (4). The effective power AP_E according to the method of the embodiment is calculated by Expression (5). Note that CEIL() is a function that rounds the value in parentheses to an integer. TP=(Ni / 32)×(No / 32)×CEIL(W / 2)×H AP_C=(Ni / 32)×(No / 32)×{(CEIL(W / 2)×H)+50+(32−1)+(32−1)} AP_E=(Ni / 32)×(No / 32)×{(CEIL(W / 2)×H)+50}

[0060] Here, the MAC unit 25 executes a processing cycle Tcyc, which is Fig. 5A and Fig. 5B, performs 2048 convolution operations within one clock cycle and processes the image data DT of 32 input channels and the image data DT of 32 output channels using CEIL(W / 2) × H clock cycles. For example, since in Example 1 N i (Number of input channels) = 64 and No (number of output channels) = 256, the number of process cycles Tcyc required to process the image data DT of all input channels and the image data DT of all output channels is 16(= 64 / 32 × 256 / 32).

[0061] In expressions (4) and (5) an overhead of (N i / 32) × (N o / 32) × {(32-1) + (32-1)} is added to the effective power AP_C according to the method of the first comparative example compared with the effective power AP_E according to the method of the embodiment. That is, in Fig. 5A, the overhead associated with 32 channels 1 clock cycle is added at the time of input and output for each processing cycle Tcyc, with the waiting time Tw set as one clock cycle. The number of processing cycles Tcyc required to process the convolutional layers increases as the number of channels increases. Therefore, particularly in the method of the first comparative example, the deviation of the effective power AP_C with respect to the theoretical power TP becomes larger in the downstream layers. When the method of the embodiment is used, this deviation can be suppressed. Specifically, an improvement effect of 44.3% (= 1-(79,872 / 143,360)) can be obtained in the downstream layer.

[0062] Fig. 6 is a schematic diagram for explaining the influence of the input / output latency that occurs in the entire image processing using the neural network, under the premise that the Fig. 4 is used. Using the method of the embodiment, unlike in Fig. 20 or Fig. 21, as in Fig. As shown in Figure 6, the input / output latency from the NNE process[1] to the NNE process[5] and the entire input / output associated with the DSP process can be reduced. In this case, since the planar format, which is a general-purpose format, is used instead of the dedicated data format, the input / output latency can be shortened even when the DSP process is specifically included. Configuration of SPM and empty space

[0063] Fig. Figure 7 is a diagram showing various exemplary configurations of the scratchpad memory (SPM) in Fig. 1. The SPM 16 includes, for example, a 2 m (= M) Number of memories MR, where m is an integer of 1 or more. The bit width of the respective memory MR is, for example, 2 k Bytes, where k is an integer of 0 or more. Here, the size FS of the image data for each channel is assumed to be GS × even number, where GS is the size of the pixel group data PGD.

[0064] The Fig. 7 shown examples 1-1 to 1-5 show examples in which 32 (= 2 5 ) Memory MR, each having a bit width of 16 bytes, are provided to configure the SPM 16 with a memory bus width of 512 bytes. In Example 1-1, a size GS of the pixel group data PGD is 16 bytes, and one pixel group data PGD is stored in one memory MR. In this case, if the same procedure as in the case of Fig. 4 is used, the number of input channels or the number of output channels capable of parallel processing is 32 (= 512 / 16) .

[0065] In addition, the size of the respective blank areas BLNK can be 16 bytes × (a number prime to 32 channels), that is, 16 bytes × odd number, in units of 16 bytes, which is the size GS of the pixel group data PGD. In addition, from the perspective of making the size of the blank area BLNK as small as possible, that is, saving memory, it is desirable that the size of the blank area BLNK be 16 bytes × 1.

[0066] In Example 1-2, a size GS of the pixel group data PGD is 32 bytes, and one pixel group data PGD is stored in two memories MR. In this case, if the same procedure as in the case of Fig. 4 is used, the number of input channels or the number of output channels in which parallel processing can be performed is 16 (= 512 / 32). In addition, the size of the respective blank areas BLNK may be 32 bytes × (a number that is prime with respect to 16, which is the number of channels), that is, 32 bytes × odd number, in units of 32 bytes, which is the size GS of the pixel group data PGD, and preferably 32 bytes × 1 thereof.

[0067] In Example 1-3, a size GS of the pixel group data PGD is 64 bytes, and one pixel group data PGD is stored in four memories MR. In this case, if the same procedure as in the case of Fig. 4 is used, the number of input channels or the number of output channels in which parallel processing can be performed is 8 (= 512 / 64). Furthermore, the size of the respective blank areas BLNK may be 64 bytes × (a number prime to 8, which is the number of channels), that is, 64 bytes × odd number, in units of 64 bytes, which is the size GS of the pixel group data PGD, and preferably 64 bytes × 1 thereof.

[0068] Similarly, in Examples 1-4, the size GS is 128 bytes, and the number of input / output channels that can be processed in parallel is 4 (= 512 / 128). The size of the respective blank areas BLNK can be 128 bytes × (a number prime to 4, which is the number of channels), that is, 128 bytes × an odd number, and preferably 128 bytes × 1 of them. In Examples 1-5, the size GS is 256 bytes, and the number of input / output channels that can be processed in parallel is 2 (= 512 / 256). The size of the blank area BLNK can be 256-byte × (a number prime to 2, which is the number of channels), that is, 128-byte × odd number, and consequently 128-byte × 1.

[0069] Examples 2-1 to 2-3 in Fig. 7 show an example in which 8 (= 2 3) Memory MRs, which have a bit width of 16 bytes, are provided to configure the SPM 16 with a memory bus width of 128 bytes. In Example 2-1, a size GS of the pixel group data PGD is 16 bytes, and one pixel group data PGD is stored in one memory MR. That is, Example 2-1 corresponds to the configuration shown in Fig. 4. The number of input channels or output channels that can be processed in parallel is 8 (= 128 / 16). In addition, the size of the respective blank areas BLNK can be 16 bytes × (the number prime to 8, which is the number of channels), that is, 16 bytes × odd number, and preferably 16 bytes × 1 of them.

[0070] Similarly, in Example 2-2, the size GS is 32 bytes, and the number of input / output channels that can be processed in parallel is 4 (= 128 / 32). The size of the blank area BLNK can be 32 bytes × (a number prime to 4, which is the number of channels), i.e., 32 bytes × odd number, and preferably 32 bytes × 1 thereof. In Example 2-3, the size GS is 64 bytes, and the number of input / output channels that can be processed in parallel is 2 (= 128 / 64). The size of the blank area BLNK can be 64 bytes × (a number prime to 2, which is the number of channels), i.e., 64 bytes × odd number, and thus 64 bytes × 1.

[0071] If the above is generalized, assuming that the SPM is 16 M (= 2 m ) memory MR, which consists of K (= 2 k ) bytes, the size GS of the pixel group data PGD as 2 (k+a)bytes, and the number N of channels processed in parallel is defined as 2 (m-a) In addition, the empty area BLNK is defined as 2 (k+a) -byte × (the number that is related to 2 (m-a) prime, which is the number of channels). Note that a is an integer of 0 or more and less than m. Further, the generalized logical address Laddr of the SPM 16 is given by Expression (6). As in Expression (2), WDaddr is the word address of each memory MR, and idx is the identification number of each memory MR. Laddr=WDaddr×(M×K)+(idx×K)

[0072] Referring to Example 2-1 of Fig. 7 and Fig. 4 is K (= 2 k ) for example 16 (= 2 4 ) bytes, M (= 2 m ) is 8 (= 2 3 ), GS is 16 (= 2 (4+0) ) bytes, and N is 8 (= 2 (3-0) ). In addition, the empty area BLNK is set to 16 (= 2 (4+0) × 1) is set.

[0073] Fig. Figure 8 is a diagram illustrating an example of a memory map different from that shown in Fig. 4 illustrated, which differs from Example 2-2 in Fig. 7 corresponds. Fig. 22 is a diagram showing an example of a memory map as a comparative example of the Fig. 8. In the Fig. 8 and Fig. In the cases shown in Figure 22, the SPM 16 has eight memories MR[0] through MR[7]. Each memory MR has a bit width of 16 bytes. This means that the memory bus width of the SPM 16 is 128 bytes (= 16 bytes × 8). The size GS of the pixel group data PGD is 32 bytes.

[0074] Here, the size of the image data of one channel is 768 bytes, consisting of 96 bytes / line × 8 lines in a raster structure. In this case, when the method of the comparative example is used, the memory map is as shown in Fig. 22 shown. In Fig. 22, the plurality of pixel group data PGD0a, PGD0b, PGD1a, PGD1b, PGD2a, PGD2b correspond to the data of the first row in the raster structure, and the plurality of pixel group data PGD3a, PGD3b, PGD4a, PGD4b, PGD5a, PGD5b correspond to the data of the second row in the raster structure. The number of channels of the image data is 4.

[0075] As in Fig. 22, in the method of the comparative example, as in the case of Fig. 19, the pixel group data PGD of the four channels CH[0] to CH[3] located at the same pixel position, for example, the pixel group data PGD0a, PGD0b of 32 bytes, are stored in the same memory pair (MR[0], MR[1]). On the other hand, in the method of the embodiment, as shown in Fig. 8, a 32-byte blank area BLNK is provided. Thus, the pixel group data PGD of the four channels CH[0] to CH[3] located at the same pixel position, for example, the pixel group data PGD0a, PGD0b of 32 bytes, are stored in memory pairs (MR[0], MR[1]), (MR[2], MR[3]), (MR[4], MR[5]), (MR[6], MR[7]) that are different from each other. Details of the main part in the semiconductor device

[0076] Fig. 9 is a block diagram showing a detailed configuration of a main part of the Fig. 1 illustrated semiconductor device. Fig. 9 mainly shows a detailed configuration example of the memory controller 29 in Fig. 1 and a detailed configuration example of the neural network software system 40. As in Fig. 9, the SPM 16 may, for example, include an arbitration circuit that arbitrates a plurality of accesses to the same MR.

[0077] In addition, the Fig. For example, the activation function calculation unit 70 illustrated in Figure 9 performs an addition of the bias parameter BS and a calculation of the activation function on the multiplication-accumulation operation result from the MAC unit 25. The pooling processing unit 71 performs pooling processing as needed. The activation function calculation unit 70 and the pooling processing unit 71 are arranged in the post-processor 26. Fig. 1. Details of the memory controller 29 and the neural network software system 40 are described below. Details of the memory control

[0078] In Fig. 9, the read access control 30a and the write access control 31a in the memory controller 29 are in Fig. 1. The read access controller 30a includes a read base address register 50, a channel stride register 51, an address counter 52, an adder 53, a read address generator 54, a read address router 55, a read data router 56, and a pending address buffer 57.

[0079] In general, the read access controller 30a generates read logic addresses of N channels in parallel, in which pixel data PD of N channels are stored, when the pixel group data PGD of the N channels, and thus the pixel data PD, are read from the SPM 16. Furthermore, the read access controller 30a translates the generated read logic addresses of the N channels into physical read addresses of the M memory MRs in parallel and outputs the read logic addresses to the SPM 16. Furthermore, the read access controller 30a rearranges the pixel data PD of the N channels read from the SPM 16 in the order of the channels and outputs them in parallel to the MAC unit 25.

[0080] Specifically, in the read base address register 50 for reading, the start address for reading the image data DT of the N channels stored in the SPM 16 is set as the base address. For example, in Fig. 4, the starting address of the channel CH[0] is set. Typically, a read address space used for input to the MAC unit 25 and a write address space used for output from the MAC unit 25 are set individually in the SPM 16. The read base address register 50 defines the position of the read address space in the SPM 16.

[0081] The address counter 52 generates a sampling address Saddr by sequentially counting from 0 in unit of the size GS of the pixel group data PGD. In the case of Fig. 4, 16 bytes are used as units. The adder 53 generates the reference logic address Raddr by adding the base address from the read base address register 50 and the scan address Saddr from the address counter 52. As a result, in the read address space, for example, the reference logic address Raddr is generated such that the pixel group data PGD of the channel CH[0] in Fig. 4 are scanned sequentially with PGD0, PGD1,..., PGD7, PGD8,...

[0082] In the channel stride register 51 for reading, the address distance between adjacent channels of the respective start addresses of the image data DT of the N channels stored in the SPM 16 is set as a channel stride. For example, in Fig. 4 the address distance between the logical address of the pixel group data PGDO in the channel CH[0] and the logical address Laddr of the pixel group data PGDO in the channel CH[1], in particular 640 + 16 bytes, is set.

[0083] The read address generator 54 adds an integer multiple of the address spacing set in the channel stride register 51 to the reference logic address Raddr input from the adder 53, thereby generating the read logic addresses CH[n]_REaddr of the N channels in parallel, in other words, in the same clock cycle. That is, the read address generator 54 generates the read logic addresses CH[0]_REaddr-CH[7]_REaddr of the N channels, in the case of Fig. 4 8 channels, based on Expression (7). In Expression (7), CHstride is an address distance set in the channel stride register 51, and n is an integer from 0 to N-1. CH[n]_REaddr=Raddr+CHstride×n

[0084] In particular, when N = 8, CHstride = 656 and Raddr = 0, the read address generator generates 54 CH[0]_REaddr = 0, CH[1]_REaddr = 656, ..., CH[7]_REaddr = 4592 in parallel. Thus, in Fig. 4 the logic addresses Laddr of the pixel group data PGDO are generated in parallel in the eight channels CH[0] to CH[7].

[0085] The read address router 55 translates the read logical addresses CH[n]_REaddr of the N channels, generated in parallel by the read address generator 54, into the physical read address MR_idx[n]_REaddr for the memory MR corresponding to each channel in the M memories MRs. Details of the read address router 55 will be described later.

[0086] In response to the physical read address MR_idx[n]_REaddr from the read address router 55, the read data router 56 rearranges the pixel data PD of the N channels read from the memory MR corresponding to each channel, detailing the memory read data MR_idx[n]_REdat arranged in the memory order, in the channel order.

[0087] The read data router 56 outputs the channel read data CH[n]_REdat obtained by the rearrangement in parallel to the MAC unit 25. Specifically, the read data router 56 stores the channel read data CH[n]_REdat in parallel in the line buffer 27, which has the storage area in the order of the channels, and outputs the data to the MAC unit 25 via the line buffer 27. Details of the read data router 56 and the pending address buffer 57 will be described later.

[0088] The write access controller 31a includes a write base address register 60, a channel stride register 61, an address counter 62, an adder 63, a write address generator 64, a write address router 65, and a write data router 66. The operations of the write base address register 60, the channel stride register 61, the address counter 62, the adder 63, and the write address generator 64 are the same as those of the read base address register 50, the channel stride register 51, the address counter 52, the adder 53, and the read address generator 54 described above.

[0089] Thus, the write access controller 31a generates write logic addresses of N channels in parallel for storing pixel data PD of N channels, respectively, when the pixel group data PGD of N channels obtained based on the multiply-accumulate operation performed by the MAC unit 25, and thus the pixel data PD, are written into the SPM 16. In addition, the write access controller 31a translates the generated write logic addresses of the N channels in parallel to the physical write addresses of the corresponding memory MR for each channel. Then, the write access controller 31a outputs the physical write addresses to the memory MR corresponding to each channel in parallel, along with the pixel data PD of the N channels obtained based on the multiply-accumulate operation performed by the MAC unit 25.

[0090] At this time, the write address router 65 translates the write logic address CH[n]_WRaddr of the N channels, generated in parallel by the write address generator 64, into the physical write address MR_idx[n]_WRaddr for the memory MR corresponding to each channel in the M memories MRs. Then, the write address router 65 outputs the translated physical write address MR_idx[n]_WRaddr to the corresponding memory MR for each channel in parallel.

[0091] Details of the write address router 65 will be described later.

[0092] On the other hand, the write data router 66 outputs the pixel data PD of the N channels obtained based on the multiplication-accumulation operation performed by the MAC unit 25, specifically the channel write data CH[n]_WRdat stored in the write buffer 28, which has the storage area in the order of the channels, in parallel to the corresponding memory MR for each channel. At this time, the write data router 66 rearranges the channel write data CH[n]_WRdat arranged in the channel order into the storage order. Then, the write data router 66 outputs the memory write data MR_idx[n]_WRdat obtained by the rearrangement to the memory MR corresponding to each channel. Details of the write data router 66 will be described later. Address router details

[0093] Fig. 10A is a schematic diagram showing a configuration example of an address router and a partial operation example of the Fig. 9 shown address router. From Fig. 10B to Fig. 10E are diagrams for describing the operations of the Fig. 10A. The address router shown in FIG. 10A corresponds to the read address router 55 or the write address router 65. Here, it is assumed that the SPM 16 includes eight memories MR[0] to MR[7], each having 64k bytes. In this configuration, the SPM 16 has a capacity of 512k bytes. It is assumed that the respective memories MR have 4096 word addresses WDaddr and a bit width of 16 bytes. That is, it is assumed, for example, that WDaddr of word addresses in Example 2-1 shown in Fig. 7 and Fig. 4 is 4096.

[0094] The address router receives the logical addresses CH[0]_addr[18:4] to CH[7]_addr[18:4] of the eight channels CH[0] to CH[7] in parallel from the address generators 54, 64. In the case of the read address router 55, the logical address CH[n]_addr corresponds to the leading logical address CH[n]_REaddr. In the case of the write address router 65, the logical address CH[n]_addr corresponds to the write logical address CH[n]_WRaddr.

[0095] The address router outputs the physical addresses MR_idx[0]_addr[11:0] to MR_idx[7]_addr[11:0] of the eight memories MR[0] to MR[7] in parallel. In the case of the read address router 55, the physical address MR_idx[n]_addr corresponds to the physical read address MR_idx[n]_REaddr. In the case of the write address router 65, the physical address MR_idx[n]_addr corresponds to the physical write address MR_idx[n]_WRaddr. Specifically, the physical address MR_idx[n]_addr corresponds, for example, to the word address WDaddr in Fig. 4.

[0096] Since access is performed in units of 16 bytes in this case, the lower 4 bits ([3:0]) in the logical address CH[n]_addr are set to 0. Accordingly, the fourth to eighteenth bits ([18:4]) of the logical address CH[n]_addr are input to the address router. The eight memories MR[0] to MR[7] correspond to the eighth index idx[0] to idx[7], respectively. The eight indexes idx[0] to idx[7] are assigned to the fourth to sixth bits ([6:4]), which are the low-order page bits, in the logical address CH[n]_addr. Thus, the logical address CH[n]_addr[6:4] can identify the correspondence between the channels and the memory MR.

[0097] Here, for example, a read operation from the SPM 16 is assumed at a certain process cycle Tcyc[t]. At this time, the logical addresses CH[0]_addr[18:4] to CH[7]_addr[18:4] of the eight channels input in parallel to the address router differ from each other in the fourth to sixth bits ([6:4]) depending on the operation of the read address generator 54 based on the channel stride register 51. The address router identifies the memory MR corresponding to each channel by the logical addresses CH[0]_addr-CH[7]_addr of the eight channels, which are a specific bit range, in this case from the fourth bit to the sixth bit ([6:4]).

[0098] In the Fig. 10A, the logical address CH[n]_addr[6:4] of the channel CH[n] represents the index idx[5], [2], [7], [4], [1], [6], [3], [0], respectively, and thus the memory MR[5], [2], [7], [4], [1], [6], [3] and [0], when n = 0, 1, 2, 3, 4, 5, 6, 7. That is, here, unlike in the case of Fig. 4, a case is shown where the size of the image data DT of one channel is (8p + 4) × 16 bytes, where p is an integer. That is, in Fig. 4, the respective channels CH[0] to CH[7] are configured, for example, by pixel group data PGD0-PGD43. Fig. 4, the size of the image data DT of one channel is (8p + 0) × 16 bytes.

[0099] Alternatively, Fig. 10A illustrates the case where the image data size DT of one channel is (8p + 5) × 16 bytes and the channel stride (CHstride) is also set to (8p + 5) × 16 bytes. The blank area BLNK is not provided in this case. That is, if the image data size DT of one channel is (8p + odd number) × 16 bytes, the blank area BLNK is not required.

[0100] The address router identifies a memory MR[q] as an output destination of the bit range from the 7th bit to the 18th bit ([18:7]), which are bits on the higher side of the logical address CH[n]_addr[18:4], by using the index value (idx) specified by the 4th to 6th bits ([6:4]) of the logical address CH[n]_addr. Then, the address router outputs the logical address CH[n]_addr[18:7] of the channel CH[n], which is the bit range to be output, to the memory MR[q] identified by the index value (idx) as the 12-bit physical address MR_idx[q]_addr[11:0].

[0101] Such an operation is carried out in Fig. 10A, the logical address CH[0]_addr[18:7] of the channel CH[0] is output to the memory MR[5] as the physical address MR_idx[5]_addr[11:0] because the index value (idx) is 5. The logical address CH[1]_addr[18:7] of the channel CH[1] is output to the memory MR[2] as the physical address MR_idx[2]_addr[11:0] because the index value (idx) is 2. The same applies to the remaining channels CH[2] to CH[7].

[0102] In the read operation from the SPM 16 in the subsequent process cycle Tcyc[t+1], the index value (idx) of the fourth to sixth bits ([6:4]) is incremented by +1 by the operation of the address counter 52. Consequently, the output destinations of the logical address CH[n]_addr[18:7] of the channels [0],

[0103] [1], [2], [3], [4], [5], [6] and [7] are switched to the memory MR[6], [3], [0], [5], [2], [7], [4] and [1] respectively. That is, with reference to Fig. 4 For example, in a certain processing cycle, the pixel group data PGD of the channel CH[0] is read from the word address “A” of the memory MR[5], and in the next processing cycle, the next pixel group data PGD of the channel CH[0] is read from the same word address “A” of the other memory MR[6].

[0104] Specifically, the address router includes, for example, a selector, a matrix switch, or the like, which determines a connection relationship between the input signals of the N channels and the output signals to the M MR memories. In this example, the input signal is the logical address CH[n]_addr[18:7] of eight channels, which is the bit range of the logical address. The output is the physical address MR_idx[n]_addr[11:0] to the eight MR memories.

[0105] The logical address CH[n]_addr[6:4] of the eight channels, which is the other bit range of the logical address, is used as a select signal of the selector or matrix switch. With this configuration, the address router can process the input signals of the N channels in parallel, in other words, in one clock cycle, and output the processed signals in parallel as output signals to the M memories MR. This can reduce the input / output latency.

[0106] In a Fig. In the operation example shown in Figure 10B, the size of the image data DT of one channel is (8p + 0) × 16 bytes, where p is an integer, and the channel stride (CHstride) is set to (8p + 1) × 16 bytes accordingly. That is, the operation example when a blank area BLNK of 16 bytes × 1 is provided at a location as shown in Figure 10B. Fig. 4 is shown. Alternatively, an operation example is shown when the size of the image data DT of one channel is (8p + 1) × 16 bytes and the channel stride (CHstride) is also set to (8p + 1) × 16 bytes. That is, in Fig. 4 shows the operation example in which the respective channels CH[0] to CH[7] are configured by, for example, the pixel group data PGD0 to PGD40 and the like, and the blank area BLNK is not required.

[0107] In a Fig. In the operation example shown in Figure 10C, the size of the image data DT of one channel is (8p + 2) × 16 bytes, and the channel stride (CHstride) is set accordingly to (8p + 3) × 16 bytes. That is, in Fig. 4 is the operation example in which the respective channels CH[0] to CH[7] are configured by, for example, the pixel group data PGD0-PGD41 and the like, and accordingly, the blank area BLNK of 16 bytes × 1 is provided. Alternatively, an exemplary operation is shown in a case where the size of the image data DT of one channel is (8p + 3) × 16 bytes and the channel stride (CHstride) is also set to (8p + 3) × 16 bytes, that is, in a case where the blank area BLNK is not required.

[0108] Similarly, Fig. 10D shows an example of operation when DT is (8p + 4) × 16 bytes and the channel stride (CHstride) is set to (8p + 5) × 16 bytes. Alternatively, an example of operation is shown when DT is (8p + 5) × 16 bytes. In a Fig. In the operation example shown in Figure 10E, the image data DT is (8p + 6) × 16 bytes, and the channel stride (CHstride) is set to (8p + 7) × 16 bytes.

[0109] Alternatively, an operation example is shown when DT is (8p + 7) × 16 bytes. Note that the connection relationship between the input signals and the output signals shown in Fig. 10A, represents a part of the operation that is Fig. 10D is illustrated.

[0110] Each of the Fig. 10B, Fig. 10C, Fig. 10D and Fig. Figure 10E shows a combination of index (idx) represented by logical addresses CH[n]_addr[6:4] of eight channels, which are select signals. Three combinations are shown here, i.e., one combination for three columns, with some omissions, but in detail, there are eight combinations. Each diagram shows which of the logical addresses CH[n]_addr[18:7] of the eight channels, which are the input signals, is connected to each of the eight physical addresses MR_idx[n]_addr[11:0], which are the output signals for each combination of the select signals.

[0111] Fig. 11 is a schematic diagram showing a configuration example and a partial operation example of the data router for writing to Fig. 9. The write data router 66, which is Fig. 11 has the same configuration as that of the address router shown in Fig. 10A, except that the input signals and the output signals differ from those in Fig. 10A, and performs the same operation. The input signals to the write data router 66 are the channel write data CH[n]_WRdat[127:0] of the eight channels, which are output from the MAC unit 25 and stored in the write buffer 28 in channel order by an operation such as an activation function. Each channel write data CH[n]_WRdat[127:0] is 16 bytes (= 128 bits).

[0112] The output signals from the write data router 66 are eight memory write data MR_idx[n]_WRdat[127:0] arranged in the memory order. The write data router 66 uses the same selection signal as in Fig. 10A, i.e., idx is based on the write logic address CH[n]_WRaddr[6:4] of the eight channels from the write address generator 64 to define the connection relationship between the input signals and the output signals. Consequently, the memory MR for each channel identified by the write address router 65, i.e., the correspondence between the channel and the memory MR, is the same as the memory MR for each channel identified by the write data router 66.

[0113] Fig. 12 is a schematic diagram showing a configuration example of a data router for reading in Fig. 9. The read data router 56, which is Fig. 12 has basically the same configuration as the write data router 66 shown in Fig. 11, except that the direction of the input and output is opposite to that of Fig. 11, and performs the same operation. The read data router 56 receives eight memory read data MR_idx[n]_REdat[127:0] read from eight MR memories and arranged in memory order. The size of each memory read data MR_idx[n]_REdat[127:0] is 16 bytes (= 128 bits).

[0114] The read data router 56 outputs eight channel read data CH[n]_REdat[127:0] arranged in the order of the channels. The read data router 56 uses the same selection signal as in FIG. 10A, i.e., idx based on the read logic address CH[n]_REaddr[6:4] of the eight channels from the read address generator 54 for reading, to define the connection relationship between the input signals and the output signals. Consequently, the memory MR for each channel identified by the read address router 55, i.e., the correspondence between the channel and the memory MR, is the same as the memory MR for each channel identified by the read data router 56.

[0115] However, during the read operation, the memory MR receives the physical address and outputs the memory read data with a read latency of a predetermined number of clock cycles. To compensate for this read latency, the pending address buffer 57 buffers a specific bit range ([6:4]) in the read logic address CH[n]_REaddr[18:4] of the eight channels from the read address generator 54 for a period based on the read latency. The pending address buffer 57 then outputs the buffered read logic address CH[n]_REaddr[6:4] to the read data router 56.

[0116] In Fig. 12, the read data router 56 includes eight selectors 58[0] to 58[7]. The selector 58[0] selects one of the eight memory read data MR_idx[0]_REdat-MR_idx[7]_REdat based on the read logic address CH[0]_REaddr[6:4] of the channel CH[0], which is the select signal of the channel CH[0]. Then, the selector 58[0] outputs the selected memory read data as the channel read data CH[0]_REdat of the channel CH[0].

[0117] Similarly, the selector 58[7] selects one of the eight memory read data MR_idx[0]_REdat-MR_idx[7]_REdat based on the read logic address CH[7]_REaddr[6:4] of the channel CH[7], which is the selection signal of the channel CH[7]. Then, the selector 58[7] outputs the selected memory read data as the channel read data CH[7]_REdat of the channel CH[7]. The address routers 55 and 65, which are in Fig. 10A, and the write data router 66 shown in Fig. 11 can also be realized with the same configuration by changing the input and output directions of the selectors. Details of the neural network software system

[0118] In Fig. 9, the neural network software system 40 includes a read control unit 75a and a write control unit 76a, which are implemented by the CPU 20 that executes the neural network software. The read control unit 75a, i.e., the CPU 20, mainly sets the setting value of each register included in the read access controller 30a as the read configuration parameter 80 and sets the setting value in each register. Here, the read control unit 75a includes a read channel stride correction unit 81. The read channel stride correction unit 81 corrects the channel stride (CHstride) for reading included in the read configuration parameter 80 as needed and sets the corrected channel stride in the channel stride register 51.

[0119] Similarly, the write control unit 76a, i.e., the CPU 20, mainly sets the setting values ​​of the respective registers included in the write access controller 31a as the write configuration parameters 85 and sets the setting values ​​in the respective registers. Here, the write control unit 76a includes a write channel stride correction unit 86. The write channel stride correction unit 86 corrects the write channel stride included in the write configuration parameter 85 as needed and sets the correction channel stride (CHstride) in the write channel stride register 61.

[0120] The read channel stride correction unit 81 can set the corrected write channel stride (CHstride) obtained by the write channel stride correction unit 86 in the read channel stride register 51 as the corrected read channel stride (CHstride). That is, the channel stride (CHstride) used for a write operation for one layer and the channel stride (CHstride) used for a read operation for a subsequent layer are usually the same. For example, in Fig. 20 the channel stride (CHstride) used at the output of the NNE process[1] and the channel stride (CHstride) used at the input of the NNE process[2], both channel strides (CHstride) applied to the same feature map FM1.

[0121] Fig. 13 is a flowchart showing an example of the processing contents of the channel stride correction unit in Fig. 9. Here, as a premise, as with reference to Fig. 7, assuming that the number of M, which is the number of memories, 2 m where m is an integer of 1 or more, and the bit width of each of the M memories is 2 k where k is an integer of 0 or more. Further, assuming that a is an integer greater than or equal to 0 and less than m, the size GS of the pixel group data PGD 2 (k+a) bytes and the number of N channels is 2 (m-a) is.

[0122] In Fig. 13, for example, the read channel stride correction unit 81 first acquires the image size FS, which is an initial value of a channel stride (CHstride) (step S101). That is, in the read configuration parameter 80, the setting value of the channel stride (CHstride), in other words, the initial value, is determined as the image size FS, as shown in Fig. 19. In addition, the read channel stride correction unit 81 refers to the size GS of the pixel group data PGD set in advance (step S102).

[0123] Subsequently, the read channel stride correction unit 81 calculates FS / GS and determines whether the calculated value is an even number (step S103). If the calculation result in step S103 is an even number, the read channel stride correction unit 81 sets the value of FS + GS × odd number in the channel stride register 51 for reading (step S104). That is, the read channel stride correction unit 81 corrects the set value of the channel stride. On the other hand, if the calculation result in step S103 is an odd number, the read channel stride correction unit 81 sets the value of FS in the channel stride register 51 (step S105). That is, the read channel stride correction unit 81 does not correct the set value of the channel stride.

[0124] For example, in the case of Fig. 4, the FS / GS calculated by step S103 is 40 (= 640 / 16). For this reason, for example, 640 + 16 × 1 is set in the channel stride register 51 (step S104). However, as shown in Fig. 4, FS / GS is an odd number, the pixel data PD of the N channels located at the same pixel position are respectively stored in memories MR that are different from each other among the M memories MR, without correcting the set value of the channel stride. The write channel stride correction unit 86 also performs the same processing as that of the read channel stride correction unit 81.

[0125] In addition, methods for correcting the channel stride (CHstride) are expressed by expressions (8) and (9) using FS bytes as an image size, K bytes as a bit width of memory MR, and M as a number of memories MR. Here, FLOOR(FS, K × M) is a function that rounds FS to a multiple of (K × M). The skip_factor is a value obtained by rounding mod(CEIL(FS / K, 1), M) up to an odd number. CEIL(FS / K, 1) is a function that rounds FS / K up to an integer. CHstride(pre-correction)=FS CHstride(post-correction)=K×skip_factor+FLOOR(FS, K×M)

[0126] Note that if K bytes, which is one bit width of the memory MR, and M, which is the number of memories MR, are not powers of 2, the channel strides can be rounded up to the number of pixel group data PGD that is prime to the number of channels. Main effect of the first embodiment

[0127] Above, in the method of the first embodiment, the memory controller 29 is provided. The memory controller 29 controls access to the SPM 16 such that the pixel data PD of the N channels located at the same pixel position are respectively stored in different memories MR among the M memories MR. Thus, the pixel data PD of the N channels can be input and output to the M memories MR in parallel. Consequently, the input / output latency between the SPM 16 and the NNE 15 can be reduced. Furthermore, since the planar format is used as the general-purpose format, the input / output latency can be shortened even when the DSP process or the like is included. Consequently, the processing time of the image processing can be shortened.

[0128] Furthermore, in the method of the first embodiment, the channel stride registers 51 and 61 and the address generators 54 and 64 are provided in the memory controller 29, and an appropriate channel stride, that is, an appropriate address pitch, is set in the channel stride registers 51 and 61, so that the input / output latency is shortened. Such a method using the channel stride registers 51 and 61 can reduce the number of necessary registers, which is advantageous in terms of the area of ​​the registers, the processing load associated with setting the registers, and the setting time. In particular, the larger the number of channels, the more advantageous the effect. Second Embodiment Details of the Main Part of the Semiconductor Device

[0129] Fig. 14 is a block diagram showing a detailed configuration of the main part in the semiconductor device according to the second embodiment. Fig. 14 is different from the semiconductor device 10 shown in Fig. 9 in the configuration of the read access controller 30b, the configuration of the write access controller 31b, and the configuration of the read control unit 75b and the write control unit 76b in the neural network software system 40.

[0130] The read access controller 30b includes a read address register unit 90 instead of the read base address register 50, the channel stride register 51 and the read address generator 54, which are Fig. 9. The read address register unit 90 includes N address registers. In each of the N address registers, the starting address CH[n]_RSaddr of the respective channels is set in the image data DT of the N channels.

[0131] The adder 53b adds the common scan address Saddr from the address counter 52 to the N start address CH[n]_RSaddr output from the read address register unit 90. As a result, the adder 53b outputs in parallel the read logic address CH[n]_REaddr of the N channels and the output from the read address generator 54 in Fig. 9 out.

[0132] According to such a difference, the read control unit 75b comprises a read address correction unit 95 instead of the read channel stride correction unit 81 provided in Fig. 9. The read address correction unit 95 determines an address distance between the channels by the same processing flow as in the case of Fig. 13 and further performs processing corresponding to the read base address register 50 and the read address generator 54 in Fig. 9. That is, the read address correction unit 95 sequentially adds the determined address distance to a base address for a specific read operation or adds an integer multiple of the address distances, thereby calculating N start addresses CH[n]_RSaddr.

[0133] Then, the address correction unit 95 sets the calculated N start addresses CH[n]_RSaddr in the N address registers in the read address register unit 90. Consequently, as in the case of Fig. 13. The address spacing between adjacent channels in the N starting address CH[n]_RSaddr is FS + GS × odd number if FS / GS is an even number. On the other hand, if FS / GS is an odd number, the address spacing between adjacent channels is FS.

[0134] Similarly, the write access controller 31b also includes a write address register unit 91 instead of the write base address register 60, the channel stride register 61 and the write address generator 64 shown in Fig. 9. The write address register unit 91 includes N address registers. In each of the N address registers, the start address CH[n]_WSaddr of the respective channels is set in the image data DT of the N channels. The adder 63b outputs the write logic address CH[n]_WRaddr of the N channels in parallel by adding the common scan address Saddr from the address counter 62 to the N start addresses CH[n]_WSaddr output from the write address register unit 91.

[0135] The write controller 76b includes a write address correction unit 96. The write address correction unit 96 determines an address spacing between the channels. The write address correction unit 96 sequentially adds the determined address spacing to a base address for a specific write operation or adds an integer multiple of the address spacing to calculate N start addresses CH[n]_WSaddr. The write address correction unit 96 then sets the calculated N start addresses CH[n]_WSaddr in the N address registers in the write address register unit 91. Main effects of the second embodiment

[0136] As described above, using the method of the second embodiment, similar effects to the various effects described in the first embodiment can be achieved. In the second embodiment, the memory controller 29 is provided with address register units 90 and 91 including N address registers, and an appropriate start address for each channel is set in the address register units 90 and 91, thereby reducing input / output latency.

[0137] Therefore, compared to the method of the first embodiment, a disadvantage is achieved from the perspective of the processing load and the setting time associated with the register area and the register setting, but an advantage is achieved from the perspective of increasing the degree of freedom of setting. For example, the blank area BLNK between the Fig. 4 can be extended by any number of 128-byte units between the channels. In particular, in a typical CNN process, such a degree of freedom is often not required, but in a specific neural network process, such a degree of freedom may be necessary. Third Embodiment Details of the Main Part of the Semiconductor Device

[0138] Fig. 15 is a block diagram showing a detailed configuration of the main part in the semiconductor device according to the third embodiment. Fig. 15 illustrates the semiconductor device 10 differs from that shown in Fig. 9 in the configuration of the read access controller 30c, the configuration of the write access controller 31c, and the configuration of the read control unit 75c and the write control unit 76c in the neural network software system 40.

[0139] The write access controller 31c includes a preliminary write channel stride register 61c, a write channel stride correction circuit 105 and a write status register 106 instead of the Fig. 9. In the preliminary write channel stride register 61c, a preliminary value of a write channel stride, in other words, a preliminary value of an address pitch, is set by the write control unit 76c.

[0140] The write channel stride correction circuit 105 corrects the preliminary value of the channel stride as needed by performing the same processing as that in Fig. 13, performs the write channel stride correction unit 86 using a dedicated hardware circuit. The write channel stride correction circuit 105 then outputs the corrected channel stride value to the write address generator 64 and the write status register 106.

[0141] Similarly, the read access controller 30c includes a preliminary read channel stride register 51c, a read channel stride correction circuit 100, and a read status register 101 instead of the one shown in Fig. 9. In the preliminary read channel stride register 51c, a preliminary value of a read channel stride, in other words, a preliminary value of an address distance, is set by the read control unit 75c.

[0142] The read channel stride correction circuit 100 corrects the preliminary value of the read channel stride as needed by performing the same processing as that of the Fig. 13 described read channel stride correction unit 81 by a dedicated hardware circuit. The read channel stride correction circuit 100 outputs the corrected channel stride to the read address generator 54 and the read status register 101. The reading control unit 75c reads from the

[0143] Write status register 106 stores the channel stride after correction by write channel stride correction circuit 105, which is defined, for example, by the intermediate layer in the previous stage. Then, read control unit 75c writes the read channel stride value into preliminary channel stride register 51c as the read channel stride value in the intermediate layer or the like in the subsequent stage.

[0144] Thus, the feature map FM generated by the intermediate layer or the like in the previous stage can be used as an input to the intermediate layer or the like in the subsequent stage. Since correction by the read channel stride correction circuit 100 is unnecessary, the read control unit 75c can, for example, output a control signal indicating that correction is unnecessary to the read channel stride correction circuit 100 at this time. In addition, the channel stride read from the write status register 106 is not limited to the NNE 15 and is also used in the DSP 18 and the DMAC 17.

[0145] On the other hand, the write control unit 76c reads the channel stride from the read status register 101, which is corrected by the read channel stride correction circuit 100 used, for example, in an intermediate layer of a certain stage. Then, the write control unit 76c writes the read channel stride value into the preliminary write channel stride register 61c as the channel stride value for writing to the intermediate layer or the like in the previous stage.

[0146] As a result, a memory map to be applied to the feature map FM output from the intermediate layer or the like in the previous stage can be determined based on the feature map FM input to the intermediate layer or the like. Since correction by the write channel stride correction circuit 105 is unnecessary, the write control unit 76c may, for example, output a control signal indicating that correction is unnecessary to the write channel stride correction circuit 105 at this time. In addition, the channel stride read from the read status register 101 is not limited to the NNE 15 and is also used in the DSP 18 and the DMAC 17.

[0147] Note that the process in the write control unit 76c is a process that returns from the back to the front in time, unlike the process in the read control unit 75c. For this reason, it is necessary to predetermine the value of the channel stride for reading before starting the processing in the intermediate layer or the like in the previous stage, for example, by providing two register areas or the like. The predetermined channel stride value is set as the write channel stride value when performing the processing in the intermediate layer or the like in the previous stage.

[0148] Fig. 16 is a block diagram illustrating a detailed configuration example of a main part different from that of the Fig. 15. As the case of Fig. 15 shows Fig. 16 that in Fig. 14, to which the method of the third embodiment is applied. The configuration shown in Fig. 16 illustrates the semiconductor device 10 differs from that shown in Fig. 14 in the configuration of the read access controller 30d, the configuration of the write access controller 31d, and the configuration of the read control unit 75d and the write control unit 76d in the neural network software system 40.

[0149] In the write access controller 31d, the write address correction circuit 115 and the write status register 116 are connected to the Fig. 14 shown configuration. The write address register unit 91 in Fig. 14 is stored by a provisional address register unit 91d in Fig. 16. In the provisional address register unit 91d, a provisional value of N start addresses assigned to N channels is set by the write control unit 76d.

[0150] The write address correction circuit 115 corrects the provisional values ​​of the N start addresses as needed by performing the same processing as that shown in Fig. 14 performs the write address correction unit 96 through a dedicated hardware circuit. The write address correction circuit 115 outputs the N corrected start addresses to the adder 63b and the write status register 116.

[0151] Similarly, in the read access controller 30d, a read address correction circuit 110 and a read status register 111 are added to the Fig. 14 shown configuration. The read address register unit 90 in Fig. 14 is stored by the provisional address register unit 90d in Fig. 16. In the preliminary address register unit 90d, a preliminary value of N start addresses assigned to N channels is set by the read control unit 75d.

[0152] The read address correction circuit 110 corrects the provisional values ​​of the N start addresses as needed by performing the same processing as that of the Fig. 14 for reading by a dedicated hardware circuit. The read address correction circuit 110 outputs the N corrected start addresses to the adder 53b and the read status register 111. The operations of the read control unit 75d and the write control unit 76d are the same as those of the read control unit 75c and the write control unit 76c described with reference to Fig. 15, except that the process destination is replaced by the start address of each channel from the channel stride. Main effects of the third embodiment

[0153] Above, by using the scheme of the third embodiment, similar effects to the various effects described in the first embodiment or the second embodiment can be obtained. Furthermore, by correcting the channel stride or the start address of each channel through the dedicated hardware circuit, the processing load of the software can be reduced. Furthermore, by causing the software to recognize the correction result through the hardware circuit via the status register, the software can determine the processing contents of each intermediate layer or the like, particularly the memory map, by reflecting the correction result. Consequently, it is possible to increase the efficiency of image processing. Fourth embodiment

[0154] Since the neural network architecture for improving image recognition accuracy is not limited to CNN, for example, vector operations, matrix transposition (Transpose), and matrix operations (Matmul, Gemm, etc.), ViT (VisionTransformer for performing the process), and the like are known. In architectures such as ViT, vector operations, matrix transposition, and matrix operations are performed by replacing the image data with matrix structures. In this case, it is necessary to handle not only three-dimensional (X-direction, Y-direction, channel direction) data as described in the first embodiment, etc., but also D-dimensional data comprising four or more dimensions.

[0155] Therefore, in the fourth embodiment, the method of the first embodiment and the like are extended to D dimensions, for example, 4 dimensions, where D is an integer of 2 or more. That is, in the SPM 16, D-dimensional data is stored in planar formats. Then, in the fourth embodiment, a method for accessing a plurality of data having D dimensions, in other words, D dimensions or D axes parallel to the SPM 16, is shown. It should be noted that the semiconductor device according to the fourth embodiment has the same configuration as the various configurations described in the first to third embodiments. Here, it is assumed that the semiconductor device 10 has the Fig. 1 and Fig. 9 has the configuration shown. Storage method of four-dimensional data in the memory

[0156] Fig. Fig. 17 is a schematic diagram showing an example of a four-dimensional data format used in the semiconductor device according to the fourth embodiment. Fig. 17, the total number of data num_ALL bytes is expressed by Expression (10). In Expression (10), num_AX1, num_AX2, num_AX3, num_AX4 are the number of elements in the first, second, third, and fourth dimensions, in other words, the first axis AX1, the second axis AX2, the third axis AX3, and the fourth axis AX4, respectively. Num_ALL=num_AX1×num_AX2×num_AX3×num_AX4

[0157] Fig. Figure 17 illustrates an example where the number of elements in the third axis AX3 is four and the number of elements in the fourth axis is three. As in Fig. As shown in Figure 17, the elements in the fourth axis AX4 from the second axis AX2 are not necessarily tightly arranged in the memory MR, but are arranged with a constant stride between the elements. Here, the logical address Laddr_4D, in which the 4-dimensional data DAT[AX4-idx] [AX3-idx] [AX2-idx] [AX1-idx] is stored, is expressed by Expression (11) using the index (idx) values ​​in the first axis AX1, the second axis AX2, the third axis AX3, and the fourth axis AX4 as AX1-idx, AX2-idx, AX3-idx, AX4-idx. Laddr_4D=AX1−idx+AX2−idx×AX2_stepde+AX3−idx×AX3_stepde+AX4−idx×AX4_stepde

[0158] For example, in expression (11), for image data, AX1-idx is an index value in the X direction (horizontal direction), AX2-idx is an index value in the Y direction (vertical direction), and AX3-idx is an index value in the channel direction. AX4-idx is an index value that further distinguishes such three-dimensional image data. Here, num_AX1 is the width (horizontal image size), num_AX2 is the height (number of rows), and num_AX3 is the number of channels.

[0159] In Expression (11), AX2_stride is a stride (byte) between adjacent elements in the second axis AX2 and is a row stride in the case of image data. AX3_stepde is a stride between adjacent elements in the third axis AX3 and is a channel stride in the case of image data. AX4_stepde is the stride between adjacent elements in the fourth axis. The stride between adjacent elements must be greater than or equal to the number of elements of an axis one dimension lower to prevent address conflicts between adjacent elements. For this reason, the constraints shown in Expression (12A), Expression (12B), and Expression (12C) are provided. AX2_stepde>=num_AX1 AX3_stride>=num_AX2×AX2_stride AX4_stride>=num_AX3×AX3_stride

[0160] Here, the memory controller 29 controls access to the SPM 16 so that in two or more axes, including the first axis AX1, a plurality of data items with the same index value are not stored in the same memory MR in the M memories MR. At this time, the neural network software system 40 determines a read stride and a write stride, which are address distances, and sets each of the determined strides in the read stride register and the write stride register, respectively, so that such access is performed. The read stride register and the write stride register correspond to the channel stride register 51 and the channel stride register 61, respectively. Fig. 9.

[0161] In other words, the SPM 16 stores D-dimensional data in which the respective data in one dimension differ by an index (idx) value, where D is an integer of 2 or more. The memory controller 29 controls access to the SPM 16 such that the number of index values ​​in the D-th dimension is N, and N data items with the same index value in the first to (D-1)-th dimensions are stored in different memories MR in the M memories MR.

[0162] Consequently, for example, in the four-dimensional data DAT [AX4-idx] [AX3-idx] [AX2-idx] [AX1-idx], N data elements consisting of DAT[0] [0] [0] [0], DAT[1] [0] [0] [0] [0],..., DAT[N-1] [0] [0] [0] are stored in mutually different memories MR. As a result, the memory controller 29 can read the N data elements in parallel and write the N data elements in parallel to the SPM 16.

[0163] Furthermore, the memory controller 29 can control access to the SPM 16 such that N1 × N2 × ... × N d-1 Data elements contained in the first to (D-1)-th dimensions are stored in memories MR that differ from each other, in the M memories MR, where the number of index values ​​in the 1, 2,..., D-1, D-th dimension is N1, N2,..., N d-1 , N d -elements. The number M of memories MR is N1 × N2 × ... × N d-1 or more. Example of memory access

[0164] Fig. 18A is a diagram illustrating a specific example of a D-dimensional format in the semiconductor device according to a fourth embodiment. In the specific example (Example 3) shown in Fig. As shown in Figure 18A, the values ​​of the respective variables are the power of 2, which is generally used. The bit width K of the respective memory MR is 8 (= 2 k) bytes and the size N1 of the minimum data unit is 16 (= 2 (a+k) = A × K) bytes. The size N1 of the minimum data unit is the size of each data in the first axis AX1 and corresponds to the size GS of the pixel group data PGD in the first embodiment or the like.

[0165] The number N2, N3, N4, N5,... of the minimum data units accessed in parallel in the second axis AX2, the third axis AX3, the fourth axis AX4, the fifth axis AX5,... is the number of minimum data units that can be input or output in parallel in one clock cycle with respect to the SPM 16, for example, the number of pixel group data PGD. Here, the number N2, N3, N4, N5,... of the minimum data units is 2, 2, 4, 1,..., respectively. The number M of memories MR constituting the SPM 16 is 32 (= 2 m). The total number N of minimum data units accessed in parallel is 16 (= N2 × N3 × N4 × N5,...). The total bit width accessed in parallel is 256 (= 16 × 16) bytes, calculated by multiplying the size N1 of the minimum data unit by the total number N.

[0166] Fig. Figure 18B is a schematic diagram illustrating an arrangement configuration of data to be accessed in parallel in a four-dimensional format, under the premise that the configuration in Fig. 18A is shown. Fig. Figure 18B shows 16 DAT0-DAT15 input and output in parallel in the same clock cycle in the four-dimensional format. In the description, a plurality of data DAT0-DAT15 are collectively referred to as data DAT.

[0167] The size of one DAT data is 16 bytes, which is the size N1 of the minimum data unit defined in the first axis AX1. Sixteen data DAT0-DAT15 are arranged in two along the second axis AX2 using the AX2_stride for the second axis. Similarly, 16 DAT0-DAT15 are arranged in two along the third axis AX3 using the AX3_stride for the third axis, and in four along the fourth axis AX4 using the AX4_stride for the fourth axis.

[0168] Fig. 18C is a diagram showing the start address and end address of the respective Fig. 18B. In each DAT, the end address is the sum of the start address and N1-1. Further, for example, the start address of data DAT1 is obtained by adding the stride AX2_stride of the second axis to the start address of data DAT0. The start address of data DAT2 is a value obtained by adding the stride AX3_stride of the third axis to the start address of data DAT0. The start address of data DAT3 is obtained by adding the stride AX2_stride of the second axis to the start address of data DAT2.

[0169] Fig. Figure 18D is a diagram illustrating an example neural network software executed by the CPU under the premise that the configuration in Fig. 18A. The CPU 20 executes multi-loop programming, for example, as shown in Fig. 18D, which increases in dimension toward the outside. At this time, the CPU 20 causes the neural network engine (NNE) 15 to perform the arithmetic process associated with the multi-loop. Accordingly, the NNE 15 must input or output N1 × N2 × N3 × N4 data elements, here 16 data elements DAT0 to DAT15, to the SPM 16.

[0170] Therefore, the CPU 20 sets, for example, a read stride or a write stride as shown in Expression (13A) to Expression (13C) for the read stride register corresponding to the channel stride register 51 shown in Fig. 9, or the write stride register corresponding to the channel stride register 61. That is, the CPU 20 corrects the stride AX2_stride for the second axis to satisfy the constraint condition of Expression (12A), and adds the size N1 (= 16 bytes) to the multiple of 256 bytes (256n) that is K × M, as shown in Expression (13A). As described above, the size N1 is the size of the minimum data units in the first axis AX1, in other words, the size of the packet group data PGD.

[0171] Further, the CPU 20 corrects the stride AX3_stride for the third axis to be a value that is a multiple of 256 bytes (256n) and adds N1 × N2, which is the multiplication result of the size N1 and the number N2, while satisfying the constraint condition of expression (12B). As described above, the number N2 is the number of the minimum data unit to be accessed in parallel in the second axis AX2. Further, the CPU 20 corrects the stride AX4_stride for the fourth axis to be a value that is a multiple of 256 bytes (256n) and adds N1 × N2 × N3, which is the multiplication result of the size N1, the number N2, and the number N3, while satisfying the constraint condition of expression (12C). As described above, the number N3 is the number of minimum data units to be accessed in parallel in the third axis AX3. AX3_stride=256n+N1 AX3_stride=256n+N1×N2 AX4_stride=256n+N1×N2×N3

[0172] As a result, a memory map is formed as shown in Figure 18E. Fig. 18E is a diagram illustrating an exemplary memory map of the respective data stored in the scratchpad memory (SPM) after the stride correction has been performed under the premise that the specific Fig. 18C. As shown in Fig. As shown in Figure 18E, by performing stride corrections, sixteen data DAT0-DAT15 are assigned to different idx in the SPM 16 and stored in different MR memories, here 32 MR memories. As a result, the NNE 15 can input and output 16 DAT0-DAT15 to and from the SPM 16 in parallel. Consequently, the input / output latency can be reduced. Main effects of the fourth embodiment

[0173] As described above, by using the method of the fourth embodiment, similar effects to the various effects described in the first to third embodiments can be obtained. Furthermore, the same effect can be obtained with respect to data in the D dimension, which is two or more dimensions.

[0174] Although the invention made by the present inventor has been specifically described based on the embodiment, the present invention is not limited to the above-described embodiment, and needless to say, various modifications can be made without departing from the gist thereof. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2023-097830

[0001] JP 2019-40403

[0004]

Claims

[1] A semiconductor device comprising: a scratchpad memory storing image data of N channels and comprising M individually accessible memories, where M is an integer of 2 or more and N is an integer of 2 or more; a memory controller configured to control access to the scratchpad memory such that pixel data of the N channels located at a same pixel position in image data of the N channels are respectively stored in different memories of the M memories; a MAC unit (multiply-accumulate) comprising a plurality of calculators for calculating pixel data of the N channels read from the scratchpad memory using the memory controller and a weighting parameter. [2] The semiconductor device according to claim 1, wherein the memory controller comprises: a read access controller configured to generate read logic addresses of the N channels for reading the pixel data of the N channels respectively when the pixel data of the N channels are read from the scratchpad memory, and a write access controller configured to generate write logic addresses of the N channels for storing the pixel data of the N channels respectively when the pixel data of the N channels is written into the scratchpad memory. [3] A semiconductor device according to claim 2, wherein the read access controller comprises a read address router configured to translate the read logical addresses of the N channels into physical read addresses of the M memories to output the physical read addresses of the N channels to the M memories each associated with one of the N channels, and wherein the write access controller comprises a write address router configured to translate the write logical addresses of the N channels into physical write addresses for the M memories, each associated with one of the N channels, to output the physical write addresses to the M memories. [4] The semiconductor device according to claim 3, wherein the write access controller comprises a write data router configured to output the pixel data of the N channels obtained based on calculation results of the MAC unit to the M memories each associated with one of the N channels. [5] A semiconductor device according to claim 3, wherein the read address router identifies a correspondence between the N channels and the M memories by a value of a specific bit range contained in the read logic addresses of the N channels, and wherein the write address router identifies the correspondence between the N channels and the M memories by a value of a specific bit range contained in the write logic addresses of the N channels. [6] A semiconductor device according to claim 5, wherein the read access control further comprises: a read data router configured to rearrange the pixel data of the N channels read from the M memories in a channel order based on the physical read addresses from the read address router, and a pending address buffer configured to buffer the particular bit range of the read logic address of the N channels for a period based on a read latency of the M memories, wherein the read data router receives the particular bit range of the read logic address of the N channels from the pending address buffer, and where the MAC unit receives the pixel data of the N channels rearranged by the read data router. [7] A semiconductor device according to claim 2, where an access control is the read access control or the write access control, and where access control includes: a channel stride register in which an address distance between start addresses of the image data of adjacent channels of the N channels stored in the scratchpad memory is set, and an address generator that adds an integer multiple of the address offset to a reference logic address to generate logic addresses for the N channels in the scratchpad memory. [8] A semiconductor device according to claim 7, wherein the access controller accesses the scratchpad memory with a pixel data group comprising a plurality of the pixel data as a data unit, where the M memories each have a bit width of 2 k bytes, where a value of M 2 m is, m is an integer of 1 or more and k is an integer of 0 or more, where the channel stride register is set to FS + GS × odd number when FS / GS is an even number, and is set to FS when FS / GS is an odd number, where FS is a byte size of each of the image data of the N channels, GS is a byte size of the pixel data group, or 2 (k+a) bytes, a is an integer of 0 or more and less than m, a value of N 2 (m-a) number is. [9] A semiconductor device according to claim 2, where an access control is the read access control or the write access control, and where access control includes: N address registers each storing N start addresses, each of the N start addresses being a start address of image data of an associated channel of the N channels, and an adder that adds a common sample address to each of the N start addresses. [10] A semiconductor device according to claim 9, wherein the access controller accesses the scratchpad memory with a pixel data group comprising a plurality of the pixel data as a data unit, where the M memories each have a bit width of 2 k bytes, where a value of M 2 m is, m is an integer of 1 or more and k is an integer of 0 or more, wherein the address distance between start addresses of the image data of adjacent channels of the N channels is set to FS + GS × odd number when FS / GS is an even number, and is set to FS when FS / GS is an odd number, where FS is a byte size of each of the image data of the N channels, GS is a byte size of the pixel data group, or 2 (k+a) bytes, a is an integer of 0 or more and less than m, a value of N 2 (m-a) number is. [11] A semiconductor device according to claim 1, further comprising: a DMA controller (Direct Memory Access controller) configured to control data transfer between the scratchpad memory and main memory. [12] A semiconductor device according to claim 1, further comprising: a DSP (Digital Signal Processor) that performs signal operations on the N-channel image data stored in the scratchpad memory. [13] A semiconductor device comprising: a scratchpad memory storing image data of N channels and comprising M individually accessible memories, where M is an integer of 2 or more and N is an integer of 2 or more; a memory controller configured to control access to the scratchpad memory based on a setting value of a register; a CPU (Central Processing Unit) configured to determine the setting value of the memory control register; and a MAC unit (multiplication-accumulation unit) comprising a plurality of computers, wherein the CPU determines the setting value of the register such that pixel data of N channels arranged at a same pixel position in image data of the N channels are respectively stored in different memories of the M memories, and wherein each of the computers performs a multiply-accumulate operation on the pixel data of the N channels read from the scratchpad memory using the memory control and a weighting parameter. [14] The semiconductor device according to claim 13, wherein the memory controller comprises: a channel stride register in which an address distance between start addresses of the image data of adjacent channels of the N channels stored in the scratchpad memory is set, and an address generator that adds an integer multiple of the address offset to a reference logic address to generate logic addresses for the N channels in the scratchpad memory, and wherein the CPU sets the address offset in the channel stride register. [15] A semiconductor device according to claim 14, wherein the memory controller accesses the scratchpad memory with a pixel data group comprising a plurality of the pixel data as a data unit, where the M memories each have a bit width of 2 k bytes, where a value of M 2 m is, m is an integer of 1 or more and k is an integer of 0 or more, where the CPU sets FS + GS × odd number in the channel stride register when FS / GS is an even number, and sets FS when FS / GS is an odd number, where FS is a byte size of each of the image data of the N channels, GS is a byte size of the pixel data group, or 2 (k+a) bytes, a is an integer of 0 or more and less than m, a value of N 2 (m-a) number is. [16] A semiconductor device according to claim 13, where the memory control includes: N address registers each storing N start addresses, each of the N start addresses being a start address of image data of an associated channel of the N channels, and an adder adding a common sampling address to each of the N start addresses, where the CPU sets the N starting addresses in the N address registers. [17] A semiconductor device according to claim 16, wherein the memory controller accesses the scratchpad memory with a pixel data group comprising a plurality of the pixel data as a data unit, where the M memories each have a bit width of 2 k bytes, where a value of M 2 m is, m is an integer of 1 or more and k is an integer of 0 or more, wherein the address distance between start addresses of the image data of adjacent channels of the N channels is set to FS + GS × odd number when FS / GS is an even number, and is set to FS when FS / GS is an odd number, where FS is a byte size of each of the image data of the N channels, GS is a byte size of the pixel data group, or 2 (k+a) bytes, a is an integer of 0 or more and less than m, a value of N 2 (m-a) number is. [18] A semiconductor device comprising: a scratchpad memory storing D-dimensional data and comprising M individually accessible memories, the D-dimensional data being configured such that the data in one dimension differs by an index value, where D is an integer of 2 or more and M is an integer of 2 or more, and a memory controller configured to control access to the scratchpad memory such that N data items having a same index value in the first to (D-1)th dimensions are respectively stored in different memories in the M memories, where the number of the index value in the D dimension is N.

Images