Semiconductor device

Abstract

The embodiment of the invention relates to a semiconductor device. A semiconductor device includes a temporary memory, a memory controller, and a multiply-accumulate (MAC) unit. The temporary memory is configured to store image data of N channels and comprises M memories which can be accessed separately, wherein M is an integer of at least 2 and N is an integer of at least 2. The memory controller controls access to the temporary memory such that pixel data of the N channels arranged at the same pixel position among the image data of the N channels is stored in different memories of the M memories, respectively. The MAC unit includes a plurality of calculators to calculate pixel data of the N channels read from the temporary memory by using the memory controller and the weight parameter.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The disclosure of Japanese Patent Application No. 2023-097830 filed on June 14, 2023, including specification, drawings and abstract, is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to a semiconductor device, and, for example, to a semiconductor device for performing a neural network process. Background Art

[0004] Patent document 1 discloses an image recognition device having an integrated coefficient table generation device and an input pattern generation circuit to efficiently perform convolution arithmetic operations in a CNN (convolutional neural network). The integrated coefficient table generation device integrates two types of 3×3 input coefficient tables into one type of 5×5 integrated coefficient table and outputs it to a 5×5 convolution arithmetic operation circuit. The input pattern generation circuit generates a pixel value of 5×5 pixels from the pixel values ​​of 3×3 pixels stored in a line buffer based on a rule set in an input pattern register, and outputs the pixel value to the 5×5 convolution arithmetic operation circuit.

[0005] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2019-40403. Summary of the invention

[0006] For example, in a semiconductor device responsible for image processing such as CNN, performance improvement, particularly reduction in processing time, is achieved by performing calculations of multiple channels in a convolution layer in parallel using a plurality of multiplication-accumulation calculators included in a multiplication-accumulation (MAC) unit. In this case, in order to further enhance effective performance, it is desirable to reduce input / output delays between a temporary storage memory (also referred to as SPM in the specification) storing image data of multiple channels and the MAC unit.

[0007] Here, in order to reduce the input / output delay between the SPM and the MAC unit, for example, a method using a dedicated data format can be envisioned in which the image data of multiple channels are integrated for data transmission between the SPM and the MAC unit. However, in a series of image processing, the image data of multiple channels stored in the SPM may be processed by a general signal processing circuit such as a digital signal processor (DSP) instead of the MAC unit. The general signal processing circuit cannot support the dedicated data format. Therefore, even when the dedicated data format is used to transmit data between the SPM and the MAC unit, there is a possibility that the processing time of the image processing cannot be sufficiently shortened.

[0008] In view of the above, the following embodiments are described, and other problems and novel features will be apparent from the description and drawings of this specification.

[0009] A semiconductor device according to one aspect includes a temporary storage memory, a memory controller, and a multiply-accumulate (MAC) unit. The temporary storage memory is configured to store image data of N channels and includes M memories that can be accessed separately, where M is an integer of at least 2 and N is an integer of at least 2. The memory controller controls access to the temporary storage memory so that the pixel data of the N channels arranged at the same pixel position in the image data of the N channels are respectively stored in different memories of the M memories. The MAC unit includes a plurality of calculators to calculate the pixel data of the N channels read from the temporary storage memory by using the memory controller and weight parameters.

[0010] According to another aspect, a semiconductor device includes a temporary storage memory, a memory controller and a CPU (central processing unit) and a multiplication-accumulation (MAC) unit. The temporary storage memory stores image data of N channels and includes M memories that can be accessed separately, where M is an integer of 2 or greater than 2, and N is an integer of 2 or greater than 2. The memory controller is configured to control access to the temporary storage memory based on the set value of the register. The CPU is configured to determine the set value of the register for the memory controller. The MAC unit includes a plurality of calculators. The CPU determines the set value of the register so that the pixel data of the N channels arranged at the same pixel position in the image data of the N channels are respectively stored in different memories of the M memories, and each of the calculators performs a multiplication-accumulation operation on the pixel data of the N channels read from the temporary storage memory by using the memory controller and the weight parameter.

[0011] According to another aspect, a semiconductor device includes a temporary storage memory that stores D-dimensional data and includes M memories that can be accessed separately, the D-dimensional data being configured so that each data in one dimension is distinguished by an index value, wherein D is an integer of 2 or greater, and M is an integer of 2 or greater; and a memory controller configured to control access to the temporary storage memory so that N data fragments having the same index value in the first dimension to the D-1th dimension are respectively stored in different memories among the M memories, wherein the number of index values ​​in the D dimension is N.

[0012] By using the semiconductor device of one or more embodiments, the processing time of image processing can be shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram illustrating a schematic configuration of a semiconductor device according to a first embodiment.

[0014] Figure 2 is a schematic diagram illustrating a configuration example of a neural network.

[0015] Figure 3 This diagram is used for Figure 1 Schematic diagram of an exemplary process for an intermediate layer in a CNN in a semiconductor device shown.

[0016] Figure 4 It is used to explain Figure 1 1 and 2. The diagram is a diagram of an operation example of a memory controller in FIG. 1 and is a diagram illustrating an example of a memory map of image data of corresponding channels stored in a temporary storage memory (SPM).

[0017] Figure 5A It shows Figure 1 A timing diagram comparing a schematic operation example of the neural network engine (NNE) in FIG. 1 with a case where the system uses the first comparative example and a case where the system uses the embodiment.

[0018] Figure 5B : is a diagram illustrating an example of a case where the number of clock cycles required to process a convolution layer of one layer is compared between a case where the method of the first comparative example is used and a case where the method of the embodiment is used.

[0019] Figure 6 is used to explain the use of Figure 4 Schematic diagram of the impact on input / output latency that occurs throughout image processing using a neural network, given the memory mapping shown.

[0020] Figure 7 It is a graphic Figure 1 Illustration of various exemplary configurations of a temporary storage memory (SPM) in FIG.

[0021] Figure 8 It is a graphic with Figure 7 The corresponding example 2-2 in Figure 4 Diagrams showing examples of different memory maps.

[0022] Fig. 9 It is a graphic Figure 1 A block diagram showing the detailed configuration of main parts of a semiconductor device.

[0023] Fig. 10A It is shown Fig. 9 A diagram showing an example of the configuration of an address router and some operation examples.

[0024] Fig. 10B It is a graphic Fig. 10A Schematic diagram of an example of the operation of an address router.

[0025] Fig. 10C It is a graphic with Fig. 10B Diagrams of different operation examples.

[0026] Fig. 10D It is a graphic with Fig. 10B Diagrams of different operation examples.

[0027] Fig.10E It is a graphic with Fig. 10B Diagrams of different operation examples.

[0028] Fig.11 It is a graphic Fig. 9 A diagram showing a configuration example and some operation examples of a data router used for writing.

[0029] Fig.12 It is a graphic Fig. 9 Schematic diagram of a configuration example of a data router for reading.

[0030] Fig.13 It is a graphic Fig. 9 Flowchart of an example of processing of a channel stride correction unit in FIG.

[0031] Fig.14 is a diagram illustrating a detailed configuration of a main part of a semiconductor device according to a second embodiment.

[0032] Fig.15 is a diagram illustrating a detailed configuration of a main part of a semiconductor device according to a third embodiment.

[0033] Fig.16 It is a graphic with Fig.15 Block diagram of a detailed configuration example of the different main parts.

[0034] Fig.17 is a schematic diagram illustrating an example of a four-dimensional data format used in the semiconductor device according to the fourth embodiment.

[0035] Fig.18A is a diagram illustrating a specific example of a D-dimensional format in the semiconductor device according to the fourth embodiment.

[0036] Fig.18B The diagram is based on Fig.18A The format shown is a schematic diagram of an arrangement configuration for data accessed in parallel in a four-dimensional format.

[0037] Fig.18C It is shown Fig.18B A diagram of the start and end addresses of corresponding data segments shown.

[0038] Fig.18D The diagram is based on Fig.18AThe format shown is a diagram of an example of neural network software executed by a CPU.

[0039] Fig.18E The diagram is based on Fig.18C The format shown is a diagram of an example of a memory mapping of corresponding data stored in a temporary storage memory (SPM) after stride correction is performed.

[0040] Fig.19 is a diagram illustrating an example of memory mapping of image data of respective channels stored in a temporary storage memory (SPM) in a semiconductor device, as a first comparative example.

[0041] Fig. 20 is used to explain the use of Fig.19 Schematic diagram of the impact on input / output latency that occurs throughout image processing using a neural network, given the memory mapping shown.

[0042] Fig.21 is a schematic diagram for explaining the influence on input / output delay occurring in the entire image processing using a neural network in a semiconductor device, as a second comparative example.

[0043] Fig. 22 It is shown as Figure 8 FIG. 1 is a diagram of an example of a memory mapping of a comparative example. DETAILED DESCRIPTION

[0044] In the following embodiments, for convenience, the embodiments are described by dividing them into a plurality of parts or embodiments, but unless otherwise specified, they are not independent of each other, and one part or embodiment relates to a modification example, details, supplementary description, etc. of some or all other parts or embodiments. In the following embodiments, the number of elements, etc. (including the number of elements, numerical value, quantity, range, etc.) is not limited to a specific number, but may not be less than or equal to a specific number, unless the number is specifically indicated and is clearly limited to the specific number in principle. In addition, in the following embodiments, it goes without saying that the constituent elements (including element steps, etc.) are not necessary, unless they are specifically specified and are considered to be obviously necessary in principle. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of the components, etc., it is assumed that the shape, etc. are substantially similar or similar to the shape, etc., unless they are specifically specified and are considered to be clear in principle, etc. This also applies to the above-mentioned numerical value and range.

[0045] The embodiments will be described in detail with reference to the accompanying drawings. In all the drawings used to explain the embodiments, components having the same function are represented by the same reference numerals, and their repeated description is omitted. In the following embodiments, unless particularly necessary, the description of the same or similar parts will not be repeated in principle.

[0046] First embodiment

[0047] Figure 1 is a schematic diagram illustrating an example of a configuration of a semiconductor device according to a first embodiment; Figure 1 The semiconductor device 10 shown is, for example, a SoC (System on Chip) or a microcontroller implemented by a single semiconductor chip. The semiconductor device 10 includes a neural network engine (also referred to herein as NNE) 15, a temporary storage memory (SPM) 16, a DMA (Direct Memory Access) controller (DMAC) 17, a DSP 18, a main memory 19, a CPU (Central Processing Unit) 20, and a system bus 21. The NNE 15, the DMAC 17, the DSP 18, the main memory 19, and the CPU 20 are connected to the system bus 21.

[0048] The NNE 15 performs a neural network process represented by CNN. The SPM 16 includes M memories MR[0] to MR[M-1], which are accessible in parallel to each other, where M is an integer of 2 or greater. In this specification, the M memories MR[0] to MR[M-1] are collectively referred to as memories MR. For example, the memory MR is an SRAM. The SPM 16 is used as a cache memory of the NNE 15, and stores image data input to and output from the NNE 15. The SPM 16 can also be accessed from the DSP 18. For example, the DSP 18 is a general signal processing circuit among general signal processing circuits, and performs a portion of the neural network processing on the image data DT stored in the SPM 16.

[0049] For example, the main memory 19 is a DRAM. The main memory 19 stores image data DT, parameters PM, etc. used in the neural network process. For example, the image data DT includes a camera image CIMG obtained from a camera and a feature map FM generated by the neural network process. The parameter PM includes a weight parameter set WTS and a bias parameter BS, and the weight parameter set WTS includes a plurality of weight parameters WT according to the kernel size. The main memory 19 may be disposed outside the semiconductor device 10.

[0050] The DMAC 17 is another general-purpose signal processing circuit, and controls data transfer between the SPM 16 and the main memory 19 via the system bus 21. For example, the CPU 20 executes software (not shown) stored in the main memory 19 to make the entire semiconductor device 10 perform desired functions. As one of them, the CPU 20 constructs a neural network software system 40 by executing neural network software. For example, the neural network software system 40 performs various settings, startup controls, etc. on the NNE 15, the DMAC 17, and the DSP 18 to control the operation sequence of the entire image processing including the neural network process.

[0051] Specifically, the NNE 15 includes a MAC unit 25, a post-processor 26, a line buffer 27, a write buffer 28, and a memory controller 29. The memory controller 29 includes a read access controller 30 and a write access controller 31. The read access controller 30 reads each pixel data PDi constituting the image data DT from the SPM 16 and stores the pixel data PDi in the line buffer 27. The write access controller 31 writes the pixel data PDo stored in the write buffer 28 to the SPM 16.

[0052] The MAC unit 25 includes i multiplication-accumulation calculators MAC[0] to MAC[i], where i is an integer of 2 or greater. In this specification, the i multiplication-accumulation calculators MAC[0] to MAC[i] are collectively referred to as multiplication-accumulation calculators MAC. The multiplication-accumulation calculator MAC performs a multiplication-accumulation operation on each pixel data PDi stored in the line buffer 27 and each weight parameter WT input in advance. At this time, the MAC unit 25 reads the weight parameter set WTS from the main memory 19 in advance by using a controller (not shown).

[0053] In addition, the multiplication-accumulation calculator MAC obtains pixel data PDo through multiplication-accumulation operations of the pixel data PDi and the weight parameters WT, etc., and stores the pixel data PDo in the write buffer 28 via the post-processor 26. The post-processor 26 generates pixel data PDo by performing addition of the bias parameter BS, operation of the activation function, pooling process, etc. according to the needs of the multiplication-accumulation operation performed by the multiplication-accumulation calculator MAC.

[0054] Figure 2 4 is a schematic diagram illustrating an example of a configuration of a neural network. The neural network generally has an input layer 45, a plurality of intermediate layers 46[1] to 46[j], and an output layer 47. For example, the input layer 45 stores image data DT including three channels of R (red), G (green), and B (blue), that is, a camera image CIMG, etc.

[0055] In the intermediate layers 46[1] to 46[j], the operation results obtained by the multiplication and accumulation operations of the previous layers and the weight parameter sets WTS1 to WTSj, etc. are stored as image data DT, that is, feature maps FM1 to FMj. The corresponding feature map FM, such as FMj, has a size of Wj×Hj×Cj, where the size in the width direction or X direction is Wj, the size in the height direction or Y direction is Hj, and the number of channels is Cj.

[0056] For example, the output layer 47 stores the operation result obtained by the multiplication and accumulation operation of the last intermediate layer 46[j] and the weight parameter set WTSo as a feature map FMo. For example, the size of the feature map FMo is 1×1×Co, where the number of channels is Co. The feature map FMo is the image processing result by using a neural network. The image processing result is usually stored in the main memory 19.

[0057] Figure 1 The semiconductor device 10 shown is configured in the following manner: Figure 2 The neural network shown performs a neural network process. (A) First, a camera input interface (not shown) in the semiconductor device 10 stores image data DT from an external camera, i.e., a camera image CIMG, in the main memory 19. (B) Subsequently, the DMAC 17 transfers the camera image CIMG stored in the main memory 19 to the SPM 16 to store the camera image CIMG in the SPM 16. Thus, the input layer 45 is formed on the SPM 16.

[0058] (C) Next, the NNE 15 or the DSP 18 performs an operation using the input layer 45 formed in the SPM 16 and the weight parameter set WTS1 stored in the main memory 19 as inputs, and stores the feature map FM1 as an operation result in the SPM 16. Thus, the intermediate layer 46[1] is formed on the SPM 16. The neural network software system 40 determines whether to use the NNE 15 or the DSP 18 to perform the operation. Such determination is similarly applied to other layers.

[0059] (D) Subsequently, the NNE 15 or the DSP 18 performs an operation using the intermediate layer 46[1] formed on the SPM 16 and the weight parameter set WTS2 stored in the main memory 19 as inputs, and stores the feature map FM2 as an operation result in the SPM 16. As a result, the intermediate layer 46[2] is formed on the SPM 16. Thereafter, by performing the same process, finally the intermediate layer 46[j] is formed on the SPM 16.

[0060] (E) Next, the NNE 15 or the DSP 18 performs an operation using the intermediate layer 46[j] of the final stage formed in the SPM 16, i.e., the feature map FMj, and the weight parameter set WTSo stored in the main memory 19 as inputs, and stores the operation result as the feature map FMo in the SPM 16. Thus, the output layer 47 is formed on the SPM 16. (F) Finally, the DMAC 17 transfers the output layer 47 formed on the SPM 16, i.e., the feature map FMo, to the main memory 19 as an image processing result.

[0061] Figure 3 This diagram is used for Figure 1 Schematic diagram of an exemplary process of an intermediate layer in a CNN in a semiconductor device shown. The intermediate layer to be processed, i.e., the convolutional layer, is provided with N convolutional layers from the previous stage. i Input channels CHi[0] to CHi[N i -1] feature maps FMi[0] to FMi[N i -1], and the N convolutional layers assigned to be processed o The weight parameter sets WTS[0] to WTS[N o -1].

[0062] Feature maps FMi[0] to FMi[N i -1] is stored in the SPM 16, and weight parameter sets WTS[0] to WTS[N o -1] is stored in the main memory 19. The feature map FM of each channel has a size of W×H, where W is the size in the width direction and H is the size in the height direction. The weight parameter sets WTS[0] to WTS[N o -1] each weight parameter set includes N kw ×N kh ×N i weight parameter WT, where N kw is the number in the width direction, N kh is the number in the height direction, and N i is the number of input channels. N kw ×N kh is the kernel size, usually 3×3 etc.

[0063] Here, in Figure 3 In the embodiment, the multiplication-accumulation calculator MAC[0] is based on the features included in the feature maps FMi[0] to FMi[N i-1] and the weight parameter set WTS[0] of the output channel CHo[0], a multiplication and accumulation operation is performed on a pixel data set PDS of a predetermined size. The addition of the bias parameter BS and the operation of the activation function are performed on the result of the multiplication and accumulation operation, so that the pixel data PDo of the pixel position serving as a reference in the feature map FMo[0] of the output channel CHo[0] is generated. Note that the pixel data set PDS includes N in the same way as the weight parameter set WTS. kw ×N kh ×N i pixel data PDi.

[0064] In addition, in parallel with the operation in the multiplication-accumulation calculator MAC[0], the multiplication-accumulation calculator MAC[N o -1] for the same pixel data set PDS as used by the multiply-accumulate calculator MAC[0] and a different output channel CHo[N] than the case of the multiply-accumulate calculator MAC[0] o -1] weight parameter set WTS[N o -1] performs a multiplication-accumulation operation. The result of the multiplication-accumulation operation is subjected to the addition of the bias parameter BS and the operation of the activation function, so that the output channel CHo[N o -1] feature map FMo[N o -1] serves as the pixel data PDo of the pixel position serving as a reference.

[0065] Then, the above process is performed while sequentially moving the reference pixel positions in the width direction or the height direction, so that the pixels constituting N o feature maps FMo[0] to FMo[N o -1] all pixel data PDo. N o feature maps FMo[0] to FMo[N o -1] is stored as image data DT in the SPM 16. For example, the image data DT stored in the SPM 16 in this way is input to the convolution layer of the next stage and is used as N i =N o The feature maps FMi[0] to FMi[N i -1]. Note that Figure 3 The convolution process shown is represented by expression (1).

[0066] [Expression 1]

[0067]

[0068]

[0069] FMo: Feature Map (output)

[0070] FMi: Feature map (input)

[0071] WT: weight parameter

[0072] [x]: Pixel position in the X direction (horizontal)

[0073] [y]: Line position in the Y direction (vertical)

[0074] [kw]: kernel position in the X direction (horizontal)

[0075] [kh]: Kernel position in the Y direction (vertical)

[0076] N o : The number of output channels

[0077] N i : The number of input channels

[0078] N kw : Kernel size in the X direction (horizontally)

[0079] N kh : Kernel size in the Y direction (vertical)

[0080] W: Pixel size in the X direction (horizontally)

[0081] H: Line size in the Y direction (vertical)

[0082] Comparison Examples

[0083] Fig.19 1 is a diagram illustrating an example of memory mapping of image data of corresponding channels stored in a temporary storage memory (SPM) in a semiconductor device, as a first comparative example. In this case, the SPM 16 has 8 memories MR[0] to MR[7], and the corresponding memory MR has a bit width of 16 bytes (=128 bits). That is, the memory bus width of the SPM 16 is 128 bytes (=16 bytes×8).

[0084] Here, the size of image data of one channel is 640 bytes, which is composed of 64 bytes / row×10 rows in a grid structure. In this case, for example, when the image data DT of 8 channels CH[0] to CH[7] are stored in the SPM 16 in a grid order by using a plane format which is one of the general formats, the image data DT is formed as follows: Fig.191. The memory mapping shown in FIG. 1. For example, the plurality of pixel group data PGD0 to PGD3 in the corresponding channel corresponds to the data of the first row in the grid structure. The plurality of pixel group data PGD4 to PGD7 corresponds to the data of the second row in the grid structure. In this specification, the plurality of pixel group data PGD0 to PGD7 are collectively referred to as pixel group data PGD.

[0085] Fig.19 The image data DT of the eight channels CH[0] to CH[7] in FIG. Figure 3 The pixel group data PGD0 included in the image data DT of the channel CH[0] corresponds to the feature maps FMi[0] to FMi[7] of the 8 channels serving as input data, or the feature maps FMo[0] to FMo[7] of the 8 channels serving as output data. Figure 3 The pixel group data PGD1 corresponds to a data group of 16 bytes (e.g., 16 pixels) in the width direction in the feature graph FMi[0] shown. The subsequent pixel group data PGD1 corresponds to a data group of 16 bytes after the pixel group data PGD0 in the width direction. As described above, the pixel group data PGD includes multiple or single pixel data PDi and is the minimum data unit when the memory controller 29 accesses the SPM 16.

[0086] Figure 3 The pixel data set PDS shown is Figure 1 The data group output from the line buffer 27 to the MAC unit 25 corresponds to the data group output from the line buffer 27 to the MAC unit 25. Specifically, the line buffer 27 outputs, for example, multiple pixel data sets PDS for convolution operations in the width direction, i.e., shifted pixel data sets PDS, to the MAC unit 25 in one clock cycle. The MAC unit 25 can also perform operations on multiple pixel data sets PDS in parallel in one clock cycle.

[0087] The line buffer 27 sequentially switches the positions of the plurality of pixel data sets PDS output to the MAC unit 25 per clock cycle in the width direction or the height direction. As the positions are switched, in addition to the pixel data PDi already acquired in the line buffer 27, in other words, in addition to the pixel data PDi that can be reused according to the convolution operation, a plurality of new pixel data PDi are required.

[0088] The read access controller 30 transfers the newly required plurality of pixel data PDi, for example, pixel group data PGD of 8 channels CH[0] to CH[7], from the SPM 16 to the line buffer 27. With this process, in a steady state, data transfer from the SPM 16 to the line buffer 27, data transfer from the line buffer 27 to the MAC unit 25, and a MAC operation in the MAC unit 25 are all processed in the pipeline.

[0089] here, Fig.19 The logical address Laddr of SPM 16 in is represented by expression (2), using the physical addresses of the M memories MR[0] to MR[M-1], such as the word address WDaddr, and an index (idx) having a range of 0 to M-1, which is used to identify the M memories MR[0] to MR[M-1].

[0090] Laddr=WDaddr×128+(idx×16)…(2)

[0091] Therefore, in Fig.19 In the memory mapping shown, N is an integer of 2 or greater, and the pixel data PD of the N channels arranged at the same pixel position in the image data DT of the N channels, or the pixel group data PGD, is stored in the same memory MR among the M memories MR. For example, the pixel group data PGD0 of the N channels arranged at the same pixel position is stored in the same memory MR[0].

[0092] Therefore, if Figure 3 As described above, when the pixel group data PGD of the N channels stored in the SPM 16 is input to the line buffer 27, it is necessary to read access to the same memory MR in a time-division manner. In addition, when the pixel data PDo of the N channels arranged at the same pixel position and obtained by a plurality of multiplication-accumulation calculators MAC and the like are output to the SPM 16, it is necessary to write access to the same memory MR in a time-division manner. Therefore, the input / output delay between the SPM 16 and the NNE 15 (particularly the MAC unit 25) increases, and there is a possibility that the processing time of the image processing cannot be sufficiently shortened.

[0093] Here, for example, Resnet50 is a well-known neural network model that has 50 layers. In the upstream intermediate layer, N i (Number of input channels) = 64, N o (number of output channels) = 256, W (X size) = 112, H (Y size) = 112, N kw (kernel size in X direction) = 1, and N kh (kernel size in Y direction) = 1. When these are applied to the above expression (1), 205520896 (= 64 × 256 × 112 × 112 × 1 × 1) multiplication-accumulation operations are required. Therefore, it is desirable to increase the parallelism of the multiplication-accumulation calculator MAC.

[0094] For example, the degree of parallelism in the input channel, the output channel, the X-direction pixel, the Y-direction pixel, the X-direction kernel, or the Y-direction kernel depends on the architecture. However, considering that raster processing is a general hardware process, increasing the degree of parallelism in the channel direction is an important requirement for the hardware. In particular, when the degree of parallelism in the channel direction increases, since the input / output delay between the SPM 16 and the NNE 15 greatly affects the effective performance, a technique for reducing the input / output delay is required.

[0095] Fig. 20 is used to explain the use of Fig.19 Schematic diagram of the impact on input / output latency that occurs throughout image processing using a neural network, given the memory mapping shown. Fig. 20 shows a schematic process of a semiconductor device, wherein Figure 2 The neural network with 5 intermediate layers shown is a process target. In this case, the semiconductor device sequentially performs a DMAC process [1], an NNE process [1], an NNE process [2], a DSP process, an NNE process [3] to an NNE process [5], and a DMAC process [2]. The DMAC process [1] is a process related to the input layer, the NNE process [1] to the NNE process [4] and the DSP process are processes related to the intermediate layer, and the NNE process [5] and the DMAC process [2] are processes related to the output layer.

[0096] In the DMAC process [1], the DMAC 17 transfers the camera image CIMG stored in the main memory 19 to the SPM 16. In the NNE process [1], the NNE 15 receives the camera image CIMG stored in the SPM 16 and performs signal processing, thereby generating a feature map FM1 and outputting the feature map FM1 to the SPM 16. In the NNE process [2], the NNE 15 receives the feature map FM1 stored in the SPM 16 and performs signal processing, thereby generating a feature map FM2 and outputting the feature map FM2 to the SPM 16.

[0097] In the DSP process, the DSP 18 receives the feature map FM2 stored in the SPM 16 and performs signal processing, thereby generating a feature map FM3 and outputting the feature map FM3 to the SPM 16. From the NNE process [3] to the NNE process [5], the NNE 15 receives the feature map FM of the intermediate layer in the previous stage stored in the SPM 16 and performs signal processing, thereby generating a feature map FM of the intermediate layer in the subsequent stage, and outputting the generated feature map FM to the SPM 16. In the DMAC process [2], the DMAC 17 transfers the feature map FMo of the output layer stored in the SPM 16 to the main memory 19.

[0098] When performing this process, the Fig.19 As shown in the memory mapping, the input / output delay associated with the input / output from the NNE process [1] to the NNE process [5] increases. In addition, in the DSP process, the input / output delay increases according to the content of the process. Therefore, the effective performance relative to the theoretical performance is greatly reduced, and the processing time of the image processing using the neural network can be increased.

[0099] Fig.21 FIG. 1 is a schematic diagram for explaining the influence on input / output delay occurring in the entire image processing using a neural network in a semiconductor device as a second comparative example. Fig. 20 The input / output delay described above may be implemented using a dedicated data format, for example by integrating multiple channels for data transmission between the SPM 16 and the NNE 15. Specifically, Fig.21 As shown, a dedicated data format is applied to the feature maps FM1 , FM4 , and FM5 stored in the SPM 16 .

[0100] However, general-purpose signal processing circuits such as the DSP 18 and the DMAC 17 cannot support the dedicated data format, and need to use a signal processing circuit such as Fig.19 The general format of the flat format shown in . Therefore, if Fig.21 As shown, even when a dedicated data format is used, input / output delays are increased when input in NNE process [1] and NNE process [3], and when output in NNE process [2] and NNE process [5]. Therefore, there is a possibility that the processing time of image processing cannot be sufficiently shortened.

[0101] Memory Controller

[0102] therefore, Figure 1 The illustrated memory controller 29 operates as follows. Figure 4 It is used to explain Figure 1 1 and 2 , and is a diagram illustrating an example of memory mapping of image data of corresponding channels stored in a temporary memory (SPM). The memory controller 29 controls access to the SPM 16 so that pixel group data PGD of N channels arranged at the same pixel position and thus pixel data PD in image data DT of N channels are stored in different memories MR among M memories MR[0] to MR[M-1], respectively.

[0103] exist Figure 4 In the case of Fig.19 In the case of , SPM 16 has 8 memories MR[0] to MR[7], and the corresponding memory MR has a bit width of 16 bytes (=128 bits). Fig.19 As shown, the size of the image data DT of one channel is 640 bytes (=64 bytes / row×10 rows). The SPM 16 stores the image data DT of eight channels CH[0] to CH[7] according to the plane format.

[0104] However, in Figure 4 In the case of Fig.19 Unlike the case of , a blank area BLNK is provided between the storage area of ​​the last pixel group data PGD39 in one channel and the storage area of ​​the first pixel group data PGD0 in another channel of two adjacent channels. In this case, the size of the blank area BLNK is 16 bytes. For example, the memory controller 29 defines a starting address, that is, an address of the storage area of ​​the first pixel group data PGD0, for each of the 8 channels CH[0] to CH[7], so that such a blank area BLNK is provided.

[0105] Therefore, for example, the first pixel group data PGD0 in the eight channels CH[0] to CH[7] is stored in the eight memories MR[0] to MR[7]. The same applies to the remaining pixel group data PGD, for example, the last pixel group data PGD39 in the eight channels CH[0], CH[1] to CH[7] is stored in the memories MR[7], MR[0] to MR[6], respectively.

[0106] Figure 5A It shows Figure 1 A timing diagram comparing a schematic operation example of the neural network engine (NNE) in FIG. 1 with a case where the system uses the first comparative example and a case where the system uses the embodiment. Figure 5B : is a diagram illustrating an example of a case where the number of clock cycles required to process a convolution layer of one layer is compared between a case where the method of the first comparative example is used and a case where the method of the embodiment is used.

[0107] The neural network engine (NNE) 15 repeatedly executes Figure 5A The processing cycle Tcyc shown is to process, for example, one convolution layer. In this case, when applying the first comparative example, that is, Fig.19 In the memory mapping shown, the pixel group data PGD at the same pixel position in each channel needs to be read out from the same memory MR, and needs to be written into the same memory MR after multiplication and accumulation operation by the multiplication and accumulation calculator MAC. Therefore, each time the read target or write target channel is switched, the SPM 16 needs to perform a time-division read operation and a time-division write operation while providing a waiting phase Tw.

[0108] On the other hand, when the application embodiment, i.e. Figure 4When the memory mapping is shown, the pixel group data PGD at the same pixel position in each channel can be read from different memories MR, and can be written into different memories MR after the multiplication and accumulation calculator MAC performs multiplication and accumulation operations. Therefore, unlike the first comparative example, there is no need to provide a waiting time Tw. In other words, the pixel group data PGD at the same pixel position can be read and written to different memories MR at the same time. Therefore, the input / output delay can be shortened.

[0109] Here, as a specific example, it is assumed that the MAC unit 25 can process 32 input channels and 32 output channels in one clock cycle, and can process 2 pixels in the X direction in the clock cycle, that is, it can perform 2048 (=32×32×2) convolution operations in parallel in one clock cycle. It is assumed that the delay of the input and output of the channel is 50 clock cycles. Figure 5B In FIG. 1 , the number of clock cycles required to process one convolution layer is compared between the first comparative example and the embodiment under the premise of using the MAC unit 25.

[0110] For example, in a typical CNN network model such as Resnet50, in the upstream convolution layer, that is, in the upstream layer, the X / Y size of the image data DT is large. As the processing continues to the downstream layer, the number of channels of the image data DT increases, while the X / Y size of the image data DT decreases. Figure 5B In Example 1 shown, W (X size) = 112, H (Y size) = 112, N i (Number of input channels) = 64, N o (Number of output channels) = 256, N kw (X size of kernel) = 1, and N kh (kernel Y size) = 1. In Example 2, assuming the downstream layer case, W = 7, H = 7, and N i =512,N o =2048,N kw =1,N kh =1.

[0111] Further, in Figure 5B In the formula (3), the theoretical performance TP is calculated by the formula (3). The effective performance AP_C of the method according to the first comparative example is calculated by the formula (4). The effective performance AP_E of the method according to the embodiment is calculated by the formula (5). Note that CEIL() is a function that rounds up the value in the brackets to an integer.

[0112] TP=(N i / 32)×(N o / 32)×CEIL(W / 2)×H…(3)

[0113] AP_C=(N i / 32)×(N o / 32)×{(CEIL(W / 2)×H)+50+(32-1)

[0114] +(32-1)}…(4)

[0115] AP_E=(N i / 32)×(N o / 32)×{(CEIL(W / 2)×H)+50}…(5)

[0116] Here, in Figure 5A and Figure 5B In one processing cycle Tcyc shown, the MAC unit 25 performs 2048 convolution operations in one clock cycle, and uses CEIL(W / 2)×H clock cycles to process the image data DT of 32 input channels and the image data DT of 32 output channels. For example, in Example 1, since N i (number of input channels) = 64 and N o (The number of output channels)=256, so the number of processing cycles Tcyc required to process the image data DT of all input channels and the image data DT of all output channels is 16 (=64 / 32×256 / 32).

[0117] In Expression (4) and Expression (5), compared with the effective performance AP_E of the method according to the embodiment, (N i / 32)×(N o / 32)×{(32-1)+(32-1)} is added to the effective performance AP_C of the method according to the first comparative example. Figure 5A , the overhead associated with 32 channels - 1 clock cycle is added at the input and output times for each processing cycle Tcyc, where the waiting stage Tw is set to one clock cycle. The number of processing cycles Tcyc required to process the convolutional layer increases as the number of channels increases. Therefore, in the method of the first comparative example, in particular, the deviation of the effective performance AP_C from the theoretical performance TP becomes larger in the downstream layer. When the method of the embodiment is used, the deviation can be suppressed. In particular, an improvement effect of 44.3% (=1-(79872 / 143360)) can be obtained in the downstream layer.

[0118] Figure 6 is used to explain the use of Figure 4A schematic diagram of the effect of input / output latency on the overall image processing using a neural network, based on the memory mapping shown. Fig. 20 or Fig.21 Different, such as Figure 6 As shown, the input / output delay can be reduced from the NNE process [1] to the NNE process [5] and all the input / output associated with the DSP processing. In this case, since a planar format as a general format is used instead of a dedicated data format, the input / output delay can be shortened even when the DSP processing is particularly included.

[0119] Configuration of SPM and white space

[0120] Figure 7 It is a graphic Figure 1 16 includes, for example, 2 m (=M) number of memories MR, where m is an integer of 1 or greater than 1. The bit width of the corresponding memory MR is, for example, 2 k Bytes, where k is an integer of 0 or greater than 0. Here, it is assumed that the size FS of the image data for each channel is GS×an even number, where GS is the size of the pixel group data PGD.

[0121] Figure 7 The illustrated examples 1-1 to 1-5 show examples in which 32 (=2 5 ) memories MR to configure the SPM 16 with a memory bus width of 512 bytes. In Example 1-1, the size GS of the pixel group data PGD is 16 bytes, and one pixel group data PGD is stored in one memory MR. In this case, when using Figure 4 In the same method as in the case of FIG. 1 , the number of input channels or the number of output channels that can be processed in parallel is 32 (=512 / 16).

[0122] In addition, the size of the corresponding blank area BLNK can be 16 bytes × (a number that is prime to 32 channels), that is, 16 bytes × an odd number, with 16 bytes as a unit, and the 16 bytes are the size GS of the pixel group data PGD. In addition, from the perspective of making the size of the blank area BLNK as small as possible, that is, saving memory, it is expected that the size of the blank area BLNK is 16 bytes × 1.

[0123] In Example 1-2, the size GS of the pixel group data PGD is 32 bytes, and one pixel group data PGD is stored in two memories MR. Figure 4In the same method as in the case of FIG. 5 , the number of input channels or the number of output channels that can perform parallel processing is 16 (=512 / 32). In addition, the size of the corresponding blank area BLNK can be 32 bytes×(a number that is a prime number relative to the number of channels 16), that is, 32 bytes×an odd number, with 32 bytes as a unit, and the 32 bytes are the size GS of the pixel group data PGD, and preferably 32 bytes×1.

[0124] In Example 1-3, the size GS of the pixel group data PGD is 64 bytes, and one pixel group data PGD is stored in four memories MR. Figure 4 In the same method as in the case of FIG. 5 , the number of input channels or the number of output channels that can perform parallel processing is 8 (=512 / 64). In addition, the size of the corresponding blank area BLNK can be 64 bytes×(a number that is prime to the number of channels 8), that is, 64 bytes×an odd number, with 64 bytes as a unit, and the 64 bytes are the size GS of the pixel group data PGD, and preferably 64 bytes×1.

[0125] Similarly, in example 1-4, size GS is 128 bytes, and the number of input / output channels that can be processed in parallel is 4 (=512 / 128). The size of the corresponding blank area BLNK can be 128 bytes × (a number that is prime to the number of channels 4), i.e. 128 bytes × odd number, and wherein preferably 128 bytes × 1. In example 1-5, size GS is 256 bytes, and the number of input / output channels that can be processed in parallel is 2 (=512 / 256). The size of the blank area BLNK can be 256 bytes × (a number that is prime to the number of channels 2), i.e. 128 bytes × odd number, and therefore 128 bytes × 1.

[0126] Figure 7 The examples 2-1 to 2-3 shown in the figure show an example in which 8 (=2 3 ) memories MR to configure the SPM 16 with a memory bus width of 128 bytes. In Example 2-1, the size GS of the pixel group data PGD is 16 bytes, and one pixel group data PGD is stored in one memory MR. That is, Example 2-1 is the same as Figure 4 The configuration shown corresponds. The number of input channels or output channels that can be processed in parallel is 8 (=128 / 16). In addition, the size of the corresponding blank area BLNK can be 16 bytes×(a number that is mutually prime with the number of channels 8), i.e. 16 bytes×odd number, and preferably 16 bytes×1.

[0127] Similarly, in Example 2-2, size GS is 32 bytes, and the number of input / output channels that can be processed in parallel is 4 (=128 / 32). The size of the blank area BLNK can be 32 bytes × (a number that is prime to the number of channels 4), i.e. 32 bytes × odd number, and wherein preferably 32 bytes × 1. In Example 2-3, size GS is 64 bytes, and the number of input / output channels that can be processed in parallel is 2 (=128 / 64). The size of the blank area BLNK can be 64 bytes × (a number that is prime to the number of channels 2), i.e. 64 bytes × odd number, and therefore 64 bytes × 1.

[0128] When summarizing the above, it is assumed that the SPM 16 includes K(=2 k ) bytes of M(=2 m ) memory MR, the size GS of the pixel group data PGD is determined to be 2 (k+a) bytes, and the number of channels N to be processed in parallel is determined to be 2 (m-a) In addition, the blank area BLNK is defined as 2 (k+a) Bytes × (relative to the number of channels 2 (m-a) is a prime number). Note that a is 0 or an integer greater than 0 and less than m. In addition, the generalized logical address Laddr of the SPM 16 is given by expression (6). As in expression (2), WDaddr is the word address of each memory MR, and idx is the identification number of each memory MR.

[0129] Laddr=WDaddr×(M×K)+(idx×K)…(6)

[0130] For example, refer to Figure 7 and Figure 4 Example 2-1, K(=2 k ) is 16(=2 4 ) bytes, M(=2 m ) is 8(=2 3 ), GS is 16 (=2 (4+0) ) bytes, and N is 8 (=2 (3-0) ). In addition, the blank area BLNK is set to 16 (=2 (4+0) ×1).

[0131] Figure 8 It is a graphic with Figure 4 A diagram showing examples of different memory mappings, which are Figure 7 This corresponds to Example 2-2 in the example. Fig. 22 It is shown as Figure 8 FIG. 1 is a diagram showing an example of a memory map of a comparative example of FIG. Figure 8 and Fig. 22In the case shown, the SPM 16 has 8 memories MR[0] to MR[7]. Each memory MR has a bit width of 16 bytes. That is, the memory bus width of the SPM 16 is 128 bytes (=16 bytes×8). The size GS of the pixel group data PGD is 32 bytes.

[0132] Here, the size of the image data of one channel is 768 bytes, which is composed of 96 bytes / row×8 rows in the grid structure. In this case, when the method of the comparative example is used, the following is formed: Fig. 22 The memory map is shown in Fig. 22 , a plurality of pixel group data PGD0a, PGD0b, PGD1a, PGD1b, PGD2a, PGD2b correspond to the data of the first row in the grid structure, and a plurality of pixel group data PGD3a, PGD3b, PGD4a, PGD4b, PGD5a, PGD5b correspond to the data of the second row in the grid structure. The number of channels of the image data is 4.

[0133] like Fig. 22 As shown, in the method of the comparative example, as Fig.19 In the case of the above, the pixel group data PGD of the four channels CH[0] to CH[3] arranged at the same pixel position (for example, 32-byte pixel group data PGD0a, PGD0b) are stored in the same memory pair (MR[0], MR[1]). On the other hand, in the method of the embodiment, as Figure 8 As shown, a 32-byte blank area BLNK is provided. Therefore, the pixel group data PGD of the four channels CH[0] to CH[3] arranged at the same pixel position (for example, 32-byte pixel group data PGD0a, PGD0b) are stored in different memory pairs (MR[0], MR[1]), (MR[2], MR[3]), (MR[4], MR[5]), (MR[6], MR[7]).

[0134] Details of the main parts of a semiconductor device

[0135] Fig. 9 It is a graphic Figure 1 A block diagram showing a detailed configuration of the main parts of the semiconductor device shown. Fig. 9 Mainly shows Figure 1 Detailed configuration examples of the memory controller 29 and the neural network software system 40 are shown in FIG. Fig. 9 As shown, SPM 16 may include, for example, arbitration circuitry that arbitrates multiple accesses to the same MR.

[0136] also, Fig. 9The activation function calculation unit 70 shown performs multiplication and accumulation operations on the results of the MAC unit 25, such as addition of the bias parameter BS and calculation of the activation function. The pooling processing unit 71 performs pooling processing as needed. The activation function calculation unit 70 and the pooling processing unit 71 are Figure 1 The details of the memory controller 29 and the neural network software system 40 will be described below.

[0137] Memory Controller Details

[0138] exist Fig. 9 In the embodiment, the read access controller 30a and the write access controller 31a are included in Figure 1 The memory controller 29 of the embodiment of the present invention is shown in FIG. 30 . The read access controller 30 a includes a read base address register 50 , a channel stride register 51 , an address counter 52 , an adder 53 , a read address generator 54 , a read address router 55 , a read data router 56 , and a pending address buffer 57 .

[0139] In general, when reading the pixel group data PGD of N channels and thus reading the pixel data PD from the SPM 16, the read access controller 30a generates the read logical addresses of the N channels in parallel, in which the pixel data PD of the N channels are stored. In addition, the read access controller 30a converts the generated read logical addresses of the N channels into read physical addresses of the M memories MR in parallel, and outputs the read logical addresses to the SPM 16. In addition, the read access controller 30a rearranges the pixel data PD of the N channels read from the SPM 16 in channel order, and outputs them to the MAC unit 25 in parallel.

[0140] Specifically, in the read base address register 50 for reading, the start address for reading the image data DT of N channels stored in the SPM 16 is set as the base address. Figure 4 In the SPM 16, the start address of the channel CH[0] is set. Generally, in the SPM 16, the read address space for input to the MAC unit 25 and the write address space for output from the MAC unit 25 are set separately. The read base address register 50 defines the location of the read address space in the SPM 16.

[0141] The address counter 52 generates the scanning address Saddr by counting sequentially from 0 in units of the size GS of the pixel group data PGD. Figure 4In the case of , 16 bytes are used as the unit. The adder 53 generates the reference logical address Raddr by adding the base address from the read base address register 50 to the scan address Saddr from the address counter 52. Therefore, in the read address space, for example, the reference logical address Raddr is generated so that the scanned addresses PGD0, PGD1, ..., PGD7, PGD8, ... are scanned in sequence. Figure 4 Pixel group data PGD of the middle channel CH[0].

[0142] In the channel stride register 51 for reading, the address interval between adjacent channels of the corresponding start addresses of the image data DT of N channels stored in the SPM 16 is set as the channel stride. Figure 4 In the example, the address interval between the logical address of the pixel group data PGD0 in the channel CH[0] and the logical address Laddr of the pixel group data PGD0 in the channel CH[1] is set, specifically 640+16 bytes.

[0143] The read address generator 54 adds an integer multiple of the address interval set in the channel stride register 51 to the reference logical address Raddr input from the adder 53, thereby generating the read logical addresses CH[n]_REaddr of the N channels in parallel (in other words, in the same clock cycle). That is, based on expression (7), the read address generator 54 generates the read logical addresses CH[0]_REaddr-CH[7]_REaddr of the N channels. Figure 4 In the expression (7), CHstride is the address interval set in the channel stride register 51, and n is an integer from 0 to N-1.

[0144] CH[n]_REaddr=Raddr+CHstride×n…(7)

[0145] Specifically, if N=8, CHstride=656 and Raddr=0, the read address generator 54 generates CH[0]_REaddr=0, CH[1]_REaddr=656, ..., CH[7]_REaddr=4592 in parallel. Figure 4 In the process, the logical address Laddr of the pixel group data PGD0 in the eight channels CH[0] to CH[7] is generated in parallel.

[0146] The read address router 55 converts the read logical addresses CH[n]_REaddr of the N channels generated in parallel by the read address generator 54 into read physical addresses MR_idx[n]_REaddr for the memory MR corresponding to each channel in the M memories MR in parallel. The read address router 55 outputs the converted read physical addresses MR_idx[n]_REaddr in parallel to the memory MR corresponding to each channel. The details of the read address router 55 will be described later.

[0147] In response to the read physical address MR_idx[n]_REaddr from the read address router 55, the read data router 56 rearranges the pixel data PD of N channels read from the memory MR corresponding to each channel in channel order, in detail, arranges the memory read data MR_idx[n]_REdat in memory order. The pending address buffer 57 is provided for performing the rearrangement in the read operation.

[0148] The read data router 56 outputs the channel read data CH[n]_REdat obtained by the rearrangement in parallel to the MAC unit 25. Specifically, the read data router 56 stores the channel read data CH[n]_REdat in parallel in the line buffer 27 having a storage area in channel order, and outputs the data to the MAC unit 25 via the line buffer 27. Details of the read data router 56 and the pending address buffer 57 will be described later.

[0149] The write access controller 31a includes a write base address register 60, a channel stride register 61, an address counter 62, an adder 63, a write address generator 64, a write address router 65, and a write data router 66. The operations of the write base address register 60, the channel stride register 61, the address counter 62, the adder 63, and the write address generator 64 are the same as the operations of the read base address register 50, the channel stride register 51, the address counter 52, the adder 53, and the read address generator 54 described above.

[0150] Therefore, when the pixel group data PGD of the N channels obtained based on the multiplication-accumulation operation performed by the MAC unit 25 and the pixel data PD are written to the SPM 16, the write access controller 31a generates the write logical addresses of the N channels for storing the pixel data PD of the N channels respectively in parallel. In addition, the write access controller 31a converts the generated write logical addresses of the N channels into write physical addresses of the memory MR corresponding to each channel in parallel. Then, the write access controller 31a outputs the write physical addresses in parallel to the memory MR corresponding to each channel together with the pixel data PD of the N channels obtained based on the multiplication-accumulation operation performed by the MAC unit 25.

[0151] At this time, the write address router 65 converts the write logical addresses CH[n]_WRaddr of the N channels generated in parallel by the write address generator 64 into write physical addresses MR_idx[n]_WRaddr for the memory MR corresponding to each channel in the M memories MR in parallel. Then, the write address router 65 outputs the converted write physical addresses MR_idx[n]_WRaddr in parallel to the memory MR corresponding to each channel. The details of the write address router 65 will be described later.

[0152] On the other hand, the write data router 66 outputs the pixel data PD of the N channels obtained based on the multiplication-accumulation operation performed by the MAC unit 25, and in detail, the channel write data CH[n]_WRdat stored in the write buffer 28 having the storage area in the channel order is output in parallel to the memory MR corresponding to each channel. At this time, the write data router 66 rearranges the channel write data CH[n]_WRdat arranged in the channel order in the memory order. Then, the write data router 66 outputs the memory write data MR_idx[n]_WRdat obtained by the rearrangement to the memory MR corresponding to each channel. The details of the write data router 66 will be described later.

[0153] Address router details

[0154] Fig. 10A It is a graphic Fig. 9 The diagrams show an example of the configuration of an address router and an example of some operations of the address router. FIG. 10B to FIG. 10E Is used to describe Fig. 10A10A corresponds to the read address router 55 or the write address router 65. Here, it is assumed that the SPM 16 includes 8 memories MR[0] to MR[7], each of which has 64 k bytes. With this configuration, the SPM 16 has a capacity of 512 k bytes. It is assumed that the corresponding memory MR has 4096 word addresses WDaddr and a bit width of 16 bytes. That is, for example, it is assumed that Figure 7 and Figure 4 In the example 2-1 shown, the word address WDaddr is 4096.

[0155] The address router receives the logical addresses CH[0]_addr[18:4] to CH[7]_addr[18:4] of the eight channels CH[0] to CH[7] in parallel from the address generators 54 and 64. In the case of the read address router 55, the logical address CH[n]_addr corresponds to the boot logical address CH[n]_REaddr. In the case of the write address router 65, the logical address CH[n]_addr corresponds to the write logical address CH[n]_WRaddr.

[0156] The address router outputs the physical addresses MR_idx[0]_addr[11:0] to MR_idx[7]_addr[11:0] of the eight memories MR[0] to MR[7] in parallel. In the case of the read address router 55, the physical address MR_idx[n]_addr corresponds to the read physical address MR_idx[n]_REaddr. In the case of the write address router 65, the physical address MR_idx[n]_addr corresponds to the write physical address MR_idx[n]_WRaddr. Specifically, the physical address MR_idx[n]_addr corresponds to, for example, Figure 4 Corresponding to the word address WDaddr in.

[0157] In this case, since the access is performed in units of 16 bytes, the lower 4 bits ([3:0]) in the logical address CH[n]_addr are fixed to 0. Therefore, the 4th to 18th bits ([18:4]) of the logical address CH[n]_addr are input to the address router. The 8 memories MR[0] to MR[7] correspond to the 8 indexes idx[0] to idx[7], respectively. The 8 indexes idx[0] to idx[7] are assigned to the 4th to 6th bits ([6:4]), which are the low-order side bits in the logical address CH[n]_addr. Therefore, the logical address CH[n]_addr[6:4] can identify the correspondence between the channel and the memory MR.

[0158] For example, it is assumed here that a read operation is performed from the SPM 16 at a certain processing cycle Tcyc[t]. At this time, the logical addresses CH[0]_addr[18:4] to CH[7]_addr[18:4] of the 8 channels input in parallel to the address router are different from each other at the 4th to 6th bits ([6:4]), depending on the operation of the read address generator 54 based on the channel stride register 51. The address router identifies the memory MR corresponding to each channel by the logical addresses CH[0]_addr-CH[7]_addr of the 8 channels, which in this case are a specific bit region from the 4th bit to the 6th bit ([6:4]).

[0159] exist Fig. 10A In the illustrated embodiment, when n=0, 1, 2, 3, 4, 5, 6, 7, the logical address CH[n]_addr[6:4] of the channel CH[n] represents the indexes idx[5], idx[2], idx[7], idx[4], idx[1], idx[6], idx[3], idx[0], and thus represents the memories MR[5], MR[2], MR[7], MR[4], MR[1], MR[6], MR[3], and MR[0]. That is, Figure 4 Unlike the case of , here is a case where the size of the image data DT of one channel is (8p+4)×16 bytes, where p is an integer. Figure 4 In , the corresponding channels CH[0] to CH[7] are configured by, for example, pixel group data PGD0-PGD43. Figure 4 In , the size of image data DT of one channel is (8p+0)×16 bytes.

[0160] or, Fig. 10A A case is shown where the size of the image data DT of one channel is (8p+5)×16 bytes, and the channel stride (CHstride) is also set to (8p+5)×16 bytes. Then the blank area BLNK is not provided. That is, if the size of the image data DT of one channel is (8p+odd number)×16 bytes, the blank area BLNK is not required.

[0161] The address router identifies the memory MR[q] as the output destination of the bit region from the 7th bit to the 18th bit ([18:7]) on the high-order side of the logical address CH[n]_addr[18:4] by using the index value (idx) indicated by the 4th bit to the 6th bit ([6:4]) of the logical address CH[n]_addr. Then, the address router outputs the logical address CH[n]_addr[18:7] of the channel CH[n], which is the bit region to be output, to the memory MR[q] identified by the index value (idx) as the 12-bit physical address MR_idx[q]_addr[11:0].

[0162] Using this operation, Fig. 10A In the illustrated embodiment, because the index value (idx) is 5, the logical address CH[0]_addr[18:7] of the channel CH[0] is output to the memory MR[5] as the physical address MR_idx[5]_addr[11:0]. Because the index value (idx) is 2, the logical address CH[1]_addr[18:7] of the channel CH[1] is output to the memory MR[2] as the physical address MR_idx[2]_addr[11:0]. The same applies to the remaining channels CH[2] to CH[7].

[0163] In the read operation from the SPM 16 in the subsequent processing cycle Tcyc[t+1], the index value (idx) of the 4th to 6th bits ([6:4]) is incremented by +1 by the operation of the address counter 52. Therefore, the output destinations of the logical addresses CH[n]_addr[18:7] of channel[0], channel[1], channel[2], channel[3], channel[4], channel[5], channel[6], and channel[7] are switched to the memories MR[6], MR[3], MR[0], MR[5], MR[2], MR[7], MR[4], and MR[1], respectively. That is, referring to Figure 4 For example, in a certain processing cycle, the pixel group data PGD of channel CH[0] is read from the word address "A" of the memory MR[5], and in the next processing cycle, the next pixel group data PGD of channel CH[0] is read from the same word address "A" of a different memory MR[6].

[0164] Specifically, the address router includes, for example, a selector, a matrix switch, etc., which determines the connection relationship between the input signals of the N channels and the output signals to the M memories MR. In this example, the input signal is the logical address CH[n]_addr[18:7] of the 8 channels, and the logical address CH[n]_addr[18:7] is the bit area of ​​the logical address. The output is the physical address MR_idx[n]_addr[11:0] to the 8 memories MR.

[0165] The logical address CH[n]_addr[6:4] of the 8 channels is another bit area of ​​the logical address, which is used as a selection signal of a selector or a matrix switch. With this configuration, the address router can process the input signals of N channels in parallel (in other words, in one clock cycle) and output the processed signals as output signals in parallel to the M memories MR. Therefore, the input / output delay can be shortened.

[0166] exist Fig. 10B In the operation example shown in FIG. 1 , the size of the image data DT of one channel is (8p+0)×16 bytes, where p is an integer, and therefore the channel stride (CHstride) is set to (8p+1)×16 bytes. Figure 4 An operation example when a 16-byte × 1 blank area BLNK is provided at the position shown. Alternatively, an operation example when the size of the image data DT of one channel is (8p+1) × 16 bytes and the channel stride (CHstride) is also set to (8p+1) × 16 bytes is shown. That is, Figure 4 , an operation example is shown in which the corresponding channels CH[0] to CH[7] are configured by, for example, pixel group data PGD0 to PGD40, and the blank area BLNK is not required.

[0167] exist Fig. 10C In the illustrated operation example, the size of the image data DT of one channel is (8p+2)×16 bytes, and the channel stride (CHstride) is set to (8p+3)×16 bytes accordingly. Figure 4 , such an operation example is shown in which the corresponding channels CH[0] to CH[7] are configured by, for example, pixel group data PGD0-PGD41, etc., and thus a blank area BLNK of 16 bytes×1 is provided. Alternatively, the exemplary operation is shown in such a case that the size of the image data DT of one channel is (8p+3)×16 bytes, and the channel stride (CHstride) is also set to (8p+3)×16 bytes, that is, in a case where the blank area BLNK is not required.

[0168] Similarly, Fig. 10DAn operation example is shown when DT is (8p+4)×16 bytes and the channel stride (CHstride) is set to (8p+5)×16 bytes. Alternatively, an operation example is shown when DT is (8p+5)×16 bytes. Fig.10E In the operation example shown, the image data DT is (8p+6)×16 bytes, and the channel stride (CHstride) is set to (8p+7)×16 bytes. Alternatively, an operation example when DT is (8p+7)×16 bytes is shown. Note that Fig. 10A The connection relationship between the input signal and the output signal shown in the figure represents Fig. 10D A portion of the operation shown.

[0169] Fig. 10B , Fig. 10C , Fig. 10D ,and Fig.10E Each of the diagrams shows a combination of indexes (idx) represented by logical addresses CH[n]_addr[6:4] of 8 channels as selection signals. Here, 3 combinations, i.e., combinations for 3 columns, are shown, some of which are omitted, but in detail, there are 8 combinations. Each diagram shows which logical address of the logical addresses CH[n]_addr[18:7] of 8 channels as input signals is connected to each of the 8 physical addresses MR_idx[n]_addr[11:0] as output signals for each combination of selection signals.

[0170] Fig.11 It is a graphic Fig. 9 A diagram showing a configuration example and some operation examples of a data router used for writing. Fig.11 The write data router 66 shown has input and output signals similar to Fig. 10A In addition to the differences in Fig. 10A The address router 66 has the same configuration as shown, and performs the same operation. The input signal to the write data router 66 is the channel write data CH[n]_WRdat[127:0] of 8 channels output from the MAC unit 25, and is stored in the write buffer 28 in channel order through operations such as activation functions. Each channel write data CH[n]_WRdat[127:0] is 16 bytes (=128 bits).

[0171] The output signal from the write data router 66 is 8 memory write data MR_idx[n]_WRdat[127:0] arranged in memory order. The write data router 66 uses Fig. 10AThe connection relationship between the input signal and the output signal is defined based on the same selection signal in the write address router 65, that is, based on the idx of the write logic address CH[n]_WRaddr[6:4] of the 8 channels from the write address generator 64. Therefore, the memory MR for each channel identified by the write address router 65, that is, the corresponding relationship between the channel and the memory MR, is the same as the memory MR for each channel identified by the write data router 66.

[0172] Fig.12 It is a graphic Fig. 9 Schematic diagram of a configuration example of a data router for reading. Fig.12 The read data router 56 shown is similar to the Fig.11 In addition to being the opposite of Fig.11 The write data router 66 shown in FIG. 5 is configured the same as the write data router 66 and performs the same operation. The read data router 56 receives 8 memory read data MR_idx[n]_REdat[127:0], which are read from 8 memories MR and arranged in memory order. The size of each memory read data MR_idx[n]_REdat[127:0] is 16 bytes (=128 bits).

[0173] The read data router 56 outputs 8 channel read data CH[n]_REdat[127:0] arranged in channel order. The read data router 56 uses the same selection signal as in 10A, that is, the idx of the read logic address CH[n]_REaddr[6:4] of the 8 channels from the read address generator 54 for reading, to define the connection relationship between the input signal and the output signal. Therefore, the memory MR for each channel identified by the read address router 55, that is, the corresponding relationship between the channel and the memory MR, is the same as the memory MR for each channel identified by the read data router 56.

[0174] However, in a read operation, the memory MR receives a physical address and outputs memory read data with a read latency of a predetermined number of clocks. In order to compensate for the read latency, the pending address buffer 57 buffers a specific bit area ([6:4]) in the read logical address CH[n]_REaddr[18:4] of the 8 channels from the read address generator 54 for a period of time based on the read latency. Then, the pending address buffer 57 outputs the buffered read logical address CH[n]_REaddr[6:4] to the read data router 56.

[0175] exist Fig.12In the embodiment, the read data router 56 includes eight selectors 58[0] to 58[7]. The selector 58[0] selects any one of the eight memory read data MR_idx[0]_REdat-MR_idx[7]_REdat based on the read logic address CH[0]_REaddr[6:4] of the channel CH[0], which is the selection signal of the channel CH[0]. Then, the selector 58[0] outputs the selected memory read data as the channel read data CH[0]_REdat of the channel CH[0].

[0176] Similarly, the selector 58[7] selects any one of the eight memory read data MR_idx[0]_REdat-MR_idx[7]_REdat based on the read logic address CH[7]_REaddr[6:4] of the channel CH[7], which is the selection signal of the channel CH[7]. Then, the selector 58[7] outputs the selected memory read data as the channel read data CH[7]_REdat of the channel CH[7]. Fig. 10A The addresses shown are routers 55 and 65 and Fig.11 The write data router 66 shown can also achieve the same configuration by changing the input and output directions of the selectors.

[0177] Details of the neural network software system

[0178] exist Fig. 9 , the neural network software system 40 includes a read control unit 75a and a write control unit 76a implemented by the CPU 20 executing the neural network software. The read control unit 75a, i.e., the CPU 20, mainly sets the set value of each register included in the read access controller 30a to the read configuration parameter 80, and sets the set value in each register. Here, the read control unit 75a includes a read channel stride correction unit 81. The read channel stride correction unit 81 corrects the channel stride (CHstride) included in the read configuration parameter 80 for reading as needed, and sets the corrected channel stride in the channel stride register 51.

[0179] Similarly, the write control unit 76a, i.e., the CPU 20, mainly sets the set value of the corresponding register included in the write access controller 31a as the write configuration parameter 85, and sets the set value in the corresponding register. Here, the write control unit 76a includes a write channel stride correction unit 86. The write channel stride correction unit 86 corrects the write channel stride included in the write configuration parameter 85 as needed, and sets the corrected channel stride (CHstride) in the write channel stride register 61.

[0180] The read channel stride correction unit 81 may set the corrected write channel stride (CHstride) obtained by the write channel stride correction unit 86 in the read channel stride register 51 as the corrected read channel stride (CHstride). That is, the channel stride (CHstride) for the write operation of a layer and the channel stride (CHstride) for the read operation of a subsequent layer are generally equal. For example, in Fig. 20 In , the channel stride (CHstride) used at the output of the NNE process [1] and the channel stride (CHstride) used at the input of the NNE process [2] are both the channel stride (CHstride) applied to the same feature map FM1.

[0181] Fig.13 It is a graphic Fig. 9 Flow chart of an example of the processing content of the channel step correction unit in FIG. Here, as a premise, as referenced Figure 7 As described, it is assumed that the number of memories M is 2 m , where m is an integer of 1 or greater, and the bit width of each of the M memories is 2 k , where k is an integer of 0 or greater. In addition, assuming that a is an integer greater than or equal to 0 and less than m, the size GS of the pixel group data PGD is 2 (k+a) bytes, and the number of N channels is 2 (m-a) .

[0182] exist Fig.13 For example, the read channel stride correction unit 81 first obtains the image size FS, which is the initial value of the channel stride (CHstride) (step S101). That is, in the read configuration parameter 80, the set value of the channel stride (CHstride), in other words, the initial value is determined as the image size FS, such as Fig.19 Furthermore, the read channel step correction unit 81 refers to the size GS of the pixel group data PGD which is set in advance (step S102).

[0183] Subsequently, the read channel stride correction unit 81 calculates FS / GS, and determines whether the calculated value is an even number (step S103). When the calculation result of step S103 is an even number, the read channel stride correction unit 81 sets the value of FS+GS×odd number in the channel stride register 51 for reading (step S104). That is, the read channel stride correction unit 81 corrects the set value of the channel stride. On the other hand, when the calculation result of step S103 is an odd number, the read channel stride correction unit 81 sets the value of FS in the channel stride register 51 (step S105). That is, the read channel stride correction unit 81 does not correct the set value of the channel stride.

[0184] For example, in Figure 4 In the case of , FS / GS calculated by step S103 is 40 (=640 / 16). For this reason, for example, 640+16×1 is set in the channel stride register 51 (step S104). Figure 4 It can be seen from FIG. 8 that if FS / GS is an odd number, the pixel data PD of the N channels arranged at the same pixel position are respectively stored in different memories MR among the M memories MR without correcting the set value of the channel stride. The write channel stride correction unit 86 also performs the same processing as the read channel stride correction unit 81.

[0185] In addition, by using the image size FS bytes, the bit width K bytes of the memory MR, and the number M of the memory MR, the method for correcting the channel stride (CHstride) is expressed by Expression (8) and Expression (9). Here, FLOOR (FS, K × M) is a function that truncates FS to a multiple of (K × M). skip_factor is a value obtained by rounding up mod (CEIL (FS / K, 1), M) to an odd number. CEIL (FS / K, 1) is a function that rounds up FS / K to an integer.

[0186] CHstride(before correction)=FS…(8)

[0187] CHstride (corrected) = K × skip_factor + FLOOR (FS, K × M)… (9)

[0188] Note that when the bit width K bytes of the memory MR and the number M of the memory MR are not a power of 2, the channel stride can be rounded up to the number of pixel group data PGD that is prime to the number of channels.

[0189] Main Effects of the First Embodiment

[0190] In the method of the first embodiment, the memory controller 29 is provided above. The memory controller 29 controls access to the SPM 16 so that the pixel data PD of the N channels arranged at the same pixel position are respectively stored in different memories MR among the M memories MR. Therefore, the pixel data PD of the N channels can be input and output in parallel with the M memories MR. Therefore, the input / output delay between the SPM 16 and the NNE 15 can be reduced. In addition, since the plane format is used as a universal format, the input / output delay can be shortened even when a DSP process or the like is included. Therefore, the processing time of the image processing can be shortened.

[0191] Furthermore, in the method of the first embodiment, channel stride registers 51 and 61 and address generators 54 and 64 are provided in the memory controller 29, and appropriate channel strides, i.e., appropriate address intervals, are set in the channel stride registers 51 and 61, thereby shortening the input / output delay. This method of using the channel stride registers 51 and 61 can reduce the number of necessary registers, which is advantageous in terms of the area of ​​the registers, the processing load associated with the setting of the registers, and the setting time. In particular, the greater the number of channels, the more beneficial the effect.

[0192] Second embodiment

[0193] Details of the main parts of a semiconductor device

[0194] Fig.14 is a block diagram showing a detailed configuration of a main portion in a semiconductor device according to a second embodiment. Fig.14 The semiconductor device 10 shown is Fig. 9 The illustrated configuration differs in the configuration of the read access controller 30 b , the configuration of the write access controller 31 b , and the configuration of the read control unit 75 b and the write control unit 76 b in the neural network software system 40 .

[0195] The read access controller 30b includes a read address register unit 90 instead of Fig. 9 The read base address register 50, the channel stride register 51 and the read address generator 54 are shown. The read address register unit 90 includes N address registers. In each of the N address registers, the start address CH[n]_RSaddr of the corresponding channel in the image data DT of the N channels is set.

[0196] The adder 53b adds the common scan address Saddr from the address counter 52 to the N start addresses CH[n]_RSaddr output from the read address register unit 90. Therefore, the adder 53b outputs the read logic addresses CH[n]_REaddr and CH[n]_REaddr of the N channels in parallel. Fig. 9The output of address generator 54 is read from .

[0197] According to this difference, the read control unit 75b includes a read address correction unit 95 instead of Fig. 9 The read channel stride correction unit 81 is shown. The read address correction unit 95 is connected to Fig.13 The same process as in the case of Fig. 9 The read base address register 50 and the read address generator 54 in the read address register 50 correspond to each other. That is, the read address correction unit 95 sequentially adds the determined address interval to the base address for a certain read, or adds an integer multiple of the address interval, thereby calculating N start addresses CH[n]_RSaddr.

[0198] Then, the address correction unit 95 sets the calculated N start addresses CH[n]_RSaddr in the N address registers in the read address register unit 90. Therefore, Fig.13 In the case of, when FS / GS is an even number, the address interval between adjacent channels in the N start addresses CH[n]_RSaddr is FS+GS×odd number. On the other hand, when FS / GS is an odd number, the address interval between adjacent channels is FS.

[0199] Similarly, the write access controller 31b also includes a write address register unit 91, which replaces Fig. 9 The write base address register 60, the channel stride register 61, and the write address generator 64 are shown. The write address register unit 91 includes N address registers. In each of the N address registers, the start address CH[n]_WSaddr of the corresponding channel in the image data DT of the N channels is set. The adder 63b outputs the write logic address CH[n]_WRaddr of the N channels in parallel by adding the common scanning address Saddr from the address counter 62 to the N start addresses CH[n]_WSaddr output from the write address register unit 91.

[0200] The write controller 76b includes a write address correction unit 96. The write address correction unit 96 determines the address interval between channels. The write address correction unit 96 sequentially adds the determined address interval to the base address for a certain write, or adds an integer multiple of the address interval to calculate N start addresses CH[n]_WSaddr. Then, the write address correction unit 96 sets the calculated N start addresses CH[n]_WSaddr in the N address registers in the write address register unit 91.

[0201] Main Effects of the Second Embodiment

[0202] By using the method of the second embodiment, the above can obtain various effects similar to those described in the first embodiment. In the second embodiment, the memory controller 29 is provided with address register units 90 and 91 including N address registers, and a suitable start address for each channel is set in the address register units 90 and 91, thereby reducing the input / output delay.

[0203] Therefore, compared with the method of the first embodiment, a disadvantage is obtained from the viewpoint of the processing load and setting time associated with the area of ​​the register and the setting of the register, but an advantage is obtained from the viewpoint of increasing the degree of freedom of setting. For example, Figure 4 The blank area BLNK between channels shown can extend any number of 128-byte units between channels. In particular, in a typical CNN process, such freedom is usually not needed, but in a specific neural network process, such freedom may be needed.

[0204] Third embodiment

[0205] Details of the main parts of a semiconductor device

[0206] Fig.15 is a block diagram showing a detailed configuration of a main portion in a semiconductor device according to a third embodiment. Fig.15 The semiconductor device 10 shown is Fig. 9 The illustrated configuration differs in the configuration of the read access controller 30 c , the configuration of the write access controller 31 c , and the configuration of the read control unit 75 c and the write control unit 76 c in the neural network software system 40 .

[0207] The write access controller 31c includes a temporary write channel stride register 61c, a write channel stride correction circuit 105, and a write status register 106 instead of Fig. 9 The write channel stride register 61 is shown. In the temporary write channel stride register 61c, a temporary value of the write channel stride, in other words, a temporary value of the address interval is set by the write control unit 76c.

[0208] The write channel stride correction circuit 105 is implemented by a dedicated hardware circuit and Fig.13 The write channel stride correction unit 86 performs the same processing as above, correcting the temporary value of the channel stride as needed. Then, the write channel stride correction circuit 105 outputs the corrected value of the channel stride to the write address generator 64 and the write status register 106.

[0209] Similarly, the read access controller 30c includes a temporary read channel stride register 51c, a read channel stride correction circuit 100, and a read status register 101 instead of Fig. 9The channel stride register 51 is shown. In the temporary read channel stride register 51c, the temporary value of the read channel stride, in other words, the temporary value of the address interval is set by the read control unit 75c.

[0210] The read channel stride correction circuit 100 is implemented by a dedicated hardware circuit and Fig.13 The read channel stride correction unit 81 performs the same processing as above, and corrects the temporary value of the read channel stride as needed. The read channel stride correction circuit 100 outputs the corrected channel stride to the read address generator 54 and the read status register 101 .

[0211] The read control unit 75c reads the channel stride corrected by the write channel stride correction circuit 105, which is defined by, for example, the intermediate layer in the previous stage, from the write status register 106. Then, the read control unit 75c writes the read value of the channel stride into the temporary channel stride register 51c as the value of the read channel stride in the intermediate layer in the subsequent stage, etc.

[0212] Therefore, the feature map FM generated by the intermediate layer or the like in the previous stage can be used as an input in the intermediate layer or the like in the subsequent stage. At this time, since the correction of the read channel stride correction circuit 100 is unnecessary, the read control unit 75c can, for example, output a control signal indicating that the correction is unnecessary to the read channel stride correction circuit 100. In addition, the channel stride read from the write status register 106 is not limited to the NNE 15, and is also used in the DSP 18 and the DMAC 17.

[0213] On the other hand, the write control unit 76c reads the channel stride corrected by the read channel stride correction circuit 100 used in, for example, an intermediate layer in a certain stage from the read status register 101. Then, the write control unit 76c writes the read value of the channel stride into the write temporary channel stride register 61c as the value of the channel stride used for writing in the intermediate layer in the previous stage, etc.

[0214] Therefore, the memory mapping to be applied to the feature map FM output from the intermediate layer etc. in the previous stage can be determined based on the feature map FM input into the intermediate layer etc. At this time, since the correction of the write channel stride correction circuit 105 is unnecessary, the write control unit 76c can, for example, output a control signal indicating that the correction is unnecessary to the write channel stride correction circuit 105. In addition, the channel stride read from the read status register 101 is not limited to the NNE 15, and is also used in the DSP 18 and the DMAC 17.

[0215] Note that, unlike the process in the read control unit 75c, the process in the write control unit 76c is a process that goes back from the back to the front in time. For this reason, for example, by providing two register surfaces, etc., it is necessary to predetermine the value of the channel stride for reading before the start of processing in the intermediate layer in the previous stage, etc. When processing is performed in the intermediate layer in the previous stage, etc., the value of the predetermined channel stride is set as the value of the write channel stride.

[0216] Fig.16 It is a graphic with Fig.15 Block diagram of a detailed configuration example of the different main parts. Fig.15 The situation is similar, Fig.16 The method of applying the third embodiment is shown Fig.14 An example configuration is shown. Fig.16 The semiconductor device 10 shown is Fig.14 The illustrated configuration differs in the configuration of the read access controller 30 d , the configuration of the write access controller 31 d , and the configuration of the read control unit 75 d and the write control unit 76 d in the neural network software system 40 .

[0217] In the write access controller 31d, a write address correction circuit 115 and a write status register 116 are added Fig.14 in the configuration shown. Fig.14 The write address register unit 91 in Fig.16 In the temporary address register unit 91d, temporary values ​​of N start addresses associated with N channels are set by the write control unit 76d.

[0218] The write address correction circuit 115 is implemented by a dedicated hardware circuit. Fig.14 The write address correction circuit 115 performs the same processing as that of the write address correction unit 96 shown in the figure, and corrects the temporary values ​​of the N start addresses as needed. The write address correction circuit 115 outputs the N corrected start addresses to the adder 63b and the write status register 116.

[0219] Similarly, in the read access controller 30d, a read address correction circuit 110 and a read status register 111 are added Fig.14 in the configuration shown. Fig.14 The read address register unit 90 is Fig.16 In the temporary address register unit 90d, temporary values ​​of N start addresses associated with N channels are set by the read control unit 75d.

[0220] The read address correction circuit 110 is implemented by a dedicated hardware circuit and Fig.14The same process as shown in the read address correction unit 95 for reading is performed to correct the temporary values ​​of the N start addresses as needed. The read address correction circuit 110 outputs the N corrected start addresses to the adder 53b and the read status register 111. The operation of the read control unit 75d and the write control unit 76d is the same as that of the reference numeral 114. Fig.15 The operation of the read control unit 75c and the write control unit 76c described is the same except that the process target is replaced by the starting address of each channel from the channel stride.

[0221] Main Effects of the Third Embodiment

[0222] By using the scheme of the third embodiment, the above can obtain effects similar to the various effects described in the first embodiment or the second embodiment. In addition, by correcting the channel stride or the start address of each channel by a dedicated hardware circuit, the processing load of the software can be reduced. In addition, by making the software recognize the correction result of the hardware circuit via the status register, the software can determine the processing content of each intermediate layer, etc., specifically, determine the memory mapping by reflecting the correction result. Therefore, the efficiency of image processing can be improved.

[0223] Fourth embodiment

[0224] As a neural network architecture for improving the recognition accuracy of an image, not limited to CNN, vector operations, matrix transposition (Transpose), and matrix operations (Matmul, Gemm, etc.) ViT (Vision Transformer for execution process) and the like are known. In an architecture such as ViT, vector operations, matrix transposition, and matrix operations are performed by replacing image data with a matrix structure. In this case, it is necessary to process not only the three-dimensional (X direction, Y direction, channel direction) data as described in the first embodiment, but also D-dimensional data including four or more dimensions.

[0225] Therefore, in the fourth embodiment, the method of the first embodiment and the like are extended to D dimensions, for example, 4 dimensions, where D is an integer of 2 or more. That is, in the SPM 16, D-dimensional data is stored in a planar format. Then, in the fourth embodiment, a method for accessing a plurality of data having D dimensions (in other words, D dimensions or D axes parallel to the SPM 16) is shown. Note that the semiconductor device according to the fourth embodiment has the same configuration as the various configurations described in the first to third embodiments. Here, it is assumed that the semiconductor device 10 has Figure 1 and Fig. 9 Configuration shown.

[0226] Method for storing four-dimensional data in memory

[0227] Fig.17: is a schematic diagram showing an example of a four-dimensional data format used in a semiconductor device according to a fourth embodiment. Fig.17 In the expression (10), the total number of data num_ALL bytes is represented by the expression (10). In the expression (10), num_AX1, num_AX2, num_AX3, num_AX4 are the number of elements in the first dimension, the second dimension, the third dimension, and the fourth dimension, in other words, the first axis AX1, the second axis AX2, the third axis AX3, and the fourth axis AX4, respectively.

[0228] num_ALL=num_AX1×num_AX2×num_AX3×num_AX4…(10)

[0229] Fig.17 An example is shown in which the number of elements in the third axis AX3 is 4 and the number of elements in the fourth axis is 3. Fig.17 As shown, in the fourth axis AX4, the elements from the second axis AX2 do not have to be arranged closely in the memory MR, but are arranged with a constant step between the elements. Here, the logical address Laddr_4D of storing the 4-dimensional data DAT [AX4-idx] [AX3-idx] [AX2-idx] using the index (idx) values ​​in the first axis AX1, the second axis AX2, the third axis AX3, and the fourth axis AX4 as AX1-idx, AX2-idx, AX3-idx, AX4-idx is represented by expression (11).

[0230] Laddr_4D=AX1-idx+AX2-idx×AX2_stride+AX3-idx×AX3_stride+AX4-idx×AX4_stride…(11)

[0231] In expression (11), for example, for image data, AX1-idx is an index value in the X direction (horizontal direction), AX2-idx is an index value in the Y direction (vertical direction), and AX3-idx is an index value in the channel direction. AX4-idx is an index value for further distinguishing such three-dimensional image data. Here, num_AX1 is the width (horizontal image size), num_AX2 is the height (number of rows), and num_AX3 is the number of channels.

[0232] In expression (11), AX2_stride is the stride (byte) between adjacent elements in the second axis AX2, and is the line stride in the case of image data. AX3_stride is the stride between adjacent elements in the third axis AX3, and is the channel stride in the case of image data. AX4_stride is the stride between adjacent elements in the fourth axis. In order to prevent address conflicts between adjacent elements, the stride between adjacent elements must be greater than or equal to the number of elements of the axis of the lower dimension. For this reason, the constraints shown in expressions (12A), (12B), and (12C) are provided.

[0233] AX2_stride>=num_AX1…(12A)

[0234] AX3_stride>=num_AX2×AX2_stride…(12B)

[0235] AX4_stride>=num_AX3×AX3_stride…(12C)

[0236] Here, the memory controller 29 controls access to the SPM 16 so that in two or more axes including the first axis AX1, a plurality of data fragments having the same index value are not stored in the same memory MR among the M memories MR. At this time, the neural network software system 40 determines a read stride and a write stride, which are address intervals, and sets each of the determined strides in a read stride register and a write stride register, respectively, thereby performing such access. The read stride register and the write stride register are respectively connected to Fig. 9 The channel stride register 51 and the channel stride register 61 in correspondence.

[0237] In other words, the SPM 16 stores D-dimensional data, where corresponding data in one dimension is distinguished by an index (idx) value, where D is an integer of 2 or greater. The memory controller 29 controls access to the SPM 16 so that the number of index values ​​in the D-th dimension is N, and N data fragments having the same index value in the first to (D-1)-th dimensions are stored in different memory MRs among the M memory MRs.

[0238] Therefore, for example, in the four-dimensional data DAT[AX4-idx][AX3-idx][AX2-idx][AX1-idx], N data fragments consisting of DAT[0][0][0][0], DAT[1][0][0][0], ..., DAT[N-1][0][0][0] are stored in mutually different memories MR. Therefore, the memory controller 29 can read the N data fragments in parallel and write the N data fragments in parallel to the SPM 16.

[0239] In addition, the memory controller 29 may control access to the SPM 16 so that N1×N2×…×N included in the first to (D-1)th dimensions d-1 The data segments are stored in different memories MR in M ​​memories MR, where the number of index values ​​in the 1st, 2nd, ..., D-1, and D dimensions are N1, N2, ..., N respectively. d-1 、N d The number of memories MR is N1×N2×…×N d-1 or more.

[0240] Memory access example

[0241] Fig.18A 1 is a diagram illustrating a specific example of a D-dimensional format in a semiconductor device according to a fourth embodiment. Fig.18A In the specific example shown (Example 3), the value of the corresponding variable is a power of 2, which is generally used. The bit width K of the corresponding memory MR is 8 (=2 k ) bytes, and the minimum data unit size N1 is 16 (=2 (a+k) =A×K) bytes. The size N1 of the minimum data unit is the size of the corresponding data in the first axial direction AX1, and corresponds to the size GS of the pixel group data PGD in the first embodiment and the like.

[0242] The number N2, N3, N4, N5, ... of the minimum data units accessed in parallel in the second axis AX2, the third axis AX3, the fourth axis AX4, the fifth axis AX5, ... is the number of the minimum data units that can be input or output in parallel in one clock cycle with respect to the SPM 16, for example, the number of pixel group data PGD. Here, the number N2, N3, N4, N5, ... of the minimum data units is 2, 2, 4, 1, ... respectively. The number M of the memories MR constituting the SPM 16 is 32 (=2 m ). The total number N of the minimum data units accessed in parallel is 16 (=N2×N3×N4×N5…). By multiplying the size N1 of the minimum data unit by the total number N, the total bit width of the parallel access is 256 (=16×16) bytes.

[0243] Fig.18B It is shown in Fig.18A Schematic diagram of the arrangement configuration of data accessed in parallel in a four-dimensional format under the premise of the configuration shown. Fig.18B 16 data DAT0-DAT15 are shown to be input and output in parallel in a four-dimensional format in the same clock cycle. In this specification, a plurality of data DAT0-DAT15 are collectively referred to as data DAT.

[0244] The size of one data DAT is 16 bytes, which is the size N1 of the minimum data unit defined in the first axial direction AX1. The 16 data DAT0-DAT15 are arranged in two in the second axial direction AX2 using the stride AX2_stride for the second axial direction. Similarly, the 16 data DAT0-DAT15 are arranged in two in the direction of the third axis AX3 using the stride AX3_stride for the third axis, and are arranged in four in the direction of the fourth axis AX4 using the stride AX4_stride for the fourth axis.

[0245] Fig.18C It is shown Fig.18B Schematic diagram of the start address and end address of the corresponding data fragment shown. In the corresponding DAT, the end address is the sum of the start address and N1-1. In addition, for example, the start address of data DAT1 is obtained by adding the stride AX2_stride of the second axis to the start address of data DAT0. The start address of data DAT2 is a value obtained by adding the stride AX3_stride of the third axis to the start address of data DAT0. The start address of data DAT3 is obtained by adding the stride AX2_stride of the second axis to the start address of data DAT2.

[0246] Fig.18D It is shown in Fig.18A FIG. 2 is a diagram of an exemplary neural network software executed by a CPU based on the configuration shown. The CPU 20 executes, for example, Fig.18D The multi-loop programming shown in FIG. 1 increases the dimension outward. At this time, the CPU 20 causes the neural network engine (NNE) 15 to perform the arithmetic process associated with the multi-loop. Therefore, the NNE 15 needs to input or output N1×N2×N3×N4 data segments to the SPM 16, which are 16 data segments DAT0 to DAT15.

[0247] Therefore, for example, the CPU 20 sets the read stride or write stride as shown in Expression (13A) to Expression (13C) for use with Fig. 9 The CPU 20 reads the stride register corresponding to the channel stride register 51 or writes the stride register corresponding to the channel stride register 61 as shown. That is, the CPU 20 corrects the stride AX2_stride for the second axis to satisfy the constraint of expression (12A), and adds the size N1 (=16 bytes) to a multiple of 256 bytes (256n), that is, K×M, as shown in expression (13A). As described above, the size N1 is the size of the minimum data unit in the first axial direction AX1, in other words, the size of the packet data PGD.

[0248] In addition, while satisfying the constraints of expression (12B), the CPU 20 corrects the stride AX3_stride for the third axis so that it becomes a multiple of 256 bytes (256n) plus N1×N2, N1×N2 being the multiplication result of size N1 and number N2. As described above, number N2 is the number of minimum data units to be accessed in parallel in the second axial direction AX2. In addition, while satisfying the constraints of expression (12C), the CPU 20 corrects the stride AX4_stride for the fourth axis so that it becomes a multiple of 256 bytes (256n) plus N1×N2×N3, N1×N2×N3 being the multiplication result of size N1, number N2, and number N3. As described above, number N3 is the number of minimum data units to be accessed in parallel in the third axial direction AX3.

[0249] AX2_stride=256n+N1…(13A)

[0250] AX3_stride=256n+N1×N2…(13B)

[0251] AX4_stride=256n+N1×N2×N3…(13C)

[0252] Therefore, the formation Fig.18E The memory map is shown. Fig.18E It is shown in Fig.18C A diagram of an exemplary memory mapping of corresponding data stored in a temporary storage memory (SPM) after performing stride correction based on the specific situation shown. Fig.18E As shown, by performing stride correction, 16 data DAT0-DAT15 are allocated to different idxs in the SPM 16 and stored in different memory MRs, here 32 memory MRs. Therefore, the NNE 15 can input 16 DAT0-DAT15 to the SPM 16 in parallel and output 16 DAT0-DAT15 from the SPM 16. Therefore, the input / output delay can be shortened.

[0253] Main Effects of the Fourth Embodiment

[0254] As described above, by using the method of the fourth embodiment, effects similar to the various effects described in the first to third embodiments can be obtained. In addition, the same effect can be obtained with respect to data in D dimension which is two or more dimensions.

[0255] Although the invention made by the present inventors has been specifically described based on the embodiments, the present invention is not limited to the above-mentioned embodiments, and it goes without saying that various modifications can be made without departing from the gist thereof.

Claims

1. A semiconductor device, comprising: a temporary memory storing image data of N channels and comprising M memories capable of being accessed individually, wherein M is an integer of 2 or greater, and N is an integer of 2 or greater; a memory controller configured to control access to the temporary memory so that pixel data of the N channels arranged at the same pixel position in the image data of the N channels are respectively stored in different memories of the M memories; A multiplication-accumulation MAC unit includes a plurality of calculators to calculate the pixel data of the N channels read from the temporary memory by using the memory controller and weight parameters.

2. The semiconductor device according to claim 1, The memory controller comprises: a read access controller configured to generate read logical addresses for the N channels respectively when reading the pixel data of the N channels from the temporary storage memory, for reading the pixel data of the N channels, and The write access controller is configured to generate write logical addresses of the N channels respectively for storing the pixel data of the N channels when the pixel data of the N channels are written into the temporary storage memory.

3. The semiconductor device according to claim 2, wherein the read access controller comprises a read address router configured to convert the read logical addresses of the N channels into read physical addresses of the M memories to output the read physical addresses of the N channels to the M memories, each of the M memories being associated with one of the N channels, and The write access controller includes a write address router configured to convert the write logical addresses of the N channels into write physical addresses for the M memories so as to output the write physical addresses to the M memories, each memory being associated with one of the N channels.

4. The semiconductor device according to claim 3, The write access controller includes a write data router configured to output the pixel data of the N channels obtained based on calculation results of the MAC unit to the M memories each of which is associated with one of the N channels.

5. The semiconductor device according to claim 3, wherein the read address router identifies the corresponding relationship between the N channels and the M memories by the value of a specific bit region included in the read logical address of the N channels, and The write address router identifies the correspondence between the N channels and the M memories by using the value of a specific bit region included in the write logical addresses of the N channels.

6. The semiconductor device according to claim 5, The read access controller further comprises: a read data router configured to rearrange the pixel data of the N channels read from the M memories in channel order based on the read physical address from the read address router, and a pending address buffer configured to buffer the specific bit regions of the read logical addresses of the N channels for a period of time based on the read latency of the M memories, wherein the read data router receives the specific bit region of the read logical address of the N channels from the pending address buffer, and The MAC unit receives the pixel data of the N channels rearranged by the read data router.

7. The semiconductor device according to claim 2, wherein the access controller is the read access controller or the write access controller, and The access controller comprises: a channel stride register in which an address interval between start addresses of the image data of adjacent channels of the N channels stored in the temporary memory is set, and An address generator adds an integer multiple of the address interval to a reference logical address to generate logical addresses for the N channels in the temporary memory.

8. The semiconductor device according to claim 7, wherein the access controller uses a pixel data group including a plurality of the pixel data as a data unit to access the temporary storage memory, The M memories each have 2 k The bit width of a byte, where the value of M is 2 m , m is an integer of 1 or more, and k is an integer of 0 or more, The channel stride register is set to FS+GS×odd when FS / GS is an even number, and is set to FS when FS / GS is an odd number, wherein FS is the byte size of each of the image data of the N channels, and GS is the byte size of the pixel data group or 2 (k+a) Byte, a is 0 or an integer greater than 0 and less than m, and the value of N is 2 (m-a) The number of.

9. The semiconductor device according to claim 2, wherein the access controller is the read access controller or the write access controller, and The access controller comprises: N address registers, storing N starting addresses respectively, each of the N starting addresses being a starting address of image data of an associated channel among the N channels, and An adder adds the common scan address to each of the N start addresses.

10. The semiconductor device according to claim 9, wherein the access controller uses a pixel data group including a plurality of the pixel data as a data unit to access the temporary storage memory, The M memories each have 2 k The bit width of a byte, where the value of M is 2 m , m is an integer of 1 or more, and k is an integer of 0 or more, The address interval between the start addresses of the image data of adjacent channels of the N channels is set to FS+GS×odd when FS / GS is an even number, and is set to FS when FS / GS is an odd number, wherein FS is the byte size of each of the image data of the N channels, and GS is the byte size of the pixel data group or 2 (k+a) Byte, a is 0 or an integer greater than 0 and less than m, and the value of N is 2 (m-a) The number of.

11. The semiconductor device according to claim 1, further comprising: The direct memory access (DMA) controller is configured to control data transfer between the temporary memory and the main memory.

12. The semiconductor device according to claim 1, further comprising: The digital signal processor DSP performs signal operations on the image data of the N channels stored in the temporary storage memory.

13. A semiconductor device comprising: a temporary memory storing image data of N channels and comprising M memories capable of being accessed individually, wherein M is an integer of 2 or greater, and N is an integer of 2 or greater; a memory controller configured to control access to the temporary storage memory based on a set value of the register; a central processing unit CPU configured to determine the set value for the register of the memory controller; as well as Multiply-accumulate MAC unit, including multiple calculators, The CPU determines the set value of the register so that pixel data of N channels arranged at the same pixel position in the image data of the N channels are respectively stored in different memories of the M memories, and each of the calculators performs multiplication and accumulation operations on the pixel data of the N channels read from the temporary memory by using the memory controller and weight parameters.

14. The semiconductor device according to claim 13, The memory controller comprises: a channel stride register in which an address interval between start addresses of the image data of adjacent channels of the N channels stored in the temporary memory is set, and an address generator, adding an integer multiple of the address interval to a reference logical address to generate logical addresses for the N channels in the temporary memory, and The CPU sets the address interval in the channel stride register.

15. The semiconductor device according to claim 14, wherein the memory controller uses a pixel data group including a plurality of the pixel data as a data unit to access the temporary storage memory, The M memories each have 2 k The bit width of a byte, where the value of M is 2 m , m is an integer of 1 or more, and k is an integer of 0 or more, The CPU sets FS+GS×odd in the channel stride register when FS / GS is an even number, and sets FS when FS / GS is an odd number, wherein FS is the byte size of each of the image data of the N channels, and GS is the byte size of the pixel data group or 2 (k+a) Byte, a is 0 or an integer greater than 0 and less than m, and the value of N is 2 (m-a) The number of.

16. The semiconductor device according to claim 13, The memory controller comprises: N address registers, storing N starting addresses respectively, each of the N starting addresses being a starting address of image data of an associated channel among the N channels, and an adder, adding the common scan address to each of the N starting addresses, The CPU sets the N starting addresses in the N address registers.

17. The semiconductor device according to claim 16, wherein the memory controller uses a pixel data group including a plurality of the pixel data as a data unit to access the temporary storage memory, The M memories each have 2 k The bit width of a byte, where the value of M is 2 m , m is an integer of 1 or more, and k is an integer of 0 or more, The address interval between the start addresses of the image data of adjacent channels of the N channels is set to FS+GS×odd when FS / GS is an even number, and is set to FS when FS / GS is an odd number, wherein FS is the byte size of each of the image data of the N channels, and GS is the byte size of the pixel data group or 2 (k+a) Byte, a is 0 or an integer greater than 0 and less than m, and the value of N is 2 (m-a) The number of.

18. A semiconductor device comprising: a temporary memory storing D-dimensional data and comprising M memories capable of being accessed individually, the D-dimensional data being configured such that each data in one dimension is distinguished by an index value, wherein D is an integer of 2 or greater, and M is an integer of 2 or greater, and A memory controller is configured to control access to the temporary memory so that N data fragments having the same index value in the first dimension to the D-1th dimension are respectively stored in different memories among the M memories, wherein the number of the index values ​​in the D dimension is N.

Images