Arithmetic processing device and operation method of arithmetic processing device

Abstract

To rearrange a matrix into data having less redundant parts without depending on the configuration of an accelerator.SOLUTION: An arithmetic processing device for executing convolution calculation of a matrix including a plurality of elements comprises: a first memory for holding the matrix; a second memory for holding data obtained by rearranging the elements of the matrix held in the first memory; a processor that rearranges the elements of the matrix and holds them in the second memory by writing a plurality of element groups arranged in a row direction, which is used for the matrix product operation with a filter in each row of the matrix, to each of a plurality of rows of the second memory in one column so as to overlap sequentially with some elements in a plurality of element groups arranged in a column direction of the matrix; and an accelerator that executes a convolution calculation of the matrix by executing a matrix product operation of a plurality of element groups in which some of the elements are redundantly written in the second memory and a filter. Thus, the processing performance of the arithmetic processing apparatus can be improved, and the efficiency of machine learning or the like can be improved.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a processor and a method of operating a processor. [Background technology]

[0002] In recent years, convolutional calculations have come to be used frequently due to the development of CNNs (Convolutional Neural Networks) such as ResNet50. Convolutional calculations require a large amount of computation because matrix multiplication operations are performed repeatedly, and most of the training time for DNNs (Deep Neural Networks), including CNNs, is often taken up by computation time.

[0003] For example, accelerators such as GPUs (Graphics Processing Units) equipped with multiple cores including many matrix multipliers that perform matrix multiplication are used for DNN training. For example, this type of accelerator performs matrix multiplication using input data that has been converted from two-dimensional to one-dimensional by im2col conversion. This allows the accelerator to efficiently read input data from memory and efficiently perform matrix multiplication.

[0004] For example, in an information processing device, the processing speed of a convolution process can be improved by adding up the costs of each combination of matrix processes executed in the convolution process and selecting the combination of matrix processes that minimizes the total cost (see, for example, Patent Document 1).

[0005] For example, an accelerator that performs a matrix multiplication operation has an im2col engine or an im2col module that performs an im2col conversion. In the conversion by the im2col engine or the im2col module, common parts of multiple input data are duplicated and stored in memory in accordance with the configuration of the matrix multiplication operation unit, thereby saving memory capacity (for example, see Patent Documents 2 and 3). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2020 / 031281 [Patent Document 2] US Patent Application Publication No. 2022-0066960 [Patent Document 3] US Patent Application Publication No. 2020-0133831 Summary of the Invention [Problem to be solved by the invention]

[0007] No method has been proposed for rearranging matrices into data with less redundancy, independent of the accelerator configuration.

[0008] In one aspect, the present invention aims to rearrange a matrix into data with less redundancy, regardless of the configuration of an accelerator. [Means for solving the problem]

[0009] According to one aspect, a processing device is a processing device that performs a convolution calculation of a matrix including a plurality of elements, and includes: a first memory that holds the matrix; a second memory that holds data obtained by rearranging the elements of the matrix held in the first memory; a processor that rearranges the elements of the matrix and holds them in the second memory by writing a group of elements arranged in the row direction, which are used for a matrix multiplication calculation with a filter in each row of the matrix, in a single column into each of a plurality of rows of the second memory so that the elements sequentially overlap with some of the elements of the group of elements arranged in the column direction of the matrix; and an accelerator that performs the convolution calculation of the matrix by executing a matrix multiplication calculation between the group of elements written in the second memory with some of the elements overlapping and the filter. Effect of the Invention

[0010] It is possible to rearrange matrices into data with less redundancy, regardless of the accelerator configuration. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram illustrating an example of a processor according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the operation of the arithmetic processing device of FIG. [Diagram 3] FIG. 13 is a block diagram showing an example of a processor according to another embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of functional blocks of software executed by the processor of FIG. [Diagram 5] 4 is a diagram illustrating an overview of an operation when the arithmetic processing device in FIG. 3 executes a convolution calculation of an input image. [Figure 6] FIG. 1 is a diagram illustrating an overview of a convolution calculation process. [Figure 7] FIG. 13 is a diagram showing an example of data arrangement when an input image is rearranged for convolution calculation. [Figure 8] FIG. 13 is a diagram showing an example of the size of each matrix in the matrix product αAb+βC. [Figure 9] FIG. 13 is a diagram showing an example of data after sorting of an input image into an im2col format. [Figure 10] FIG. 13 is a diagram showing an example of data after conversion of an input image into an im2col reduced format. [Figure 11] FIG. 13 is a diagram showing an example of the size of data after sorting of an input image in the im2col format and the im2col reduced format. [Figure 12] FIG. 13 is a diagram showing an example in which a multi-channel image is input, a convolution calculation is performed, and then a multi-channel image is output. [Figure 13] FIG. 13 is a diagram showing an example of rearranging a multi-channel input image into an im2col reduced format. [Figure 14] FIG. 2 shows an example of an arrangement of elements of a multi-channel filter used to obtain a multi-channel output image. [Figure 15]FIG. 15 is a diagram showing an outline of a matrix multiplication operation executed using the multi-channel input images rearranged in FIG. 13 and the multi-channel filters rearranged in FIG. 14. [Figure 16] FIG. 13 is a diagram showing an example of a code description for sorting input images into an im2col reduced format. [Figure 17] FIG. 17 is a diagram illustrating an image of writing data to be sorted into the im2col reduced format by the code illustrated in FIG. 16 into a memory area. [Figure 18] FIG. 17 is a flow diagram showing an example of a sorting process executed by the code shown in FIG. [Figure 19] FIG. 20 is a flowchart showing an example of the processes of steps S140 and S180 in FIG. 18. [Figure 20] FIG. 13 is a diagram showing an example of a code description for performing a convolution calculation (matrix multiplication operation) of an input image in the im2col reduced format. [Figure 21] FIG. 2 illustrates an example of matrix elements stored in memory when performing a matrix multiplication operation. [Figure 22] FIG. 21 is a flow diagram showing an example of a convolution calculation (matrix multiplication operation) process executed by the code shown in FIG. 20. [Figure 23] FIG. 13 is a diagram showing an example of a result of performing a convolution calculation in the im2col format and the im2col reduced format. [Figure 24] FIG. 13 is a diagram showing an example of the execution time of each process in convolution calculation in the im2col reduced format. [Diagram 25] 4 is a diagram showing an example of a matrix multiplication operation when an input image with an NCHW layout is input to the arithmetic processing device of FIG. 3. [Figure 26] FIG. 13 is a diagram illustrating an example of sorting of input images into an im2col reduced format by a calculation processing device according to another embodiment. [Figure 27] FIG. 11 is a diagram illustrating an example of filter movement when there is a stride in a calculation processing device according to another embodiment. [Figure 28]FIG. 13 is a flow diagram showing an example of a process for sorting input images into an im2col reduced format when there is a stride. [Figure 29] FIG. 29 is a flowchart showing an example of the process of step S180A in FIG. 28. [Diagram 30] FIG. 13 is a flow diagram showing an example of a process of convolution calculation (matrix multiplication operation) of an input image in the im2col reduced format when there is a stride. [Diagram 31] FIG. 13 is a block diagram showing an example of a processor according to another embodiment. [Diagram 32] FIG. 32 is a flow diagram showing an example of a process of rearranging a matrix into an im2col reduced format and a convolution calculation process performed by the arithmetic processing device of FIG. 31. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment will be described with reference to the drawings. In the following, image data may be simply referred to as an image.

[0013] FIG 1 illustrates an example of a processing device according to an embodiment. The processing device 100 illustrated in FIG 1 includes a processor 110, an accelerator 120, a first memory 130, and a second memory 140. The first memory 130 holds matrices to be subjected to a matrix multiplication operation. The matrices may be image data or numerical data. The second memory 140 holds data in which the elements of the matrices are rearranged into the im2col reduced format illustrated in FIG 2.

[0014] For example, the arithmetic processing device 100 may be in the form of one chip, or may be in the form including multiple chips. In the case of a multiple chip form, the processor 110, the accelerator 120, the first memory 130, and the second memory 140 may each have the form of a chip. Alternatively, the processor 110, the second memory 140, the accelerator 120, and the first memory 130 may each have the form of a chip. The processor 110, the accelerator 120, the second memory 140, and the first memory 130 may each have the form of a chip.

[0015] The arithmetic processing device 100 may be a server. The arithmetic processing device 100 may include a plurality of processors 110, a plurality of accelerators 120, a plurality of first memories 130, and a plurality of second memories 140, which are connected to each other via a network, and may be in the form of a cloud. The filter may be held in the second memory 140.

[0016] The processor 110 reads out the matrix held in the first memory 130, rearranges the elements of the read out matrix into the im2col reduced format, and writes the rearranged data to the second memory 140. The accelerator 120 has a plurality of matrix multiplication calculators (not shown). The accelerator 120 reads out data in the im2col reduced format from the second memory 140, and executes a matrix multiplication calculation between the read out data and a filter, thereby executing a matrix convolution calculation.

[0017] Fig. 2 shows an example of the operation of the arithmetic processing device 100 of Fig. 1. In the example shown in Fig. 2, the matrix to be processed includes data of 3 rows and 3 columns and zero padding arranged around the data, and includes elements of 5 rows and 5 columns. The filter has 3 rows and 3 columns. Hereinafter, 9 elements of 3 rows and 3 columns, the same size as the filter, are also referred to as element groups. Each element group is a collection of elements used for a matrix multiplication operation with the filter. The element groups may include zero padding.

[0018] The element groups centered around element numbers 11, 12, and 13 are in row 0. The element groups centered around element numbers 21, 22, and 23 are in row 1. The element groups centered around element numbers 31, 32, and 33 are in row 2.

[0019] When rearranging a matrix into data in the im2col reduced format, the processor 110 writes a plurality of element groups arranged in the row direction of the matrix in a column along a second direction D2 intersecting with the first direction D1 to each of a plurality of rows arranged in a first direction D1 of the second memory 140. The plurality of element groups arranged in the column direction are any of three element groups centered on element numbers 11, 21, and 31, three element groups centered on element numbers 12, 22, and 32, or three element groups centered on element numbers 13, 23, and 33.

[0020] Furthermore, when writing the element groups to the second memory 140, the processor 110 writes the element groups in a line along the second direction D2, with some of the elements of adjacent element groups overlapping each other among the element groups arranged in the column direction of the matrix. That is, the processor 110 writes the element groups arranged in the row direction in each row of the matrix in a line into each of the rows of the second memory 140, so that they sequentially overlap some of the elements of the element groups arranged in the column direction of the matrix. In the example shown in FIG. 2, the rows of the second memory 140 are three rows arranged in the first direction D1.

[0021] The sizes of the matrix and the filter are not limited to those shown in FIG. 2. Even if the size of the matrix is ​​larger than that shown in FIG. 2, the processor 110 rearranges the multiple element groups arranged in the row direction of the matrix in order in the first direction D1 of the second memory 140, and writes the multiple elements of each of the multiple element groups arranged in the row direction in a line in the second direction D2 intersecting the first direction D1. Furthermore, when writing the element groups to the second memory 140, the processor 110 overlaps some of the elements of adjacent element groups among the multiple element groups arranged in the column direction of the matrix. This makes it possible to reduce the size of the second memory 140 compared to the case where the matrix is ​​rearranged in the im2col format and written to the second memory 140.

[0022] Furthermore, for example, the processor 110 may divide a matrix into a plurality of divided matrices in accordance with the size of the memory area of ​​the second memory 140 or the number of matrix multiplication operators of the accelerator 120, and rearrange the data into the im2col reduced format for each divided matrix. That is, the rearrangement of data into the im2col reduced format shown in Fig. 2 can be executed without depending on the configuration of the matrix multiplication operators of the accelerator 120. For this reason, the rearrangement process into the im2col reduced format shown in Fig. 2 can be executed for matrices of various sizes by various arithmetic processing devices having a processor and an accelerator.

[0023] The accelerator 120 reads data of each row of the matrix stored in the second memory 140 in the im2col reduced format, and performs a matrix multiplication operation between each of the three element groups of each row and the filter. For example, two-thirds of the elements in the 0th row of the matrix can be used for the matrix multiplication operation of the 1st row of the matrix, and one-third of the elements in the 0th row of the matrix can be used for the matrix multiplication operation of the 2nd row of the matrix.

[0024] Therefore, when the accelerator 120 can hold the elements of the 0th row of the matrix read from the second memory 140 in a register or the like, in the matrix multiplication operations of the 1st row and onward, the accelerator 120 can execute the matrix multiplication operations by simply reading one-third of the elements used in the matrix multiplication operations from the second memory 140. Therefore, the amount of data read from the second memory 140 to the accelerator 120 can be reduced, and the access load of the accelerator 120 to the second memory 140 can be reduced.

[0025] As described above, in this embodiment, the size of the second memory 140 can be reduced compared to the case where matrices are rearranged into the im2col format and written to the second memory 140. Furthermore, rearrangement of matrices into the im2col reduced format can be executed without depending on the configuration of the matrix multiplication unit of the accelerator 120. For this reason, the rearrangement process into the im2col reduced format shown in FIG. 2 can be executed on matrices of various sizes by various arithmetic processing devices 100 having the processor 110 and the accelerator 120. That is, it is possible to rearrange matrices into data in the im2col reduced format, which is data with less redundant portions, without depending on the configuration of the accelerator 120.

[0026] Fig. 3 illustrates an example of a processing device according to another embodiment. The processing device 200 illustrated in Fig. 3 includes a plurality of streaming multiprocessors (SMP) 210 and a global memory 220 that is used in common by the plurality of streaming multiprocessors 210. The streaming multiprocessors 210 are an example of an accelerator, and the global memory 220 is an example of a first memory. For example, the processing device 200 may be a GPU (Graphics Processing Unit). Hereinafter, the streaming multiprocessor 210 may be referred to as an SMP 210.

[0027] Each streaming multiprocessor 210 has a plurality of cores 212, a shared memory 214 shared by the plurality of cores 212, and a matrix multiplier (LMP) 216 operating as an accelerator. The cores 212 are an example of a processor, and the shared memory 214 is an example of a second memory. Hereinafter, the matrix multiplier 216 may be referred to as the LMP 216. For example, the cores 212 may have a function capable of executing operations on a plurality of data in parallel with one instruction, such as a single instruction multiple data (SIMD) operation or a single instruction multiple threads (SIMT) operation. Each SMP 210 may have a cache (for example, a last level cache (LLC)).

[0028] For example, the shared memory 214 may be a scratch pad memory that can operate with the same access latency as an LLC. The shared memory 214 may also include a cache. The scratch pad memory is, for example, a static random access memory (SRAM). For example, the global memory 220 has a large capacity and a high latency compared to the shared memory 214. The global memory 220 may be a three-dimensional stacked memory such as a high bandwidth memory (HBM).

[0029] The arithmetic processing device 200 may be in the form of a single chip, or in the form of multiple chips having a processor chip including multiple SMPs 210 and a memory chip including global memory 220. In this case, the processor chip and the memory chip may be connected to each other, for example, via wiring formed in an interposer disposed on a package substrate.

[0030] Fig. 4 shows an example of functional blocks of software executed by the arithmetic processing device 200 of Fig. 3. The arithmetic processing device 200 executes a kernel 300 to realize the operations shown in Fig. 5 and subsequent figures. The kernel 300 manages multiple threads 312 and enables multiple cores 212 to execute the multiple threads 312 in parallel.

[0031] Although not limited to this, the multiple threads 312 managed by the kernel 300 are included in one of multiple thread blocks 310 respectively assigned to the SMP 210. The threads 312 included in each thread block 310 are assigned to one of multiple thread groups. For example, the multiple threads 312 included in a thread group may operate in synchronization with each other and execute the same instruction (processing). The multiple threads 312 in each thread block 310 execute processing using information held in the shared memory 214 included in the SMP 210 corresponding to the thread block 310.

[0032] Fig. 5 shows an overview of the operation when the arithmetic processing device 200 of Fig. 3 executes convolution calculation of an input image. For example, the global memory 220 is allocated with storage areas for holding an input image (NCHW arrangement, NHWC arrangement), an input image (im2col reduced format), a filter (NCHW arrangement, for im2col reduced), and an output image (NHWC arrangement, NCHW arrangement). Note that the arithmetic processing device 200 may execute convolution calculation of a matrix other than the input image.

[0033] In the NCHW and NHWC arrangements, each alphabet indicates N as a batch, C as an image channel C, H as image height (columns), and W as image width (rows). In the NCHW arrangement, each element of an image is arranged by prioritizing image width W, image height H, channel C, and batch N in that order. In the NHWC arrangement, each element of an image is arranged by prioritizing channel C, image width W, image height H, and batch N in that order. For example, data sorting and convolution calculations by SMP 210 may be performed for the number of channels that can be processed at one time with the memory size available in shared memory 214.

[0034] First, the SMP 210 reads the input image in the NCHW arrangement from the global memory 220 into the shared memory 214, rearranges it into an input image in the NHWC arrangement, and stores it in the global memory 220 via the shared memory 214. Note that if an input image in the NHWC arrangement has been prepared in advance, rearrangement into the NHWC arrangement is not necessary.

[0035] Next, the SMP 210 reads the NHWC-arranged input image from the global memory 220 into the shared memory 214, converts it into an im2col reduced format, and stores it in the global memory 220 via the shared memory 214.

[0036] Furthermore, the SMP 210 reads the filter data of the NCHW array from the global memory 220 into the shared memory 214, converts it for use in the im2col reduced format, and stores it in the global memory 220 via the shared memory 214. The order of rearrangement of the input image and conversion to the im2col reduced format, and rearrangement of the filters may be reversed.

[0037] Next, the SMP 210 reads the input image (im2col reduced format) and the filter (for im2col reduced) from the global memory 220 into the shared memory, and executes a matrix multiplication operation (convolution calculation). The SMP 210 stores the result of the matrix multiplication operation as an output image (NHWC arrangement) in the global memory 220 via the shared memory 214.

[0038] Next, the SMP 210 reads the output image (NHWC arrangement) from the global memory 220 into the shared memory, rearranges it into the output image (NCHW arrangement), and stores it in the global memory 220 via the shared memory 214.

[0039] Although not particularly limited, the rearrangement of the input image and the filter may use a function (e.g., cudnnTransformTensor) of cuDNN (Deep Neural Network), which is a library provided by NVIDIA. Also, the matrix multiplication operation may use a matrix multiplication function (e.g., cublasGemmStridedBatchedEx) of cuBLAS (Basic Linear Algebra Subprograms), which is a library that executes linear operations provided by NVIDIA. Note that, although in the example shown in FIG. 5, the SMP 210 performs a convolution calculation on the input image and outputs an output image, the convolution calculation may also be performed on data other than an image.

[0040] Figure 6 shows an overview of the convolution calculation process. In the example shown in Figure 6, elements shown in shaded areas are added by zero padding around the image data of 5 vertical elements and 5 horizontal elements shown in the bold frame, and 49 elements are used as the input image. Then, while a window of 3 vertical elements and 3 horizontal elements is moved sequentially in the input image, a matrix multiplication operation (convolution calculation) is performed using a filter of 3 vertical elements and 3 horizontal elements, and an output image of the same size as the original image data is generated.

[0041] In the following description, an input image in which elements are added by 0 padding to image data is used, and an output image of the same size as the image data is generated. In the following description, each element of the input image, filter, and output image is also referred to as data.

[0042] Fig. 7 shows an example of data arrangement when input images are rearranged for convolution calculation. For simplicity of explanation, in Fig. 7, each input image shown in channels CH0-CH4 is assumed to have three elements vertically and three elements horizontally. That is, in Fig. 7, eight elements except for one element in the center of the input image are added by zero padding. Also, one batch is assumed to include input images of channels CH0-CH4.

[0043] In rearrangement into the NCHW layout, the highest priority is given to the image width W, followed by the image height H and then the channel C. Therefore, the elements in the 0th row, 1st row and 2nd row of channel CH0 are extracted in order and placed in a memory area etc., and similarly, the elements of the next channels CH1-CH4 are extracted in order and placed in a memory area etc.

[0044] In sorting to the NHWC arrangement, the priority is given to the channel C, the image width W, and the image height H, in that order. Therefore, elements are extracted one by one from all channels CH0-CH4, with the width W being given priority, and arranged in a memory area or the like. In this embodiment, the convolution calculation is performed using the NHWC arrangement. Therefore, when the input image is in the NCHW arrangement, it is sorted in advance to the input image in the NHWC arrangement.

[0045] Fig. 8 shows an example of the size of each matrix in the matrix product αAb+βC. A matrix product of m rows and n columns is obtained from matrix A of m rows and k columns and matrix B of k rows and n columns. Therefore, matrix C of m rows and n columns can be added to the result of the matrix product of matrices A and B. Note that α and β are coefficients (scalars).

[0046] FIG. 9 shows an example of data after rearrangement of an input image into the im2col format. For the sake of simplicity, FIG. 9 shows an example of rearrangement of one input image. The input image is in an NHWC arrangement (effectively a HW arrangement, since there is one batch number N and one channel number C). Rearrangement of the input image in im2col enables efficient execution of matrix multiplication with the filter. In FIG. 9, an input image of 10 rows and 10 columns with 100 elements is rearranged, with 0 padding elements added around the image data of 64 elements in 8 rows and 8 columns. The filter has 9 elements in 3 rows and 3 columns.

[0047] For example, in rearrangement into the im2col format, as shown in Fig. 6, nine elements to be sequentially convolved with a filter are arranged in each row of a memory area such as the shared memory 214 in Fig. 3. The nine elements shown in two bold frames in the memory area correspond to the nine elements shown in two bold frames in the input image, respectively.

[0048] The data used for convolution of the 0th row of the input image is arranged in 8 rows of the memory area. Similarly, the data used for convolution of the 1st row of the input image is arranged in the next 8 rows of the memory area. Then, the rearranged data of each row indicated by the row number in the input image is written sequentially in units of 8 rows into the memory area. Note that while FIG. 9 shows an example in which rearrangement is performed with priority given to the width W (rows) of the input image, rearrangement may also be performed with priority given to the height H (columns) of the input image.

[0049] In the im2col format, the same elements are written to the two dashed framed areas in the memory area where the rearranged data is written, which causes a problem of wasting memory area. In other words, the im2col format prioritizes the efficiency of matrix multiplication, which results in wasting memory area that holds data for the operation.

[0050] Fig. 10 shows an example of data after conversion of an input image into the im2col reduced format. The input image to be rearranged and the filter used for the convolution calculation are the same as those of the input image in Fig. 9. Also in Fig. 10, for the sake of simplicity, an example of rearranging one input image is shown.

[0051] A predetermined number of cores 212 of the SMP 210 shown in Fig. 3 writes a plurality of elements of each of a plurality of element groups arranged in the row direction of a matrix in the input image shown in Fig. 9 in a column along the second direction D2 to each of a plurality of rows arranged in the first direction D1 of the memory area. At this time, among a plurality of element groups arranged in the column direction in the input image, elements of adjacent element groups are written in a column along the second direction D2 while being overlapped. In other words, the core 212 writes a plurality of element groups arranged in the row direction in each row of the input image in a column along each of a plurality of rows of the memory area so as to sequentially overlap with a portion of the elements of the plurality of element groups arranged in the column direction of the input image.

[0052] In the im2col reduced format, the rearranged data is written to the memory area with some of the data used for the convolution of three rows of the input image overlapping. In the example shown in Fig. 10, two-thirds of the data used for the convolution of the first row overlaps with the data used for the convolution of the zeroth row, and one-third of the data used for the convolution of the second row overlaps with the data used for the convolution of the zeroth row.

[0053] In the im2col reduced format, in the memory area where the rearranged data is written, some of the data used for convolution of three rows of the input image are written in duplicate to the memory area, so that it is possible to reduce the frequency of redundantly writing the same data to the memory area. This makes it possible to reduce the usage of the shared memory 214, etc., where the rearranged data is written.

[0054] Therefore, it is possible to reduce the memory bandwidth required for writing rearranged data of the input image to the shared memory 214 etc., and the memory bandwidth required for reading data from the shared memory 214 etc. to execute the matrix multiplication operation. As a result, it is possible to improve the processing efficiency of the convolution calculation by the arithmetic processing device 200, compared to the case where data rearranged in the im2col format is used.

[0055] Note that, although FIG. 10 illustrates an example in which elements of the input image are rearranged into the im2col reduced format so as to overlap elements in the row direction, elements of the input image may also be rearranged into the im2col reduced format so as to overlap elements in the column direction.

[0056] FIG. 11 shows an example of the size of data after rearrangement of an input image in the im2col format and the im2col reduced format. The input image is in an NHWC arrangement (effectively an HW arrangement since the number of batches N and the number of channels C are one each). In FIG. 11, for the sake of simplicity, it is assumed that the input image has five elements vertically and five elements horizontally. For example, 16 elements around the nine elements of the central three elements vertically and three elements horizontally of the input image are added by zero padding. Note that, although FIG. 11 shows an example of rearrangement with priority given to the width W (rows), rearrangement with priority given to the height H (columns) may also be performed.

[0057] In sorting into the im2col format, 81 elements obtained by sorting are written to the memory area. In contrast, in sorting into the im2col reduced format, 45 elements obtained by sorting are written to the memory area. That is, in the example shown in FIG. 11, the size of the memory area where the data after sorting into the im2col reduced format is written can be 56% of the size of the memory area where the data after sorting into the im2col format is written.

[0058] The size of the elements obtained by rearranging the input image into the im2col format and the size of the elements obtained by rearranging the input image into the im2col reduced format are shown by the following formulas (1) and (2). The layout of the input image is assumed to be the NHWC layout [Batch, Iheight, Iwidth, Cin], and the layout of the filters is assumed to be [Kheight, Kwidth, Cin, Cout] as shown in FIG.

[0059] Batch・Iheight・Iwidth・Kheight・Kwidth・Cin ‥(1) Batch·[(Iheight+Kheight-1)·Kwidth·Cin]·Iwidth ‥(2) Here, when the height Iheight of the input image is sufficiently large, the size of the elements rearranged in the im2col reduced format can be set to 1 / Kheight of the size of the elements rearranged in the im2col format.

[0060] Fig. 12 shows an example in which a multi-channel image is input and a convolution calculation is performed, and then a multi-channel image is output. In Fig. 12, the symbol Cin indicates the number of channels of the input image, and the symbol Cout indicates the number of channels of the output image. The symbol Kwidth indicates the width of the filter, and the symbol Kheight indicates the height of the filter.

[0061] 12, a convolution calculation for three channels is performed using a three-channel input image representing a color image such as RGB and a three-channel filter, and an output image for one channel is generated. Then, convolution calculations for a plurality of color images such as RGB are respectively performed, and a plurality of output images are generated.

[0062] In the following, an example will be described in which four 3-channel input images are rearranged into an im2col reduced format and then a 4-channel output image is generated by convolution calculation, as shown in Fig. 12. However, the number of channels of the input image and the number of channels of the output image are not limited to the example shown in Fig. 12.

[0063] FIG. 13 shows an example of rearranging a multi-channel input image into an im2col reduced format. In FIG. 13, a three-channel input image of the same size as that in FIG. 9 is rearranged into an im2col reduced format. In FIG. 13, the symbol Iwidth indicates the width (column direction) of the input image, and the symbol Iheight indicates the height (row direction) of the input image. Each channel of the input image is represented by Cin0, Cin1, and Cin2. Note that in FIG. 13, the Iwidth direction and Iheight direction of the elements rearranged into the im2col reduced format are swapped with respect to FIG. 10.

[0064] When the input image has multiple channels, in the im2col reduced format, the elements of the channels Cin0-Cin2 are alternately arranged in the Iheight direction (row direction) of the memory area. For this reason, for example, the first nine elements shown in bold frames in each channel Cin0-Cin2 of the input image are written to the first column shown in bold frames in the memory area. Then, a one-channel output image can be obtained by performing a convolution calculation on the rearranged data shown in FIG. 13 using a three-channel filter. By repeating the process of rearranging the three-channel input images shown in FIG. 13 into the im2col reduced format and writing them to the memory area, a data string used to generate multiple output images by convolution calculation can be obtained.

[0065] FIG. 14 shows an example of the arrangement of elements of a multi-channel filter used to obtain a multi-channel output image. The elements of each filter and the elements of each input image before rearrangement are arranged in NHWC. The multi-channel filter is rearranged, for example, by prioritizing the number of channels Cout of the output image, the number of channels Cin of the input image, the width Kwidth of the file, and the height Kheight of the filter in this order. That is, the rearranged filter is rearranged with the number of channels Cout of the output image as the innermost and the height Kheight of the filter as the outermost, as shown in FIG. 14 as [Kheight, Kwidth, Cin, Cout].

[0066] In addition, when the input image is arranged in NHWC, the filter is often arranged in [Kheight, Kwidth, Cin, Cout]. In this case, rearrangement of the filter elements is not necessary.

[0067] In the example shown in Fig. 14, the multi-channel filter data after element rearrangement includes filter elements corresponding to four output images Cout0-Cout3 arranged in the Cout direction. The filter elements corresponding to the four output images Cout0-Cout3 may be the same as or different from each other. Each row of the filter arranged in the Cout direction includes elements used for convolution calculation of three-channel input images.

[0068] Fig. 15 shows an overview of a matrix multiplication operation performed using the multi-channel input image rearranged in Fig. 13 and the multi-channel filter rearranged in Fig. 14. For example, the matrix multiplication operation of the nine elements (9 elements × 3 channels shown in bold frames) of each channel Cin0-Cin2 arranged in the vertical direction of the memory area in the input image is performed using the elements shown in bold frames arranged in the horizontal direction of the memory area in the filter. The matrix multiplication operation may be performed in parallel (simultaneously) on eight elements of 9 elements × 3 channels arranged in the width direction Iwidth of the input image.

[0069] After the convolution calculation in the 0th row of the channels Cin0-Cin2, the convolution calculations in the 1st to 7th rows of the channels Cin0-Cin2 are sequentially performed using the same filter. That is, batch matrix multiplication is performed. At this time, as described in FIG. 10, some of the elements of the input image on which the convolution calculation is performed are the same as the elements used in the previous row and the elements used in the row two rows before.

[0070] As shown in Fig. 12, when there are four 3-channel input images, the matrix multiplication process shown in Fig. 15 is repeated four times (loop processing). In this case, the matrix multiplication process for the second 3-channel input image is performed using the elements in the second row of the filter. The matrix multiplication process for the third 3-channel input image is performed using the elements in the third row of the filter. The matrix multiplication process for the fourth 3-channel input image is performed using the elements in the fourth row of the filter.

[0071] 4, the elements of the four 3-channel input images corresponding to the four rows of the filter may be arranged in the Iwidth direction of the memory area. In this case, the matrix multiplication of the four 3-channel input images can be executed in parallel.

[0072] Fig. 16 shows an example of a code description for rearranging an input image into an im2col reduced format. The code shown in Fig. 16 is a source code written in the C language, and is executed by the core 212 after being compiled into an executable object code by the core 212 in Fig. 3. The numbers in brackets on the left side of the code description indicate line numbers.

[0073] For example, the function name of the code that rearranges the input image into the im2col reduced format is im2colreduced. The input image is pre-arranged into the NHWC arrangement, and in the image input in the first line, [BATCH] is the outermost and [INCH] is the innermost. In addition, the input image has multiple dimensions, but is expanded into one dimension and stored in a memory area such as the shared memory 214. The output image is an image that has been rearranged into the im2col reduced format.

[0074] Among the variables used in the code shown in Figure 16, the variables input from outside are listed below. BATCH: Number of batches HEIGHT: Height of the image (image data) WIDTH: Image (image data) width INCH: Image channel KERNEL_HEIGHT: The height of the filter. KERNEL_WIDTH: The width of the filter.

[0075] The constant PADY in the code indicates the number of 0 paddings in the height direction of the image, and the constant PADX in the code indicates the number of 0 paddings in the width direction of the image. The constants PADY and PADX are set as follows based on the KERNEL_HEIGHT and KERNEL_WIDTH input from outside. PADY=(KERNEL_HEIGHT-1) / 2 PADX=(KERNEL_WIDTH-1) / 2

[0076] The variables batch, height, width, inch, and kerw are index variables corresponding to the batch, height direction of the input image, width direction of the input image, channel direction of the input image, and width direction of the kernel, respectively. The variable tmp is used to temporarily store the elements of the input image that have been read. Note that when "BATCH" in "batch in BATCH" in the code is "3", the variable batch is 0, 1, or 2. When "BATCH" is "2", the variable batch is 0 or 1.

[0077] Fig. 17 shows an image of writing data to a memory area that is rearranged into the im2col reduced format by the code shown in Fig. 16. Writing data to the memory area is performed while shifting the position of the memory area to which the data is written in the loop from lines 10 to 15 of the code shown in Fig. 16.

[0078] For example, the data of one column of zero padding at the left and right ends of the input image shown in "(1)" is written once to the memory area, as shown by one circle. The data of the second column from the left and second column from the right ends of the input image shown in "(2)" is written twice to the memory area, as shown by two circles. The data of the third column from the left end of the input image and onwards, through the third column from the right end of the input image shown in "(3)", is written three times to the memory area, as shown by three circles.

[0079] Fig. 18 shows an example of the sorting process executed by the code shown in Fig. 16. Sorting of input images into the Im2col reduced format is performed on a batch-by-batch basis since it does not depend on batches (step S100).

[0080] In each batch, each element in the height direction of the input image, including 0 padding, is set as a sorting target according to the height of the filter, and sorting is performed for each height direction (each row) (step S110). In addition, in each batch, each element in the width direction of the input image, including 0 padding, is set as a sorting target according to the width of the filter, and sorting is performed for each width direction (each column) (step S120). Furthermore, sorting to the im2col reduced format is performed in the channel direction of the input image by the loop of step S130. As described above, the input image in the NHWC arrangement can be sequentially acquired and sorted by accessing elements in consecutive directions in the memory area.

[0081] As shown in Fig. 17, the input image to be processed for the convolution calculation contains zero padding. Therefore, in step S140, it is determined whether the element of the input image to be accessed is a zero padding area or an image data area. If it is a zero padding area, in step S150, the variable tmp is set to 0. If it is image data, in step S160, the element to be read of the image data is read as the variable tmp.

[0082] In the loop of step S170, the processes of steps S180 and S190 are executed for each position in the width direction of the filter (0, 1, and 2 in this embodiment). In steps S180 and S190, when the element acquired in steps S150 and S160 is used as the horizontal index variable kerw of the filter, the element is written in the corresponding column of the output image in the memory area and in the row corresponding to the index variable kerw.

[0083] For example, if the arithmetic processing device 200 in Fig. 3 has a SIMD arithmetic function or a SIMT arithmetic function capable of efficiently processing a large number of consecutive elements, it is possible to efficiently obtain a large number of elements of an input image. For example, when rearranging an input image in an NHWC arrangement, it is possible to efficiently obtain elements of the channel direction inch of the input image by placing the channel direction inch of the input image inside a loop.

[0084] In addition, since the channel direction inch of the input image is placed inside the loop and the elements of the input image are rearranged, the elements of the output image are also continuous in the channel direction inch. Therefore, the elements can be rearranged and written efficiently to the memory area as the output image. In addition, as shown in FIG. 17, the elements of the read input image can be written to the memory area a maximum number of times corresponding to the width of the filter, further improving the writing efficiency.

[0085] Fig. 19 shows an example of the processing of steps S140 and S180 in Fig. 18. In step S140, if the determination in one of steps S142 and S144 is negative (NO), the core 212 determines that the region is a zero padding region, and if the determination in both steps S142 and S144 is positive (YES), the core 212 determines that the region is not a zero padding region.

[0086] In step S180, if the determination in step S182 is affirmative (YES), core 212 determines that it is within the range of the output image, and if the determination in step S182 is negative (NO), it determines that it is not within the range of the output image.

[0087] Fig. 20 shows an example of a code description for performing a convolution calculation (matrix multiplication calculation) of an input image in the im2col reduced format. The code shown in Fig. 20 is a source code written in the C language style, similar to Fig. 16, and is compiled into an executable object code by the core 212 in Fig. 3, and then executed by the core 212. The numbers in brackets at the left end of the code description indicate line numbers.

[0088] For example, the function name of the code that executes the convolution calculation of an input image in the im2col reduced format is batch_gemm, and XGEMM is a function that executes a matrix multiplication operation. In the code shown in Fig. 20, an input image in the im2col reduced format and filters rearranged when the input image is arranged in NCHW are input, and the result of the convolution calculation is output. In Fig. 20, the same variables as those used in Fig. 16 are used. The variable OUTCH indicates the channel of the output image.

[0089] FIG. 21 shows an example of matrix elements stored in memory when performing a matrix multiplication operation. For example, the matrix assumed in the matrix multiplication operation included in BLAS, a library that performs linear operations, assumes that the elements of the matrix (e.g., input image) are stored in memory in column-major order. Also, in BLAS, it is possible to perform a matrix multiplication operation by specifying a submatrix included in a larger matrix. In this case, focusing on the first element on which the matrix multiplication operation is performed, the number of elements up to the first element of the next column is treated as the leading dimension, and is denoted as lda for matrix A.

[0090] Furthermore, if you want to transpose matrices A and B before using them in a matrix multiplication operation, you can call the function by specifying whether or not to transpose, as shown in the following example. dgemm ( character TRANSA, (←Specifies whether to transpose matrix A before calculation) character TRANSB, (←Specifies whether to transpose matrix B before calculation) integer M, integer N, integer K, double precision ALPHA, double precision,dimension(lda,*) A, integer LDA, (←leading dimension of matrix A) double precision, dimension(ldb,*) B, integer LDB, (←leading dimension of matrix B) double precision BETA, double precision, dimension(ldc,*) C, integer LDC (←leading dimension of matrix C) )

[0091] Fig. 22 shows an example of the convolution calculation (matrix multiplication) process executed by the code shown in Fig. 20. The convolution calculation (matrix multiplication) is executed on a batch-by-batch basis (step S200) since it does not depend on the batches. Note that the filter is common to the batches, so there is no need to move the pointer.

[0092] When sorted in the im2col reduced format, convolution is calculated by performing a matrix multiplication operation for each index in the height direction of the image. Therefore, by performing the matrix multiplication operation the number of times equal to the height HEIGHT of the image (image data), the convolution calculation of the entire input image can be performed (step S210).

[0093] In addition, in the input image in the im2col reduced format, images of multiple channels overlap in the height direction of the image, as shown in Fig. 13. For this reason, correct calculation can be performed by shifting the head pointer by the product of the filter width KERNEL_WIDTH and the input channel INCH for each loop. Since the output image, which is the result of the convolution calculation, is calculated for each batch in the height direction of the image, it is only necessary to shift by the product of the output channel OUTCH and the image (image data) width WIDTH for each loop (step S220).

[0094] Figure 23 shows an example of the results of convolution calculations using the im2col format and the im2col reduced format. The convolution calculations were performed using single-precision floating-point numbers (FP32) using the NVIDIA A100 GPU. The environment variable used was NVIDIA_TF32_OVERRIDE=0, and the cuDNN algorithm used was CUDNN_CONVOLUTION_FWD_ALGO_GEMM.

[0095] The convolution calculation was performed for six patterns in which the number of input channels, the size of the input image, the number of output channels, and the size of the filter were changed. In No. 1 to No. 3, the convolution calculation was performed by fixing the number of input channels, the size of the input image, and the number of output channels, and changing the size of the filter. In No. 4 to No. 6, the convolution calculation was performed by extracting a 3x3 filter portion from ResNet50, one of the benchmarks.

[0096] In No.1 to No.3, the sorting time in the im2col reduced format is shorter than that in the im2col format. Also, in No.1 to No.3, the additional memory usage is approximately one-fifth of the filter size (height or width) compared to the im2col format.

[0097] In the calculation of No. 3 using cuDNN, an additional 49 GB of memory was required to sort the data into im2col format, which exceeded the memory (40 GB) installed in the GPU that performed the convolution calculation. For this reason, it is presumed that another algorithm was run because the memory could not be secured.

[0098] For ResNet50, the sorting time with the im2col reduced format is longer than the sorting time with the im2col format, but the additional memory usage is almost one-third that of the im2col format.

[0099] Figure 24 shows an example of the execution time of each process in convolution calculation using the im2col reduced format. The batch size, number of input channels, input image size, number of output channels, and filter size for each of No.1 to No.6 are the same as those in Figure 23. For the convolution calculation, the cuBLAS matrix multiplication function cublasGemmStridedBatchedEx was used.

[0100] As shown in No. 4 and No. 6, when the size of the input image is relatively small, the execution time of the matrix multiplication operation is longer than the others, but this is presumably due to poor compatibility with the matrix multiplication function used.

[0101] Fig. 25 shows an example of a matrix multiplication operation when an input image in an NCHW arrangement is input to the arithmetic processing device 200 in Fig. 3. Detailed description of elements similar to those in Fig. 13 will be omitted. When the input image has an NCHW arrangement, the arithmetic processing device 200 rearranges the input image in the NCHW arrangement into an NHWC arrangement, and then rearranges it into the im2col reduced format.

[0102] Also, when the input image has an NCHW arrangement, a filter in the [Cout, Cin, Kheight, Kwidth] format corresponding to the NCHW arrangement is input to the arithmetic processing device 200. In this case, the arithmetic processing device 200 regards the filter as an image and converts it into an NHWC arrangement [Cout, Kheight, Kwidth, Cin]. Then, the arithmetic processing device 200 executes a matrix multiplication operation between the transposed filter and the input image in the im2col reduced format.

[0103] As described above, this embodiment can also provide the same effects as those of the above-mentioned embodiment. For example, rearrangement of an input image into an im2col reduced format can be executed without depending on the configuration of a matrix multiplication calculator mounted on the LMP216 and the size of the input image. Therefore, rearrangement processing into an im2col reduced format with less redundancy and convolution calculation processing can be executed by various arithmetic processing devices having cores and accelerators.

[0104] Since the input image can be rearranged into data in the im2col reduced format with less redundant parts, the size of the memory area for holding the rearranged data can be reduced. In other words, the number of elements of the input image that can be held in the shared memory 214 after rearrangement into the im2col reduced format can be increased compared to the case of rearrangement into the im2col format.

[0105] Furthermore, in this embodiment, elements at the same position in a group of elements at the same position in a plurality of input images are written adjacently in the same row of a memory area, thereby rearranging the plurality of input images into an im2col reduced format. Also, a plurality of filters used for the matrix multiplication operation of a plurality of input images arranged in the Cin direction are stored in a row of a memory area with elements at the same position in the plurality of filters adjacent to each other.

[0106] This makes it possible to execute the matrix multiplication of multiple input images in parallel using data that is rearranged into the im2col reduced format and written to the memory area for multiple input images arranged in the Cin direction. As a result, it is possible to efficiently use the matrix multiplication calculator that executes the matrix multiplication, thereby improving the processing performance of the convolution calculation.

[0107] Furthermore, when performing convolution calculations for a plurality of input image groups each including a plurality of input images arranged in the Cin direction and arranged in the Cout direction, the filters are stored in different rows of the memory area for each of the plurality of input image groups. By storing a plurality of filters to be used repeatedly in advance in the memory area and reading out the corresponding filters from the memory area to sequentially perform matrix multiplication calculations for the plurality of input images arranged in the Cin direction, the convolution calculations can be performed efficiently.

[0108] 26 shows an example of rearrangement of input images into the im2col reduced format by a processing device in another embodiment. The rearrangement into the im2col reduced format shown in FIG. 25 is executed by the processing device 200 shown in FIG.

[0109] For example, the arithmetic processing device 200 divides the input image into a plurality of images IMG (IMG0-IMG3) and sorts each image IMG into the im2col reduced format. In this case, the arithmetic processing device 200 can hide the second and subsequent sorting processes by performing a matrix multiplication process on the sorted image IMG every time the image IMG is sorted. Therefore, when the input images are sorted into the im2col reduced format to suppress the usage of the shared memory 214, the time required for the sorting process can be made invisible.

[0110] As described above, this embodiment can also provide the same effects as the above-mentioned embodiment. For example, rearrangement of an input image into an im2col reduced format can be executed without depending on the configuration of a matrix multiplication calculator mounted on the LMP216 in Fig. 3 and the size of the input image. Therefore, rearrangement processing into an im2col reduced format with less redundancy and convolution calculation processing can be executed by various arithmetic processing devices having cores and accelerators.

[0111] Furthermore, in this embodiment, by sorting each image IMG obtained by dividing the input image into the im2col reduced format and performing a matrix multiplication operation on the sorted images IMG, the sorting process can be hidden by the matrix multiplication operation. Therefore, when the input image is sorted into the im2col reduced format to suppress the usage of the shared memory 214, the time required for the sorting process can be made invisible.

[0112] 27 shows an example of filter movement when there is a stride in a calculation processing device of another embodiment. The processes shown in Fig. 27 to Fig. 30 are executed by the calculation processing device 200 of Fig. 3.

[0113] When there is a stride, the size of an element obtained by rearranging an input image into the im2col format and the size of an element obtained by rearranging an input image into the im2col reduced format are expressed by formulas (3) and (4), respectively. In formulas (3) and (4), the symbol stridey indicates the stride width in the height direction, and the symbol stridex indicates the stride width in the width direction. Note that the arrangement of the input image is assumed to be the NHWC arrangement [Batch, Iheight, Iwidth, Cin], and the arrangement of the filter is assumed to be [Kheight, Kwidth, Cin, Cout] shown in FIG. 14. Batch・Iheight / stridey・Iwidth / stridex・Kheight・Kwidth・Cin ‥(3) Batch·[(Iheight+Kheight-1)·Kwidth·Cin]·Iwidth / stridex ‥(4)

[0114] As shown in FIG. 27, a part of the filter is overlapped. When elements in the vertical direction are overlapped, (stride width stridey)<(filter height Kheight). When (stride width stridey)<(filter height Kheight), the number of sorting elements in the Im2col reduced format can be made smaller than the number of sorting elements in the im2col format.

[0115] FIG. 28 shows an example of a process of sorting an input image into an im2col reduced format when there is a stride. The process shown in FIG. 28 is executed by the core 212 in FIG. 3. The same step numbers are indicated for the same processes as in FIG. 18, and detailed descriptions are omitted. The sorting process shown in FIG. 28 is the same as the sorting process shown in FIG. 18, except that steps S180A and S190A are executed instead of steps S180 and S190 in FIG. 18.

[0116] In the sorting process shown in Fig. 28, data is written to the memory area in step S190A for each stride width sx. Therefore, in step S180A, it is confirmed whether the element (data) to be written to the memory area is an element to be written. As shown in Fig. 27, since the filters overlap each other in the height direction, elements are written out in the height direction regardless of the number of strides.

[0117] Fig. 29 shows an example of the process of step S180A in Fig. 28. In order to skip elements when there is a stride, when rearranging the input image, the core 212 checks whether an element of interest in the horizontal direction of the input image is an element to be written and whether it is within the writing range.

[0118] In step S181A, if the width of the input image "width" is divisible by the width of the kernel "kerw", the core 212 executes step S182A, and if not, the element is not to be output, so the core 212 ends the processing shown in Fig. 29. If the determination in step S182A is positive (YES), the core 212 determines that it is within the range of the output image, and if the determination in step S182A is negative (NO), the core 212 determines that it is not within the range of the output image.

[0119] Fig. 30 shows an example of the convolution calculation (matrix multiplication) process of an input image in the im2col reduced format when there is a stride. Detailed explanations of the same processes as in Fig. 22 are omitted. In the height loop, the LMP 216 skips elements by the stride and executes the convolution calculation, so executes the matrix multiplication process for each stride width sx.

[0120] For example, the LMP 216 in Fig. 3 performs horizontal convolution calculation of the input image in one matrix multiplication operation. In this case, the number of horizontal convolutions of the input image is reduced because the convolution calculation is performed for each stride width. When there is a stride, the number of times the image is output in the height direction is reduced in the output image direction compared to when there is no stride, so the pointer that shifts the output image in the height direction is adjusted by the stride width.

[0121] As described above, this embodiment can also provide the same effects as those of the above-mentioned embodiment. For example, rearrangement of an input image into an im2col reduced format can be executed without depending on the configuration of a matrix multiplication calculator mounted on the LMP216 and the size of the input image. Therefore, rearrangement processing into an im2col reduced format with less redundancy and convolution calculation processing can be executed by various arithmetic processing devices having cores and accelerators.

[0122] Furthermore, in this embodiment, even when a stride is present, the memory capacity of the memory area for storing the input image used in the convolution calculation can be reduced by rearranging the data into the im2col reduced format and executing the convolution calculation.

[0123] Fig. 31 illustrates an example of a processing device according to another embodiment. The processing device 400 illustrated in Fig. 31 includes at least one core memory group (CMG) 410, a matrix multiplication processor (LMP: Large Matrix-multiplication Processor) 420, a cache (LLC: Last Level Cache) 430 shared by the multiple core memory groups 410, and a memory 440. Hereinafter, the core memory group 410 will also be referred to as CMG 410, the matrix multiplication processor 420 will also be referred to as LMP 420, and the cache 430 will also be referred to as LLC 430.

[0124] Each CMG 410 has multiple cores 412 and a cache 414 shared by the multiple cores 412. Each core 412 may have a first cache, in which case the cache 414 functions as a second cache. The CMG 410 is an example of a core group, the LLC 430 is an example of a second memory, and the memory 440 is an example of a first memory. The LMP 420 is an example of an accelerator.

[0125] The arithmetic processing device 400 may have a one-chip form or a form including multiple chips. In the case of a multiple-chip form, the CMG 410, the LLC 430, the LMP 420, and the memory 440 may each have a chip form, and the arithmetic processing device 400 may function as an information processing device such as a server. The LMP 420 is an example of an accelerator that can operate asynchronously with the core 412 and execute a matrix multiplication operation.

[0126] For example, the memory 440 may be a main memory such as an HBM or a DDR-SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory). The LLC 430 may include a scratch pad memory, or the scratch pad memory may be arranged in place of the LLC 430.

[0127] FIG. 32 shows an example of a process of rearranging a matrix into an im2col reduced format and a convolution calculation process by the arithmetic processing device 400 of FIG. 31. Detailed description of operations similar to those of the above-mentioned embodiment will be omitted. For example, the arithmetic processing device 400 may rearrange the elements of a matrix for each matrix of an input image, etc., or may rearrange the elements of a matrix for each divided matrix obtained by dividing a matrix of an input image, etc. into four, as shown in FIG. 26. Furthermore, the arithmetic processing device 400 may rearrange the matrices of the multi-channel input image, etc. shown in FIG. 13 for each matrix or for each divided matrix.

[0128] For example, the core 412 rearranges the elements of a matrix for each group of elements to be rearranged. Each core 412 may rearrange the elements included in each of the regions obtained by further dividing a matrix into a plurality of divided matrices. By rearranging the elements included in the divided matrices obtained by dividing a matrix or the regions obtained by further dividing each of the divided matrices, it is possible to execute a plurality of rearrangement processes and a plurality of matrix multiplication processes in parallel, as shown in FIG. 26. In addition, the time required for the rearrangement process can be hidden by the matrix multiplication. One loop of the flow shown in FIG. 32 shows an example in which the elements of the divided matrices obtained by dividing a matrix are rearranged by each core 412.

[0129] In the flow shown in FIG. 32, the rearrangement of matrix elements is performed by the core 412 in FIG. 31, and the matrix multiplication operation is performed by the matrix multiplication unit 420 in FIG.

[0130] First, in step S300, the arithmetic processing device 400 operates one or more cores 412 to rearrange matrix data of an input image, etc. Before the rearrangement process, the matrix of the input image, etc. is transferred from the memory 440 to the LLC 430 or a scratch pad memory in the LLC 430. The rearranged data may be held in the LLC 430 or may be held in the scratch pad memory in the LLC 430.

[0131] When a matrix is ​​sorted into the im2col reduced format, the memory size for holding the sorted data can be made smaller than when the matrix is ​​sorted into the im2col format. In other words, the limited memory capacity can be used effectively. Therefore, when the sorted data is held in the LLC 430, the possibility of the data being written back to the memory 440 can be reduced, and a decrease in the processing efficiency of the subsequent matrix multiplication operation can be suppressed.

[0132] Next, in step S310, the core 412 instructs the LMP 420 to execute a matrix multiplication operation using the rearranged data and the filter. Next, in step S320, the LMP 420 executes the matrix multiplication operation instructed by the core 412, and stores the execution result in a memory. For example, the memory in which the execution result is stored is the LLC 430 or a scratch pad memory in the LLC 430.

[0133] Next, in step S330, when rearrangement of all elements of the matrix or partitioned matrix is ​​completed, each core 412 executes step S340. When rearrangement of elements of the matrix or partitioned matrix is ​​not completed, each core 412 returns to step S300. In step S340, each core 412 waits for completion of the matrix multiplication operation by the LMP 420, and when the matrix multiplication operation is completed, ends the process shown in FIG.

[0134] As described above, this embodiment can also provide the same effects as the above-mentioned embodiment. For example, rearrangement of an input image into an im2col reduced format can be executed without depending on the configuration of a matrix multiplication calculator mounted on the LMP420 and the size of the input image. Therefore, rearrangement processing into an im2col reduced format with less redundancy and convolution calculation processing can be executed by various scales of arithmetic processing devices having cores and accelerators.

[0135] Regarding the above embodiment, the following supplementary notes are further disclosed. (Appendix 1) A processing device for performing convolution calculations on a matrix including a plurality of elements, comprising: a first memory for holding said matrix; a second memory for storing data obtained by rearranging the elements of the matrix stored in the first memory; a processor that writes a group of elements arranged in a row direction, which are used for a matrix multiplication operation with a filter in each row of the matrix, into each of a plurality of rows of the second memory in a single column so as to sequentially overlap with some of the elements of a group of elements arranged in a column direction of the matrix, thereby rearranging the elements of the matrix and storing them in the second memory; and an accelerator that performs a matrix multiplication operation between a group of elements, some of which are written to the second memory in a duplicated manner, and the filter, thereby performing a convolution calculation of the matrix. Processing unit. (Appendix 2) When the accelerator executes a plurality of convolution calculations of the matrices, The processor writes elements in the same positions of the element groups in the plurality of matrices adjacent to each other in the same row of the second memory. 2. The processor of claim 1. (Appendix 3) The plurality of matrices whose elements are to be rearranged and stored in the first memory are in a so-called NHWC format, in which the matrices are arranged in the order of a channel, which is each of the plurality of matrices, a height direction of the matrix, a width direction of the matrix, and a batch including the plurality of matrices. 3. The processor according to claim 2. (Appendix 4) The filters used for the matrix multiplication of the matrices are stored in one row of the second memory with elements at the same positions in the filters adjacent to each other. 3. The processor according to claim 2. (Appendix 5) When performing convolution calculations on a plurality of matrix groups each including a plurality of the matrices, The filters are stored in different rows of the second memory for each of the plurality of matrix groups. 5. The processor according to claim 4. (Appendix 6) The matrix is ​​a divided matrix obtained by dividing a matrix having a size larger than the matrix into a plurality of divided matrices, The processor rearranges elements of the plurality of partitioned matrices in order; The accelerator sequentially executes a matrix multiplication operation of the partitioned matrices whose elements have been rearranged; The rearrangement of the elements of the second and subsequent partitioned matrices is performed in parallel with the matrix multiplication of the other partitioned matrices. 6. The processor according to claim 1, (Appendix 7) the matrix includes target data for a matrix multiplication operation and zero padding to be added to the target data; Each row of the matrix is ​​a row group including one row of the target data and a predetermined number of rows adjacent to both sides of the one row. 6. The processor according to claim 1, (Appendix 8) A core group having a plurality of cores; a first memory for holding a matrix including a plurality of elements; a second memory for storing data obtained by rearranging the elements of the matrix stored in the first memory; an accelerator for performing a matrix multiplication operation; the plurality of cores rearrange elements of the divided matrices obtained by dividing the matrix into a plurality of parts, write the rearranged data to the second memory, and then instruct the accelerator to execute a matrix multiplication operation on the rearranged data; the accelerator executes a matrix multiplication operation on the rearranged data based on the instruction, and writes a result of the operation to the second memory; When there is an unsorted divided matrix among the plurality of divided matrices, the plurality of cores perform a sorting operation, write the sorted data to the second memory, and instruct the execution of a matrix multiplication operation of the sorted data, and when sorting of all of the plurality of divided matrices is completed, wait for the completion of the matrix multiplication operation by the accelerator. Processing unit. (Appendix 9) A method for operating a processor having a first memory for storing a matrix including a plurality of elements, a second memory for storing data obtained by rearranging the elements of the matrix stored in the first memory, a processor, and an accelerator, comprising: the processor writes a group of elements arranged in a row direction, which are used for a matrix multiplication operation with a filter in each row of the matrix, into each of a plurality of rows of the second memory in a single column so as to sequentially overlap with a portion of elements of a group of elements arranged in a column direction of the matrix, thereby rearranging the elements of the matrix and storing them in the second memory; The accelerator executes a matrix multiplication operation between a group of elements, some of which are written to the second memory in a duplicated manner, and the filter, thereby executing a convolution calculation of the matrix. A method of operating a processor.

[0136] The features and advantages of the embodiments will be apparent from the above detailed description. It is intended that the claims cover the features and advantages of the embodiments as described above without departing from the spirit and scope of the claims. In addition, any improvements and modifications can be easily made by a person having ordinary skill in the art. Therefore, it is not intended to limit the scope of the inventive embodiments to the above, and appropriate improvements and equivalents within the scope of the embodiments disclosed can be used. [Explanation of symbols]

[0137] 100 Processing unit 110 Processor 120 Accelerator 130 First Memory 140 Second Memory 200 Processing unit 210 Streaming Multiprocessor (SMP) 212 cores 214 Shared Memory 216 Matrix multiplication unit (LMP) 220 Global Memory 300 Kernels 310 Thread Block 312 threads 400 Processing unit 410 Core Memory Group (CMG) 412 cores 414 Cache 420 Matrix product operator (LMP) 430 Cash (LLC) 440 Memory

Claims

1. A processing device for performing convolution calculations on a matrix including a plurality of elements, comprising: a first memory for storing said matrix; a second memory for storing data obtained by rearranging the elements of the matrix stored in the first memory; a processor that writes a group of elements arranged in a row direction, which are used for a matrix multiplication operation with a filter in each row of the matrix, into each of a plurality of rows of the second memory in a single column so as to sequentially overlap with some of the elements of a group of elements arranged in a column direction of the matrix, thereby rearranging the elements of the matrix and storing them in the second memory; an accelerator that performs a matrix multiplication operation between a group of elements, some of which are written to the second memory in a duplicated manner, and the filter, thereby performing a convolution calculation of the matrix; Processing unit.

2. When the accelerator executes a plurality of convolution calculations of the matrices, The processor writes elements at the same positions of the element groups at the same positions in the plurality of matrices adjacent to each other in the same row of the second memory. The processor according to claim 1 .

3. The filters used for the matrix multiplication of the matrices are stored in one row of the second memory with elements at the same positions in the filters adjacent to each other. The processor according to claim 2 .

4. When performing convolution calculations on a plurality of matrix groups each including a plurality of the matrices, The filters are stored in different rows of the second memory for each of the plurality of matrix groups. The processor according to claim 3 .

5. The matrix is ​​a divided matrix obtained by dividing a matrix having a size larger than the matrix into a plurality of divided matrices, The processor rearranges elements of the plurality of partitioned matrices in order; The accelerator sequentially executes a matrix multiplication operation of the partitioned matrices whose elements have been rearranged; The rearrangement of the elements of the second and subsequent partitioned matrices is performed in parallel with the matrix multiplication of the other partitioned matrices. The processor according to any one of claims 1 to 4.

6. A core group having a plurality of cores; a first memory for holding a matrix including a plurality of elements; a second memory for storing data obtained by rearranging the elements of the matrix stored in the first memory; an accelerator for performing a matrix multiplication operation; the plurality of cores rearrange elements of the divided matrices obtained by dividing the matrix into a plurality of parts, write the rearranged data to the second memory, and then instruct the accelerator to execute a matrix multiplication operation on the rearranged data; The accelerator executes a matrix multiplication operation on the rearranged data based on the instruction, and writes a result of the operation to the second memory; When there is an unsorted divided matrix among the plurality of divided matrices, the plurality of cores sort the divided matrices, write the sorted data to the second memory, and instruct the execution of a matrix multiplication operation of the sorted data, and when sorting of all of the plurality of divided matrices is completed, wait for completion of the matrix multiplication operation by the accelerator. Processing unit.

7. A method for operating a processor having a first memory that holds a matrix including a plurality of elements, a second memory that holds data obtained by rearranging the elements of the matrix held in the first memory, a processor, and an accelerator, comprising: the processor writes a group of elements arranged in a row direction, which are used for a matrix multiplication operation with a filter in each row of the matrix, into each of a plurality of rows of the second memory in a single column so as to sequentially overlap with a portion of elements of a group of elements arranged in a column direction of the matrix, thereby rearranging the elements of the matrix and storing them in the second memory; The accelerator executes a matrix multiplication operation between a group of elements, some of which are written to the second memory in a duplicated manner, and the filter, thereby executing a convolution calculation of the matrix. A method of operating a processor.

Images