Division system determination method, determination device, calculation system, and program

Abstract

To solve the problem in which it is not easy to determine an optimal policy since there are various conceivable division policies when implementing calculation that can be parallelized by recursive division on a hierarchical memory type SIMD computer.SOLUTION: One aspect of the present disclosure relates to a division system determination method that includes: a step in which one or more processors calculate data related to a data transfer time based on a transfer system determined from parallelization axes, the size of a problem to be calculated, and communication bandwidth between layers, about each combination of the parallelization axes in each layer of a hierarchical memory type computer; and a step in which the one or more processors determine the combination of specific parallelization axes based on the data related to the data transfer time calculated for each combination of the parallelization axes.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , ,

[0005] , ,

[0001] The present disclosure relates to a method for determining a partitioning method, a determining apparatus, a computing system, and a program.

Background Art

[0002] A parallel computer can execute computational processing quickly and efficiently by simultaneously using a plurality of processors. In parallel computing, a problem to be computed is divided into small tasks, and then each processor processes them in parallel. As a specific example of a parallel computer, a hierarchical memory type SIMD (Single Instruction Multiple Data) parallel computer used in a supercomputer or the like having a hierarchical memory architecture can be mentioned. Such a computer has a hierarchical structure consisting of a total of N + 1 layers of memories from the memory at the highest layer (LN memory) that stores data to be computed to the memory at the lowest layer (L0 memory) directly connected to a main arithmetic unit.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When implementing a computation that can be parallelized by recursive partitioning on a hierarchical memory type SIMD computer, there are a wide variety of conceivable partitioning policies, but it is not easy to determine the optimal one.

Means for Solving the Problems

[0005] To solve the above problems, one aspect of the present disclosure relates to a method for determining a partitioning scheme, comprising the steps of: one or more processors calculating data relating to data transfer time for each combination of parallelization axes in each layer of a hierarchical memory computer, based on a transfer method determined from the parallelization axes, the size of the problem to be computed, and the communication bandwidth between layers; and one or more processors determining a specific combination of parallelization axes based on the data transfer time data calculated for each combination of parallelization axes. [Brief explanation of the drawing]

[0006] [Figure 1] This is a schematic diagram showing a calculation system according to one embodiment of the present disclosure. [Figure 2] This is a block diagram showing the functional configuration of a parallelization axis determination device according to one embodiment of the present disclosure. [Figure 3] This is a schematic diagram showing a data transfer method according to one embodiment of the present disclosure. [Figure 4] This is a schematic diagram showing a data transfer method according to one embodiment of the present disclosure. [Figure 5] This is a schematic diagram showing a data transfer method according to one embodiment of the present disclosure. [Figure 6] This is a schematic diagram showing a data transfer method according to one embodiment of the present disclosure. [Figure 7] This is a schematic diagram illustrating a data transfer process according to one embodiment of the present disclosure. [Figure 8] This is a schematic diagram illustrating a data transfer process according to one embodiment of the present disclosure. [Figure 9] This is a schematic diagram illustrating a data transfer process according to one embodiment of the present disclosure. [Figure 10] This is a flowchart showing the parallelization axis determination process according to one embodiment of the present disclosure. [Figure 11] This is pseudocode illustrating a procedure for determining parallelization axes according to one embodiment of the present disclosure. [Figure 12] This figure shows the simulation results according to one embodiment of the present disclosure. [Figure 13]This is a block diagram showing the hardware configuration of a parallelization axis determination device according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0007] Embodiments of the present disclosure are described below. The following embodiments disclose a method for determining a partitioning scheme and a determination apparatus for a computer having a hierarchical memory architecture, such as a hierarchical memory SIMD parallel computer. [Summary of this disclosure] In summary, the decision device (hereinafter referred to as the full parallelization axis decision device) calculates data on data transfer time for each combination of parallelization axes in each layer of a hierarchical memory computer, based on the data transfer method determined by the parallelization axes, the size of the problem, and the communication bandwidth between layers, for problems that can be parallelized by recursive partitioning.

[0008] The parallelization axis refers to the method of partitioning a problem that can be parallelized by recursive partitioning. For example, if the problem is the calculation of the dense matrix product A × B = C, there are two ways to partition A into submatrix products for parallelization: partitioning A column-wise and partitioning A row-wise. Each of these is a parallelization axis. In problems that can be parallelized by recursive partitioning, the same parallelization axis exists again in the partitioned problems. When performing parallelization on a hierarchical memory computer, it is necessary to determine the parallelization axis at each layer.

[0009] Determining the parallelization axis means determining the data transfer method between layers. If the computer has the corresponding data transfer method between the layers of interest, a partitioning method along that parallelization axis can be implemented. Typical data transfer methods that hierarchical memory computers have at each layer include broadcast, distribute, reduce, and combine. In the dense matrix multiplication example described above, the selection of two parallelization axes can be achieved by combining these four data transfer methods.

[0010] In a hierarchical memory computer, the main arithmetic units are directly connected to L0 memory, where most calculations are performed. Assuming that the overall amount of computation does not change significantly with the selection of parallelization axes, the difference in processing time for each selection is mainly determined by the difference in data transfer time between LN memory and L0 memory. Therefore, the combination of parallelization axes that minimizes this data transfer time may be optimal for increasing the processing speed of the program. Even when the amount of computation changes as a result of axis selection, it is sufficient to choose the combination that minimizes the sum of data transfer time and computation time, and a fully parallelized axis determination device can be built with simple extensions. For example, consider a case where the memory of a hierarchical memory computer consists of three layers: the lowest layer L0 memory directly connected to the main arithmetic units, the next layer L1 memory, and the highest layer L2 memory. First, the all-parallel axis determination device selects one of all possible combinations of parallelization axes, i.e., for each all-parallel axis, it calculates the data transfer time between L0 memory and L1 memory and between L1 memory and L2 memory based on the problem size in each layer derived from the size of the problem to be computed (for example, the number of elements in each dimension of the matrix in a problem involving the calculation of dense matrix products) and the communication bandwidth between each layer. Then, it determines the largest of the calculated data transfer times as the data transfer time for that all-parallel axis (Step 1). Next, it determines the combination of parallelization axes with the smallest data transfer time among the data transfer times calculated for each combination of data transfer methods as the optimal all-parallel axis (Step 2). For example, in this specific example, if the data transfer time for the all-parallel axis where the parallelization axis in L0 memory is in the row direction and the parallelization axis in L1 memory is in the column direction is the smallest among all all-parallel axes, the all-parallel axis determination device determines the all-parallel axis with the smallest data transfer time as the optimal all-parallel axis. Here, "all parallelization axes" refers to the entire set of combinations of parallelization axes, such as "row direction or column direction" being a "parallelization axis," "row direction at layer 0, row direction at layer 1, ..." being a "total parallelization axis," and "row direction at layer 0, row direction at layer 1, ... or row direction at layer 0, column direction at layer 1, ..." being "all parallelization axes."What needs to be calculated is the best "fully parallelized axis", and for that, it is necessary to consider "all fully parallelized axes". The reason for the two appearances of "all" is that we are discussing the combination of combinations.

[0011] According to the fully parallelized axis determination device of the present disclosure, it is possible to automatically select the parallelized axis in each memory layer of a hierarchical memory type computer that was conventionally performed manually. [Computing System] First, referring to FIG. 1, a computing system according to an embodiment of the present disclosure will be described. FIG. 1 is a schematic diagram showing a computing system according to an embodiment of the present disclosure. As shown in FIG. 1, the computing system 10 includes a hierarchical memory type computer 50, an external computer 60, and a fully parallelized axis determination device 100.

[0012] The hierarchical memory type computer 50 is composed of a hierarchical memory from L1 memory to LN memory, and a plurality of processing elements (PEs) composed of an arithmetic unit and L0 memory. For example, it may be a hierarchical memory type SIMD parallel computer. Further, the hierarchical memory type computer 50 may have a communication interface (IF) for communicating with the external computer 60. The communication IF may be communicatively connected to the LN memory of the highest layer without being limited thereto.

[0013] For example, when the hierarchical memory type computer 50 is composed of three memory layers of L0 layer, L1 layer, and L2 layer, the L2 memory may be realized by DRAM (Dynamic Random Access Memory), the L1 memory and L0 memory may be realized by SRAM (Static Random Access Memory), and the arithmetic unit may be realized by a double-precision inner product arithmetic unit. Here, when the problem to be calculated is the calculation of a dense matrix product (A×B = C), the implementation of this problem means reading matrices A and B from DRAM to the PE and writing back the calculation result C in the PE to DRAM. Note that some operation such as reduction may be performed during the write-back to the highest-level memory.

[0014] The external computer 60 is a computer outside the hierarchical memory type computer 50 and is communicatively connected to the hierarchical memory type computer 50. The external computer 60 may be any computer, or may be the same or similar computer as the hierarchical memory type computer 50.

[0015] The all-parallelization axis determination device 100 is communicatively connected to the hierarchical memory type computer 50 and determines a parallelization axis indicating a combination of memory layers in the hierarchical memory type computer 50, as will be described later.

[0016] In the illustrated embodiment, the all-parallelization axis determination device 100 is realized as an external device independent of the hierarchical memory type computer 50. However, the all-parallelization axis determination device 100 according to the present disclosure is not limited thereto and may be mounted in the hierarchical memory type computer 50. Further, the all-parallelization axis determination device 100 may be further communicatively connected to the external computer 60 and may also determine the parallelization axis of the hierarchical memory in not only the hierarchical memory type computer 50 but also the external computer 60. [All-parallelization axis determination device] Next, referring to FIGS. 2 to 9, the all-parallelization axis determination device 100 according to an embodiment of the present disclosure will be described. FIG. 2 is a block diagram showing a functional configuration of the all-parallelization axis determination device 100 according to an embodiment of the present disclosure. As shown in FIG. 2, the all-parallelization axis determination device 100 includes a data transfer time calculation unit 110 and an optimal all-parallelization axis determination unit 120.

[0017] The data transfer time calculation unit 110 calculates the data transfer time for each combination of parallelization axes in each layer of the hierarchical memory type computer 50 based on the data transfer method determined by the parallelization axis, the size of the problem to be calculated, and the communication bandwidth between the layers. The calculated data transfer time is an example of data regarding the data transfer time.

[0018] For example, a data transfer method may include broadcasting, distribution, reduction, and combining. Broadcasting, as shown in Figure 3, is a data transfer method that transfers the same data from a source memory to all destination memories connected to that source memory. Distribution, as shown in Figure 4, is a data transfer method that transfers equally divided data from a source memory to all destination memories connected to that source memory. Reduction, as shown in Figure 5, is a data transfer method that reads data of the same size from all source memories connected to a destination memory, performs any operation (e.g., summation), and transfers the result to that destination memory. Combining, as shown in Figure 6, is a data transfer method that reads data of the same size from all source memories connected to a destination memory, combines the read data, and transfers the result to that destination memory.

[0019] However, the data transfer schemes described herein are not limited to broadcasting, distribution, reduction, and combination, and any other suitable data transfer scheme may be used. Such data transfer schemes are available if hardware implementations are available at each layer.

[0020] The data transfer time calculation unit 110 calculates the data transfer time between layers for all combinations of parallelization axes in the memory layers of the hierarchical memory computer 50. For example, if the hierarchical memory computer 50 consists of three memory layers L0, L1, and L2, and the problem to be computed has two parallelization axes, then two parallelization axes are selected for each of L1 and L0, resulting in a total of 22 possible parallelization axes. The data transfer time calculation unit 110 calculates the data transfer time for all 22 of these possible parallelization axes.

[0021] Once all parallelization axes are determined, the data transfer time calculation unit 110 can determine the problem size for each layer of the problem to be calculated. For example, if the problem to be calculated is to find the matrix product (A × B = C), there are two parallelization axes, one in the column direction and one in the row direction, and the problem size is a set of the number of rows in A (= number of rows in C), the number of columns in A (= number of rows in B), and the number of columns in B (= number of columns in C).

[0022] In column-direction parallelization, as shown in Figure 7, matrix A is decomposed columnarily and matrix B rowarily, and the matrix product C can be obtained by adding the multiple matrices obtained by multiplying each submatrix subA and subB. In this case, distribution can be used to decompose matrices A and B, and contraction can be used to add the multiple matrices obtained by multiplying each submatrix. When column-direction parallelization is performed, the number of columns in A becomes smaller in the next layer.

[0023] In row-direction parallelization, as shown in Figure 8, matrix A can be decomposed row by row, and the matrix product C can be obtained by combining multiple submatrices obtained by multiplying each submatrix with matrix B. In this case, distribution is used to decompose matrix A, broadcast is used for matrix B, and join can be used to combine the multiple matrices obtained by multiplying each submatrix. When row-direction parallelization is performed, the number of rows in A decreases in the next layer.

[0024] The optimal total parallelization axis determination unit 120 determines a specific combination of parallelization axes based on the data transfer time calculated for each combination of parallelization axes. For example, by combining the two types of parallelization axes described above, it is possible to perform matrix multiplication calculations as shown in Figure 9. In the illustrated embodiment, the memory layer consists of four layers, and the parallelization axes are determined independently for each layer. Specifically, the parallelization axes for Layer 2, Layer 1, and Layer 0 are row, row, and column directions. For matrix A, the data transfer methods for Layer 3 → Layer 2, Layer 2 → Layer 1, and Layer 1 → Layer 0 are all distribution, and for matrix B, the data transfer methods for Layer 3 → Layer 2, Layer 2 → Layer 1, and Layer 1 → Layer 0 are broadcast, broadcast, and distribution, respectively. Furthermore, for matrix multiplication C, the data transfer methods for Layer 0 → Layer 1, Layer 1 → Layer 2, and Layer 2 → Layer 3 are reduction by sum, join, and join, respectively.

[0025] As shown in the figure, in each layer, matrices A and B are transferred from layer 3 to layer 0, divided into submatrices according to the parallelization axis, the calculation results in layer 0 are aggregated via the data transfer method from layer 0 to layer 3, and the matrix product C is obtained in layer 3.

[0026] For problems where such a divide-and-conquer method is effective (e.g., matrix multiplication), given the axes of parallelization, once the axes of parallelization at each layer are determined, the problem size at each layer can be recursively determined, and therefore the data transfer time can be estimated. Once the relationship between the selection of parallelization axes and the data transfer time is known, the optimal total parallelization axis determination unit 120 can automatically determine the axes of parallelization that minimize the data transfer time, and these total parallelization axes are considered optimal. [Parallelization axis determination process] Next, with reference to Figures 10 and 11, a parallelization axis determination process that implements the partitioning method according to one embodiment of the present disclosure will be described. This parallelization axis determination process can be performed by the above-described all-parallelization axis determination device 100, or more specifically, by the processor of the all-parallelization axis determination device 100. Figure 10 is a flowchart showing the parallelization axis determination process according to one embodiment of the present disclosure.

[0027] As shown in Figure 10, in step S101, the full parallelization axis determination device 100 calculates the data transfer time for each combination of parallelization axes in each layer of the hierarchical memory computer 50 based on the size of the problem to be computed and the communication bandwidth between layers. Specifically, the full parallelization axis determination device 100 calculates the data transfer time for each parallelization axis in the data transfer method determined by the parallelization axis for each memory layer of the hierarchical memory computer 50. For example, the full parallelization axis determination device 100 calculates the data transfer time in the downstream direction from the top layer (LN layer) to the bottom layer (L0 layer) and the data transfer time in the upstream direction from the bottom layer (L0 layer) to the top layer (LN layer) for each combination of parallelization axes, and may determine the data transfer time for each combination of parallelization axes as the sum of the downstream data transfer time and the upstream data transfer time.

[0028] Furthermore, if a particular data transfer method cannot be used due to the limitations of the hierarchical memory computer 50, combinations including parallelization axes that require such an unusable data transfer method may be excluded.

[0029] In step S102, the overall parallelization axis determination device 100 determines the combination of parallelization axes that has the minimum data transfer time among the data transfer times calculated for each combination of parallelization axes as the optimal overall parallelization axis. That is, from all possible combinations of parallelization axes calculated in step S101, the overall parallelization axis determination device 100 identifies the combination of parallelization axes that achieves the minimum data transfer time, and determines the identified combination of parallelization axes as the optimal overall parallelization axis.

[0030] Figure 11 is pseudocode showing a parallelization axis determination procedure according to one embodiment of the present disclosure. This parallelization axis determination procedure is executed by a total parallelization axis determination device 100 on a hierarchical memory computer 50 consisting of four memory layers. In this embodiment, for a problem involving dense matrix multiplication, each layer except L3 independently selects whether to use the column direction or the row direction for parallelization, and the combination of parallelization axes that achieves the minimum data transfer time is determined as the optimal total parallelization axis. The number of selection patterns for the total parallelization axis is (number of candidate parallelization axes)^(number of layers-1), which in this example is 2^(4-1)=8. It is assumed that this number is similarly sufficiently small in real-world examples, and the determination of the optimal total parallelization axis can be performed by exhaustive search.

[0031] First, in layers 0 to 3, as in the embodiment shown in Figure 9, layer 3 is the top layer and layer 0 corresponds to the lowest layer in the PE.

[0032] Next, L[n][m],n,m∈{0,1,2,3} is a string intended to represent the connection from layer n to layer m, and modifies directional operations between layers, such as data transfer. As an exception, the parallelization axes set between each layer are not directional, but this string is used for convenience.

[0033] The variable `transfer_time` represents the minimum data transfer time among the sum of the maximum downstream data transfer time between each layer from Layer 3 to Layer 0 and the maximum upstream data transfer time between each layer from Layer 0 to Layer 3. Initially, the variable `transfer_time` is set to infinity.

[0034] The variable `axes_set` represents the combination of parallelization axes selected between layers when the variable `transfer_time` is updated. Initially, the variable `axes_set` is set to (nil,nil,nil).

[0035] The constant Axes represents a candidate for the parallelization axis, and in this embodiment, it is set to Axes = {Column (column direction), Row (row direction)}.

[0036] The variable aL[i][i-1]∈Axes,i∈{1,2,3} represents the parallelization axis when dividing the problem from layer i to layer i-1.

[0037] The variables N0, N1, N2, and N3 represent the problem sizes in layers 0, 1, 2, and 3, respectively. The variable N[i] can be represented, for example, as a tuple with one or more elements. In the example of matrix multiplication, it can be represented as a tuple with two elements. Furthermore, a hierarchy is defined between problem sizes. For example, in matrix multiplication, if N[i]=(m[i],n[i],k[i]),i∈{0,1,2,3}, then N2>N1 is defined as m2>m1, n2>n1, and k2>k1.

[0038] The function Partition(N, axis, layers) returns the problem size when N is divided according to the parallelization axis specified by axis. `layers` is an argument that specifies which layers are being considered for the division and sets the necessary parameters for the division. In the pseudocode, the problem size of layer i-1 is used to determine the problem size of layer i and the parallelization axis aL[i][i-1], and N0 is calculated inductively from N3. Incidentally, if N[i] and N[i-1] have a one-to-one correspondence with the parallelization axis and various parameters fixed, it is theoretically possible to calculate the problem size of a higher layer from the problem size of a lower layer. That is, once all parallelization axes and one of the problem sizes N0, N1, N2, or N3 are determined, the problem sizes of the remaining layers are also uniquely determined. In Figure 11, the search for the parallelization axis is performed with the problem size N3 fixed (i.e., the calculation procedure fixed). Note that the problem can be calculated when N3 ≥ N' for the problem size N' that we want to solve. If N3 ≥ N' is not true, one possible approach is to divide the problem size N' that you want to solve and perform the calculations in multiple steps.

[0039] Here, for Layer 0, we can consider a lower bound on the problem size that can be efficiently computed by the arithmetic units. For example, the number of elements that can be processed at once by the dot product unit, and the number of parallel executions required to hide the latency of the arithmetic units and L0 memory, affect this lower bound. On the other hand, we can also consider an upper bound on the problem size that can be computed from the perspective of memory capacity. For example, in the calculation of matrix multiplication, if we store the matrices A0 and B0 to be multiplied by a single PE, and the resulting matrix C0, in L0 memory, then the maximum value that N0 can take can be calculated from the size of the L0 memory. The upper bound on the problem size due to memory capacity can be considered similarly for layers other than Layer 0.

[0040] The function ToSizeD(N) calculates the amount of data required by a problem represented by N in the downstream direction, transferring data from Layer 3 to Layer 0. On the other hand, the function ToSizeU(N) calculates the amount of data required by a problem represented by N in the upstream direction, transferring data from Layer 0 to Layer 3. In the example of matrix multiplication, the data transferred downstream are matrices A and B, so ToSizeD(N1) = m1 × k1 + k1 × n1, and the data transferred upstream is matrix C, so ToSizeU(N1) = m1 × n1.

[0041] The function GetBW(axis, layers) calculates the communication bandwidth when axis is selected for inter-layer data transfer specified by layers.

[0042] The variable tL30 represents the data transfer time from Layer 3 to Layer 0 and is calculated from the problem size and communication bandwidth. On the other hand, the variable tL03 represents the data transfer time from Layer 0 to Layer 3. In hierarchical memory computers, inter-layer communication can often overlap, so the inter-layer communication that takes the most time becomes the bottleneck. Inter-layer communication overlap means, for example, that communication from Layer 2 to Layer 1 starts before communication from Layer 3 to Layer 2 is finished. Using this, the pseudocode uses the maximum value max for all inter-layer communication in this calculation. In other words, the data transfer time for each combination of parallelization axes is calculated based on the calculated maximum data transfer time required for inter-layer communication. In hierarchical memory computers where inter-layer communication overlap is not possible, the sum of all inter-layer communication times may be used.

[0043] As can be understood from the pseudocode shown, for all combinations of parallelization axes of layers, first, the problem sizes N2, N1, and N0 of each layer are sequentially determined from a given problem size N3. Then, the downstream data transfer time tL30 and the upstream data transfer time tL03 are determined based on the problem size and communication bandwidth of each layer. If the sum of the data transfer times is less than the current transfer_time, the sum of the data transfer times is set as the new transfer_time, and the combination of parallelization axes is set in axes_set. This process is repeated for all combinations of parallelization axes, and the combination set in the final axes_set becomes the optimal total parallelization axis.

[0044] For example, in the embodiment shown in Figure 9, aL32, aL21, and aL10 are Row, Row, and Column, respectively. The problem in Layer 3 is the product of a 16x4 matrix and a 4x4 matrix, so the problem size N3 is expressed as (16,4,4). Next, in N2=Partition(N3,aL32,"L32"), the problem in Layer 3 is divided by Row (row-direction partitioning), so the problem in Layer 2 is the product of a 4x4 matrix and a 4x4 matrix, and N2 is expressed as (4,4,4). Similarly, the problem sizes N1 and N0 are expressed as (1,4,4) and (1,1,4), respectively. Note that only the division to N0 is divided by Column. The communication bandwidth between each layer is determined from the chip specifications. In this way, the downstream data transfer time tL30 and the upstream data transfer time tL03 are determined based on the problem size and communication bandwidth in each layer. [Simulation Results] Next, the simulation results according to one embodiment of the present disclosure will be explained with reference to Figure 12. Figure 12 is a diagram showing the simulation results according to one embodiment of the present disclosure.

[0045] In the graph in Figure 12, d(0|1)(0|1)(0|1)m.dat represents the following: the leftmost (0|1) indicates that the parallelization axis of Layer 2 is in the column or row direction; the middle (0|1) indicates that the parallelization axis of Layer 1 is in the column or row direction; and the rightmost (0|1) indicates that the parallelization axis of Layer 0 is in the column or row direction.

[0046] When the matrix size M is large, it is most efficient to use column partitioning (column-direction partitioning) for all layers, while when the matrix size M is small, it is efficient to use row partitioning (row-direction partitioning) only for data transfer between layer 3 and layer 2. Furthermore, some combinations of transfer methods result in significantly low estimated maximum efficiency, which is due to combinations of data transfer methods that do not provide sufficiently sized submatrices for the multiple PEs (Performance Entities) present throughout the computer.

[0047] The above-described embodiment focused on double-precision dense matrix multiplication (DGEMM) A×B=C, but this disclosure is not limited to this and can be applied to other methods such as C=αA×B, C+=A×B, single-precision (SGEMM), half-precision (HGEMM), and other level 3 BLAS operations. Furthermore, this disclosure is not limited to the dense matrix calculations described above, but can also be applied to fully connected layers and convolutional layers in neural networks. In addition, although the procedure for writing the calculation results back to the top layer was described, this disclosure is not limited to this and can also be applied when the calculation results placed in any lower layer are reused as input data for another calculation. For example, in the convolutional layer of the neural network described above, the batch direction, spatial direction, and channel direction can be considered as parallelization axes. [Hardware configuration of the fully parallelized axis determination system] In the fully parallelized axis determination device 100 of this embodiment, each function may be a circuit composed of an analog circuit, a digital circuit, or a mixed analog-digital circuit. Furthermore, a control circuit for controlling each function may be provided. Each circuit may be implemented using an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like.

[0048] In all of the above descriptions, at least a part of the fully parallelized axis determination device 100 may be composed of hardware, or it may be composed of software, and the CPU (Central Processing Unit) or the like may perform the information processing by the software. If it is composed of software, the program that realizes the functions of the fully parallelized axis determination device 100 and at least a part of it may be stored in a non-temporary computer-readable storage medium and loaded into a computer for execution. The storage medium is not limited to removable ones such as magnetic disks (e.g., flexible disks) or optical disks (e.g., CD-ROMs or DVD-ROMs), but may also be a fixed storage medium such as a hard disk drive or an SSD (Solid State Drive) that uses memory. In other words, the information processing by software may be concretely implemented using hardware resources. Furthermore, the processing by software may be implemented in a circuit such as an FPGA and executed by hardware. Job execution may be performed using an accelerator such as a GPU (Graphics Processing Unit).

[0049] For example, a computer can be made into the device of the above embodiment by reading dedicated software stored on a computer-readable storage medium. The type of storage medium is not particularly limited. Alternatively, a computer can be made into the device of the above embodiment by installing dedicated software downloaded via a communication network. In this way, information processing by software is concretely implemented using hardware resources.

[0050] Figure 13 is a block diagram showing an example of the hardware configuration of a fully parallelized axis determination device 100 in one embodiment of the present disclosure. The fully parallelized axis determination device 100 can be realized as a computer device comprising a processor 101, a main memory 102, an auxiliary memory 103, a network interface 104, and a device interface 105, which are connected via a bus 106.

[0051] Although the fully parallelized axis determination device 100 in Figure 13 has one of each component, it may have multiple identical components. Also, although one fully parallelized axis determination device 100 is shown, the software may be installed on multiple computer devices, and each of these multiple fully parallelized axis determination devices 100 may execute different parts of the software. In this case, each of the multiple fully parallelized axis determination devices 100 may communicate via a network interface 104 or the like.

[0052] The processor 101 is an electronic circuit (processing circuit, processing circuitry) that includes the control unit and arithmetic unit of the fully parallelized axis determination device 100. The processor 101 performs arithmetic processing based on data and programs input from each of the internal components of the fully parallelized axis determination device 100, and outputs the calculation results and control signals to each of the components. Specifically, the processor 101 controls each component that makes up the fully parallelized axis determination device 100 by executing the OS (Operating System) and applications of the fully parallelized axis determination device 100. The processor 101 is not particularly limited as long as it can perform the above processing. The fully parallelized axis determination device 100 and each of its components are realized by the processor 101. Here, a processing circuit may refer to one or more electrical circuits arranged on one chip, or one or more electrical circuits arranged on two or more chips or devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless.

[0053] The main memory 102 is a storage device that stores instructions executed by the processor 101 and various data, and the information stored in the main memory 102 is directly read by the processor 101. The auxiliary storage device 103 is a storage device other than the main memory 102. These storage devices refer to any electronic component capable of storing electronic information, and may be either memory or storage. Memory can be either volatile memory or non-volatile memory. Memory for storing various data within the parallelization axis determination device 100, for example, may be implemented by the main memory 102 or the auxiliary storage device 103. For example, at least a portion of the memory may be implemented in the main memory 102 or the auxiliary storage device 103. As another example, if an accelerator is provided, at least a portion of the aforementioned memory may be implemented in the memory provided in the accelerator.

[0054] The network interface 104 is an interface for connecting to the communication network 200 wirelessly or via a wired connection. The network interface 104 may conform to existing communication standards. Information may also be exchanged with an external device 109A connected via the communication network 108 through the network interface 104.

[0055] External devices 109A include, for example, cameras, motion capture devices, output destination devices, external sensors, and input source devices. External devices 109A may also be devices that perform some of the functions of the components of the fully parallelized axis determination device 100. Furthermore, the fully parallelized axis determination device 100 may receive some of its processing results via a communication network 108, similar to a cloud service.

[0056] The device interface 105 is an interface such as USB (Universal Serial Bus) that connects directly to the external device 109B. The external device 109B may be an external storage medium or a storage device. Memory may also be implemented by the external device 109B.

[0057] External device 109B may also be an output device. The output device may be, for example, a display device for displaying images, or a device that outputs sound, etc. Examples include, but are not limited to, LCDs (Liquid Crystal Displays), CRTs (Cathode Ray Tubes), PDPs (Plasma Display Panels), OLED (Electroluminescence) displays, and speakers.

[0058] External device 109B may also be an input device. The input device includes devices such as a keyboard, mouse, touch panel, and microphone, and provides the information input by these devices to the fully parallelized axis determination device 100. Signals from the input device are output to the processor 101.

[0059] For example, the data transfer time calculation unit 110 and the optimal full-parallel axis determination unit 120 of the full-parallel axis determination device 100 in this embodiment may be implemented by a processor 101. Furthermore, the memory of the parallel axis determination device 100 may be implemented by a main memory 102 or an auxiliary memory 103. Also, the full-parallel axis determination device 100 may be equipped with one or more memory devices.

[0060] In this specification, the expression "at least one of a, b, and c" or "at least one of a, b, or c" includes any combination of a, b, c, ab, ac, bc, and abc. It also covers combinations with multiple instances of any of the elements, such as aa, abb, and aabbcc. It further covers adding other elements other than a, b, and / or c, such as having abcd.

[0061] Although embodiments of this disclosure have been described in detail above, this disclosure is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of this disclosure as described in the claims. [Explanation of Symbols]

[0062] 10 Computing Systems 50-tier memory computer 60 External computer 100 fully parallel axis determination device 110 Data transfer time calculation unit 120 Optimal Full Parallelization Axis Determination Unit 101 Processors 102 Main storage 103 Auxiliary storage device 104 Network Interfaces 105 Device Interfaces 108 Communication Networks 109 External device

Claims

1. A processing method for a computing system including at least one processor and a hierarchical memory type computing device including three or more hierarchical memory layers, comprising: the at least one processor calculates data on a first data transfer time for each of a plurality of parallelization axes, each of the plurality of parallelization axes determining a method of dividing a calculation target in each of the three or more memory layers, the data on the first data transfer time being calculated based on data on a plurality of second data transfer times calculated between two different layers of the three or more memory layers; the hierarchical memory computer executes the computation of the computation target based on all parallelization axes selected from the plurality of all parallelization axes, the all parallelization axes being selected based on data regarding the first data transfer time calculated for each of the plurality of all parallelization axes; Processing methods.

2. 2. The processing method according to claim 1, wherein the calculation target is a matrix, and each of the plurality of parallelization axes determines a method of dividing the matrix in a row direction or a column direction in each of the three or more memory layers.

3. The processing method according to claim 1 or claim 2, wherein the three or more memory layers include at least a first layer, a second layer, and a third layer, and the data relating to the multiple second data transfer times includes at least data relating to data transfer times between the first layer and the second layer and data relating to data transfer times between the second layer and the third layer.

4. 4. The processing method according to claim 1, wherein the data relating to the first data transfer time is a maximum value of the data relating to the plurality of second data transfer times.

5. 5. The processing method according to claim 1, wherein the hierarchical memory type computer is a computer capable of initiating communication between another memory layer of the three or more memory layers before communication between a memory layer of the three or more memory layers is completed.

6. 4. The processing method according to claim 1, wherein the data relating to the first data transfer time is a sum of the data relating to the plurality of second data transfer times.

7. The processing method according to any one of claims 1 to 3 or claim 6, wherein the hierarchical memory type computer is a computer that is unable to start communication between another memory layer of the three or more memory layers until communication between the memory layers of the three or more memory layers is completed.

8. 8. The processing method according to claim 1, wherein the at least one processor determines a data transfer method between the two layers, and calculates data regarding the second data transfer time based on the determined data transfer method.

9. The processing method according to claim 1 , wherein the calculation target is a layer included in a neural network.

10. The processing method according to claim 1 , wherein the parallelization axes included in the total parallelization axes include at least one of a batch direction, a spatial direction, and a channel direction.

11. A computing system including at least one processor and a hierarchical memory type computing device including three or more hierarchical memory layers, the at least one processor calculates data on a first data transfer time for each of a plurality of parallelization axes, each of the plurality of parallelization axes determining a method of dividing a calculation target in each of the three or more memory layers, the data on the first data transfer time being calculated based on data on a plurality of second data transfer times calculated between two different layers of the three or more memory layers; the hierarchical memory computer executes the computation of the computation target based on all parallelization axes selected from the plurality of all parallelization axes, the all parallelization axes being selected based on data regarding the first data transfer time calculated for each of the plurality of all parallelization axes; Calculation system.

12. 12. The computing system of claim 11, wherein the computation target is a matrix, and each of the plurality of parallelization axes determines a method of dividing the matrix in a row direction or a column direction in each of the three or more memory layers.

13. 13. The computing system of claim 11 or 12, wherein the three or more memory layers include at least a first layer, a second layer, and a third layer, and the data relating to the second data transfer times includes at least data relating to data transfer times between the first layer and the second layer and data relating to data transfer times between the second layer and the third layer.

14. 14. The computing system according to claim 11, wherein the data relating to the first data transfer time is a maximum value of the data relating to the plurality of second data transfer times.

15. 15. The computing system according to claim 11, wherein the hierarchical memory type computer is a computer capable of initiating communication between another memory layer of the three or more memory layers before communication between a memory layer of the three or more memory layers is completed.

16. 14. The computing system according to claim 11, wherein the data relating to the first data transfer time is a sum of the data relating to the plurality of second data transfer times.

17. The computing system according to any one of claims 11 to 13 or claim 16, wherein the hierarchical memory type computer is a computer that is unable to start communication between another memory layer of the three or more memory layers until communication between the memory layers of the three or more memory layers is completed.

18. 18. The computing system according to claim 11, wherein the at least one processor determines a data transfer method between the two layers, and calculates data relating to the second data transfer time based on the determined data transfer method.

19. The computing system according to claim 11 , wherein the computation target is a layer included in a neural network.

20. 20. The computing system according to claim 11, wherein the parallelization axes included in the total parallelization axes include at least one of a batch direction, a spatial direction, and a channel direction.

Images