Matrix multiplication method, device and system for distributed heterogeneous computing nodes

Abstract

The present invention relates to a matrix multiplication method, device and system for distributed heterogeneous computing nodes, the method comprising: deploying a Spark system to a heterogeneous distributed computer cluster, wherein each computer node has a different CPU or GPU configuration; using Spark as a distributed computing execution engine to read matrix data from a distributed file system; quantifying the CPU or GPU computing performance on each working node in the cluster; dividing the matrix data into blocks according to the performance to obtain matrix blocks; performing lossless floating point compression on the matrix blocks and transmitting them to each working node; each working node assigning the matrix blocks to CPU or GPU tasks according to the computing performance ratio of the hardware, and in each task, multiplying the input matrix blocks to generate the intermediate blocks of the output matrix blocks; and aggregating the intermediate blocks of the output matrix blocks to generate the final output matrix. The present invention can improve the computing efficiency of large-scale distributed matrix multiplication and solve the technical problems existing in traditional solutions.

Description

Technical Field

[0001] The present invention relates to the field of distributed matrix computing technology, and in particular to a matrix multiplication method, device and system for distributed heterogeneous computing nodes. Background Art

[0002] Matrix computing is essentially widely used in various fields such as databases and machine learning. With the explosive growth of data scale, fast and scalable distributed matrix computing systems, especially distributed matrix multiplication, have become more important than ever. Since traditional parallel technology relies on distributed file systems, when the scale of data to be processed is too large, the bandwidth connecting the distributed file system network becomes a bottleneck restricting system performance.

[0003] The MapReduce framework (a programming model for parallel computing of large-scale data sets) launched by Google has now developed into a standard high-level programming model for large-scale distributed data processing. Hadoop (a framework for distributed data and computing) based on it provides a large-capacity, scalable, and highly reliable distributed storage system (Hadoop Distributed File System, abbreviated as: HDFS), which effectively utilizes the local characteristics of data and solves the bottleneck problem of network bandwidth. However, Hadoop is relatively inefficient for some iterative, interactive, and complex data processing tasks. Each task needs to write and read the disk, resulting in high latency. Spark, a distributed computing framework developed by the "Algorithms, Machines, and Humans" (AMP) laboratory, caches data in memory so that memory objects can be directly passed between MapReduce operations. By using memory cache, low-latency computing of distributed data and iterative computing of shared data can be effectively performed in memory without accessing the disk.

[0004] For large matrices, distributed matrix multiplication is usually performed in the following three steps: (1) repartition the input matrix between tasks; (2) perform local matrix multiplication within each task; (3) aggregate the intermediate results of the local matrix multiplication by merging them. For the matrix repartitioning and aggregation steps, the following three methods have been proposed or used: broadcast matrix multiplication (BMM), which broadcasts smaller input matrices to all tasks; cross-product-based matrix multiplication (CPMM), which performs multiple outer products between two input matrices and aggregates the results of the outer products; and replication-based matrix multiplication (RMM), which repartitions the input matrix into smaller units (called blocks). However, the above methods may lead to insufficient memory due to high memory usage of each task or degrade performance due to high communication costs. In addition, they do not take advantage of the hardware acceleration of multi-core accelerators such as graphics processing units (GPUs).

[0005] In view of this, how to overcome the defects of the existing technology and solve the problems of the existing technology is a difficult problem to be solved in this technical field. Summary of the invention

[0006] The embodiments of the present invention provide a matrix multiplication method, device and system for distributed heterogeneous computing nodes, which can solve the technical problems in traditional solutions that may lead to insufficient memory due to high memory usage of each task, performance degradation due to high communication costs, and no hardware acceleration using multi-core accelerators.

[0007] The purpose of the embodiment of the present invention is achieved through the following technical solutions:

[0008] In order to solve the above technical problems, in a first aspect, an embodiment of the present invention provides a matrix multiplication method for distributed heterogeneous computing nodes, including:

[0009] Deploy the Spark system to a heterogeneous distributed computer cluster, where each computer node has a different CPU or GPU configuration; use Spark as a distributed computing execution engine to read matrix data from a distributed file system;

[0010] Quantify the CPU or GPU computing performance on each worker node in the cluster; divide the matrix data into blocks according to the performance to obtain matrix blocks; perform lossless floating point compression on the matrix blocks and transmit them to each worker node;

[0011] Each working node allocates matrix blocks to CPU or GPU tasks according to the computing performance ratio of the hardware. In each task, the input matrix blocks are multiplied to generate an intermediate block of the output matrix block;

[0012] The intermediate blocks of the output matrix blocks are aggregated to generate the final output matrix.

[0013] In some embodiments, the quantifying the CPU or GPU computing performance on each working node in the cluster specifically includes:

[0014] According to the input matrix A, let α and 1 α is the workload ratio allocated to GPU and CPU respectively, where 0≤α≤1;

[0015] Using Tg(α˙A) and Tc((1 α)˙A) represents the GPU and CPU threads updating α˙A and (1 α)˙The time taken by the elements in A, and recorded as Tg(α) and Tc(1 α);

[0016] Use fg(α) and fc(1-α) to represent the cost functions of GPU and CPU threads. and The estimate of , denoted as fg(α) ,fc(α) ;

[0017] Use the cluster manager to run the matrix multiplication test program on N worker nodes in the cluster and obtain the CPU cost model of the Ni-th worker node. and GPU cost model .

[0018] In some embodiments, dividing the matrix data into blocks according to performance specifically includes:

[0019] The total time each worker node takes to run the matrix multiplication test program for:

[0020] ;

[0021] in, Represents the number of CPU threads, Represents the number of GPUs; i represents the i-th working node;

[0022] According to the calculated cost function, when the load between resources is balanced, the total running time is minimized, and the following optimization problem is solved to obtain , and obtain :

[0023] ;

[0024] Set weights based on different computing performance of working nodes :

[0025] ;

[0026] According to the weight ratio of each node, the rectangular block algorithm is called to divide the matrices A and B into blocks.

[0027] In some embodiments, performing lossless floating point compression on the matrix block and transmitting the compression to each working node includes:

[0028] Use compressed sparse row representation to represent matrix blocks;

[0029] Perform a lossless floating point compression algorithm on the non-zero elements of the matrix in the array V produced by the compressed sparse row representation;

[0030] Compressed sparse row representation of compressed matrix blocks;

[0031] The compressed matrix blocks are transmitted to the corresponding working nodes, and then the decompression algorithm is executed.

[0032] In some embodiments, the intermediate block of multiplying the input matrix block to generate the output matrix block specifically includes:

[0033] Based on GPU workload ratio , the matrix cuboid consisting of the matrix blocks of the input matrix Divide into two sub-cubic blocks along the k axis and , used for CPU and GPU to perform matrix multiplication, respectively, where:

[0034] Use the CPU to perform matrix multiplication, running directly in memory;

[0035] Use GPU to perform matrix multiplication. Specifically: Copy from main memory to GPU memory; perform matrix multiplication in GPU memory; copy the result from GPU memory back to main memory;

[0036] Will Use the CPU to perform matrix multiplication and The calculation results of matrix multiplication performed using the GPU are aggregated to generate an intermediate block of output matrix blocks.

[0037] In some embodiments, it also includes: using a GPU monitor to periodically check the GPU utilization, and the adaptive scheduler reads the result; when the GPU utilization is lower than a preset value, increasing the number of GPU containers through the adaptive scheduler to fully utilize the GPU.

[0038] In some embodiments, aggregating the intermediate blocks of the output matrix blocks to generate the final output matrix includes:

[0039] An aggregation operation is performed on the intermediate blocks of output matrix blocks from the same group of worker nodes to obtain the final output block of the final output matrix.

[0040] In a second aspect, an embodiment of the present invention provides a matrix multiplication system for distributed heterogeneous computing nodes, which is used to implement the matrix multiplication method for distributed heterogeneous computing nodes as described in the first aspect, including a matrix data reading module, a matrix block acquisition module, an intermediate block acquisition module, and a final output matrix acquisition module, wherein:

[0041] The matrix data reading module is used to read matrix data from a distributed file system;

[0042] The matrix block acquisition module is used to quantify the CPU or GPU computing performance on each working node in the cluster; divide the matrix data into blocks according to the performance to obtain matrix blocks; perform lossless floating point compression on the matrix blocks and transmit them to each working node;

[0043] The intermediate block acquisition module is used to allocate matrix blocks to CPU or GPU tasks according to the computing performance ratio of the hardware at each working node, and in each task, multiply the input matrix blocks to generate the intermediate blocks of the output matrix blocks;

[0044] The final output matrix acquisition module is used to aggregate the intermediate blocks of the output matrix blocks to generate the final output matrix.

[0045] In a third aspect, an embodiment of the present invention provides a matrix multiplication device for distributed heterogeneous computing nodes, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the matrix multiplication method for distributed heterogeneous computing nodes as described in the first aspect.

[0046] In a fourth aspect, an embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the matrix multiplication method for distributed heterogeneous computing nodes as described in the first aspect.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows: Different from the prior art, the embodiments of the present invention provide a matrix multiplication method, device and system for distributed heterogeneous computing nodes, which can reasonably allocate matrix computing tasks and make full use of different hardware resources, reduce data communication costs through lossless floating-point matrix compression, and improve the computational efficiency of large-scale distributed matrix multiplication, solving the technical problems of insufficient memory due to high memory usage of each task and performance degradation due to high communication costs in traditional solutions. In addition, the method can not only speed up task execution by combining CPU and GPU, but also effectively utilize the computing resources available in heterogeneous systems, solving the technical problem of hardware acceleration without multi-core accelerators in traditional solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise specified, the figures in the drawings do not constitute proportional limitations.

[0049] Figure 1 It is a flow chart of a matrix multiplication method for distributed heterogeneous computing nodes provided by an embodiment of the present invention;

[0050] Figure 2 is an example diagram of a three-dimensional model of matrix multiplication provided by an embodiment of the present invention;

[0051] Figure 3 is an example diagram of a rectangular parallelepiped partition provided by an embodiment of the present invention;

[0052] Figure 4 is an example diagram of matrix three-dimensional model division provided by an embodiment of the present invention;

[0053] Figure 5 is a distributed matrix multiplication step diagram provided by an embodiment of the present invention;

[0054] Figure 6 It is a block diagram of a matrix multiplication system for distributed heterogeneous computing nodes provided by an embodiment of the present invention;

[0055] Figure 7 It is a schematic diagram of the structure of a matrix multiplication device for distributed heterogeneous computing nodes provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, and all are within the protection scope of the present invention. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flow chart.

[0059] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0060] Based on the problems existing in the existing technology, the input matrix can be divided into multiple blocks called cuboids, which have the optimal size in terms of communication cost and memory usage for each task, and also support GPU acceleration. However, most of the existing methods are based on homogeneous nodes, and do not take into account that each node may have a central processing unit (CPU) or GPU with different configurations in actual situations. The combination of CPU and GPU can not only speed up task execution, but also effectively utilize the computing resources available in heterogeneous systems. At the same time, in order to further reduce data communication costs and memory usage, floating point lossless compression can be implemented on data during transmission. For large-scale distributed matrix multiplication, it is necessary to improve the time efficiency and space efficiency of matrix calculations. Therefore, studying the fast multiplication method of high-dimensional matrices for distributed heterogeneous computing nodes is of great significance to the related research on distributed matrix computing.

[0061] The embodiment of the present invention provides a matrix multiplication method for distributed heterogeneous computing nodes, which can divide the matrix into blocks, compress and transmit it, effectively utilize the computing resources in the heterogeneous system, and perform parallel computing based on Spark, thereby improving the computational efficiency of matrix multiplication.

[0062] The symbols involved in the embodiment of the present invention are summarized in Table 1 below.

[0063] Table 1. Summary of symbols

[0064]

[0065] In order to improve the computational efficiency of the matrix-matrix multiplication method, the embodiment of the present invention proposes some definitions in the matrix multiplication method for distributed heterogeneous computing nodes:

[0066] Definition 1 (Cost Model): The cost model is a function that reveals the relationship between the input data size and the corresponding execution time of a specific worker thread.

[0067] Definition 2 (GPU cost model): Given a matrix multiplication task, the total GPU processing time is spent on two parts: (1) data transfer between CPU and GPU through the PCI-e bus; (2) GPU execution kernel.

[0068] Data transfer has two directions: from CPU to GPU and from GPU to CPU. The data transfer time is recorded as and Taking the data transmission from CPU to GPU as an example, we use the following formula for fitting modeling, where |A| represents the size of the matrix data transferred from CPU to GPU, and τ represents the threshold.

[0069] ;

[0070] Where a and b are coefficients. The cost model of GPU execution kernel is as follows: represents the time spent by the GPU executing kernel A, function Indicates the processing speed at which the GPU executes kernels.

[0071] ;

[0072] The overall cost model of the GPU depends on the maximum value between the data transfer time between the CPU and GPU and the execution time of the GPU kernel, as shown below:

[0073] .

[0074] Definition 3 (CPU cost model): The CPU cost model is represented by a linear function as follows:

[0075] .

[0076] Definition 4 (Blocked matrix multiplication): In the matrix multiplication of C = A × B, each block in Ci,j (0≤i<I, 0≤j<J) can be calculated as follows, where the input matrices A and B have I×K and K×J blocks respectively, and the output matrix C has I×J blocks.

[0077] .

[0078] Definition 5 (Three-dimensional model of matrix multiplication): Matrix multiplication can be represented as a three-dimensional model with i-axis, j-axis and k-axis, where 0≤i<I, 0≤j<J, 0≤k<K. The ik plane of the model represents matrix A, the kj plane represents matrix B, and the ij plane represents matrix C.

[0079] Definition 6 (Volume element): The volume of the model is regarded as a volume element of I×J×K. Each volume element represents a computational unit of matrix multiplication, that is, an intermediate block Ci,j is calculated by multiplying two blocks Ai,k˙Bk,j, denoted as Cki,j, and the index of the volume element is recorded as vi,j,k.

[0080] Definition 7 (decimal digits, significant digits, starting significant digit): given v in decimal format , decimal places If for all (Right now is the first number not equal to 0), then it is called It is the starting valid bit. , called the number of effective digits.

[0081] Definition 8 (IEEE754 floating point format): According to the IEEE754 standard, double-precision values It uses 64 binary bits for storage, 1 bit for the sign bit s and 11 bits for the exponent , 52 bits for the mantissa . Double precision value Can be divided into normal numbers and special numbers. If v is a normal number, its value satisfies:

[0082] ;

[0083] Among them, e is The decimal value of , the binary format of v , then:

[0084] .

[0085] Definition 9 (Tail-prefix number): Given a number with A double precision value v with The double precision value of called The number of the mantissa prefix of Make , denoted as .

[0086] Definition 10 (Grammar): A grammar is a straight-line program (SLP) consisting of a set of The rules consist of is a non-terminal symbol, It can be either a terminal or a non-terminal. Given a non-terminal , its extended use Representation, defined as the (unique) sequence obtained by repeatedly applying the replacement rules of the SLP grammar until only terminal symbols remain.

[0087] Specifically, the embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0088] Example 1

[0089] Embodiment 1 of the present invention provides a matrix multiplication method for distributed heterogeneous computing nodes, supports distributed matrix multiplication on heterogeneous computer clusters, divides matrix data into blocks, and then transmits them to each working node through lossless compression, and calls the CPU or GPU to perform matrix multiplication. Figure 1 The matrix multiplication method for distributed heterogeneous computing nodes includes but is not limited to the following steps:

[0090] Step S101: deploy the Spark system to a heterogeneous distributed computer cluster, where each computer node has a different CPU or GPU configuration; use Spark as a distributed computing execution engine to read matrix data from a distributed file system. .

[0091] Step S102: quantify the CPU or GPU computing performance on each working node in the cluster; divide the matrix data into blocks according to the performance to obtain matrix blocks; perform lossless floating point compression on the matrix blocks and transmit them to each working node. In this step, the cost model can be used to estimate the CPU or GPU computing performance on each working node, and then divide the matrix data into blocks, and then transmit them to each node after lossless floating point compression.

[0092] Step S103: Each working node allocates the matrix blocks to the CPU or GPU tasks according to the computing performance ratio of the hardware. In each task, the input matrix blocks are multiplied to generate the intermediate blocks of the output matrix blocks. During the processing, the GPU monitor can be used to regularly check the GPU utilization, and the result is read by the adaptive scheduler; when the GPU utilization is low (lower than the preset value, which is set based on experience), the number of GPU containers is increased through the adaptive scheduler to make full use of the GPU.

[0093] Step S104: Aggregate the intermediate blocks of the output matrix blocks to generate the final output matrix. In this step, the intermediate blocks of the output matrix blocks from the same group of working nodes may be aggregated to obtain the final output block of the final output matrix.

[0094] Through the above steps, the technical problems in the traditional solution that may lead to insufficient memory due to high memory usage of each task, performance degradation due to high communication cost, and hardware acceleration without utilizing multi-core accelerators can be solved.

[0095] For step S102, in some embodiments, the quantification of the CPU or GPU computing performance on each working node in the cluster specifically includes: according to the input matrix A, let α and 1 α is the workload ratio assigned to GPU and CPU, where 0≤α≤1; Tg(α˙A) and Tc((1 α)˙A) represents the GPU and CPU threads updating α˙A and (1 α)˙The time taken by the elements in A, and recorded as Tg(α) and Tc(1 α); use fg(α) and fc(1-α) to represent the cost functions of GPU and CPU threads and The estimate of , denoted as fg(α) ,fc(1-α) ; Use the cluster manager to run the matrix multiplication test program on N worker nodes in the cluster to obtain the CPU cost model of the Ni-th worker node and GPU cost model .

[0096] In some embodiments, the partitioning of matrix data according to performance specifically includes: the total time for each working node to run the matrix multiplication test program for:

[0097] ;

[0098] in, Represents the number of CPU threads, represents the number of GPUs; according to the calculated cost function, when the load between resources is balanced, the total running time is minimized, and the following optimization problem is solved to obtain , and obtain :

[0099] ;

[0100] Set weights based on different computing performance of working nodes :

[0101] ;

[0102] According to the weight ratio of each node, the rectangular block algorithm is called to divide the matrices A and B into blocks.

[0103] In some embodiments, the performing lossless floating-point compression on the matrix block and transmitting it to each working node includes: using a compressed sparse row representation to represent the matrix block; performing an Elf lossless floating-point compression algorithm on the matrix non-zero elements in the array V generated by the compressed sparse row representation; using a recursive paring algorithm to compress the compressed sparse row representation of the matrix block; transmitting each compressed matrix block to the corresponding working node, and then performing a decompression algorithm.

[0104] For step S103, in some embodiments, the multiplication of the input matrix block to generate the intermediate block of the output matrix block specifically includes: according to the GPU workload ratio , the matrix cuboid consisting of the matrix blocks of the input matrix Divide into two sub-cubic blocks along the k axis and , used for CPU and GPU to perform matrix multiplication, respectively, where: Use the CPU to perform matrix multiplication, running directly in memory; Use GPU to perform matrix multiplication. Specifically: Copy from memory to GPU memory; perform matrix multiplication in GPU memory; copy the calculation result from GPU memory back to memory; Use the CPU to perform matrix multiplication and The calculation results of matrix multiplication performed using the GPU are aggregated to generate an intermediate block of output matrix blocks.

[0105] In summary, an embodiment of the present invention provides a matrix multiplication method for distributed heterogeneous computing nodes, which can reasonably allocate matrix computing tasks and make full use of different hardware resources, reduce data communication costs through lossless floating-point matrix compression, and improve the computational efficiency of large-scale distributed matrix multiplication, solving the technical problems of insufficient memory due to high memory usage of each task and performance degradation due to high communication costs in traditional solutions. In addition, the method can not only speed up task execution by combining CPU and GPU, but also effectively utilize the computing resources available in heterogeneous systems, solving the technical problem of hardware acceleration without multi-core accelerators in traditional solutions.

[0106] The above description is an overall description of the matrix multiplication method for distributed heterogeneous computing nodes described in this embodiment. The following is an expanded description through more specific embodiments.

[0107] Example 2

[0108] Based on the matrix multiplication method for distributed heterogeneous computing nodes provided in Example 1, this Example 2 further explains its use in actual application scenarios.

[0109] 1. Deploy Spark to a local cluster. The cluster consists of 1 Master node and N Worker nodes. Each node has a different CPU or GPU configuration, forming a heterogeneous distributed computer cluster.

[0110] 2. Use the input operations provided by Spark to load the data of matrices A and B from distributed storage and convert these data into key-value RDDs (Resilient Distributed Datasets), where the keys are the row and column indices of the matrices and the values ​​are the elements in the matrices.

[0111] 3. According to the input matrix A (only A is taken as an example, the same applies to B), let α and 1 α is the workload ratio assigned to GPU and CPU, where 0≤α≤1. Using Tg(α˙A) and Tc((1 α)˙A) represents the GPU and CPU threads updating α˙A and (1 α)˙The time taken by the elements in A, and recorded as Tg(α) and Tc(1 α). Use fg(α) (or fc(1-α) ) to represent the cost function of the GPU (or CPU) thread (or ), denoted as fg(α) ,fc(1-α) .

[0112] Use the YARN cluster manager to run the matrix multiplication test program on N Worker nodes in the cluster and obtain the CPU cost model of the Ni-th Worker node. (Algorithm 1) and GPU cost model (Algorithm 2).

[0113] Algorithm 1: Get node CPU computing performance

[0114] (1) Input: RDD of matrix A (A is taken as an example, the same applies to B).

[0115] (2) Shuffle the matrix A to avoid uneven data distribution, and then divide A into N disjoint parts S1, S2, S3…SN.

[0116] (3) The CPU execution kernel performs matrix multiplication tests on the above data sets. The array Pc records the execution time and corresponding data size on S1, S1+S2, S1+S2+S3…S1+S2+S3+…+SN. To eliminate noise interference, the execution time is the average of 10 tests.

[0117] (4) Use the array Pc to fit the CPU cost model (Definition 3) to generate the CPU cost function .

[0118] Algorithm 2: Obtaining node GPU computing performance

[0119] (1) Input: RDD of matrix A (only A is taken as an example, the same applies to B).

[0120] (2) Shuffle the matrix A to avoid uneven data distribution, and then divide A into N disjoint parts S1, S2, S3…SN.

[0121] (3) The GPU execution kernel performs matrix multiplication tests on the above data sets. The array Pg records the execution time and corresponding data size on S1, S1+S2, S1+S2+S3…S1+S2+S3+…+SN. To eliminate noise interference, the execution time is the average of 10 tests.

[0122] (4) Use the array Pg to fit the GPU cost model (Definition 2) to generate the GPU execution kernel cost function .

[0123] (5) Based on the prior knowledge of data transfer speeds of different sizes, the cost function is fitted according to the data transfer model from CPU to GPU (Definition 2) .

[0124] (6) GPU overall cost function Depends on the maximum between the data transfer time from CPU to GPU and the execution time of the GPU kernel, as follows:

[0125] .

[0126] 4. The total time for each Worker node to run the matrix multiplication test program As shown in the following formula, Represents the number of CPU threads, Represents the number of GPUs:

[0127] ;

[0128] According to the calculated cost function, when the load between resources is balanced, the total running time is minimized, and the following optimization problem is solved to obtain , and obtain :

[0129] .

[0130] 5. As shown in the following formula, set the weight according to the different computing performance of the Worker node :

[0131] ;

[0132] According to the weight ratio of each node, call Algorithm 3 and Algorithm 4 to divide matrices A and B into blocks. Figure 2 , Figure 3 and Figure 4 shown.

[0133] Algorithm 3: (P, Q, R)-Cuboid Block Algorithm: Divide the 3D model space into multiple cuboid-shaped volume element blocks, and make the size of each cuboid (the sum of the block sizes of A, B, and C) conform to the maximum value that can be accommodated in the task memory.

[0134] (1) Input: matrices A and B.

[0135] (2) The three-dimensional model space constructed by the input matrix is ​​divided into blocks, generating a total of P˙Q˙R cuboids, where P, Q, and R represent the number of partitions on the i-axis, j-axis, and k-axis, respectively (0<P≤I, 0<Q≤J, 0<R≤K), and Dp,q,r is used to represent the index of a specific cuboid (0≤p<P, 0≤q<Q, 0≤r<R).

[0136] (3) The optimal parameters P*, Q*, R* of P, Q, and R are obtained by solving the following optimization problem:

[0137] ;

[0138] in, Indicates the amount of memory that each task can use, function Indicates that each task uses parameters Memory usage at:

[0139] ;

[0140] function represents the communication cost and is defined as follows: represents the amount of data copying of A and B in the matrix block step, Represents the amount of data in C during the matrix aggregation step:

[0141] ;

[0142] In practice, the RDD of the input matrix is ​​repartitioned into matrix blocks. The matrix blocks are represented using the data structure Key-ValueRDD, where the key is the row and column index of the block, the value is the element in the matrix (represented using a two-dimensional array), and a label is stored to identify the index of the matrix block.

[0143] (3) Returns the RDD of matrices A and B.

[0144] Algorithm 4: , , ) - Sub-cuboid partitioning algorithm: Divide each cuboid into multiple sub-cubes so that the size of each sub-cuboid fits the GPU memory space capacity .

[0145] (1) Input: a volume element block in the shape of a cuboid;

[0146] (2) Divide the cuboid into sub-cubic blocks, generating a total of A sub-cuboid, , , Respectively represent the number of partitions on the i-axis, j-axis, and k-axis in a single cuboid, using Represents the index of a specific sub-cuboid;

[0147] (3) , , The best parameters By solving the following optimization problem:

[0148] ;

[0149] .

[0150] 6. In order to further reduce the communication cost, the matrix blocks of the repartitioned matrices A and B are compressed:

[0151] (1) Represent the matrix block using the compressed sparse row / value (CSRV) representation (Algorithm 5);

[0152] (2) Execute the Elf lossless floating-point compression algorithm (Algorithm 6) on the non-zero elements (floating-point numbers) of the matrix in the array V generated by the above CSRV representation;

[0153] (3) Compress the CSRV representation of the matrix block using the Recursive Paring algorithm (Algorithm 7);

[0154] (4) Each matrix block is transferred to the corresponding Worker node, and then the decompression algorithm (Algorithm 8) is executed.

[0155] Algorithm 5: Compressed Sparse Row / Value (CSRV) Representation, refer to the following matrix and its CSRV representation example:

[0156]

[0157] in:

[0158] (1) Input: RDD of matrix block M;

[0159] (2) Use array V to store all the different non-zero values ​​in M;

[0160] (3) Generate array S by scanning M row by row:

[0161] a. For each non-zero matrix element M[i][j]≠0, use the binary<l,j> Indicates that l represents the index in V and j represents the column number, so that V[l] = M[i][j];

[0162] b. After scanning each row, add a special element $ to S;

[0163] (4) Compress the two-dimensional array used to store the matrix in the RDD of the matrix block M into the CSRV representation, that is, the generated array V and array S.

[0164] Algorithm 6: Erasing-based Lossless Floating-Point Compression: Given a double-precision value ,turn up The best tail prefix number ,according to Lossless precision compression and recovery . ( )

[0165] (1) Input: double precision value ;

[0166] (2) Calculate the number of decimal places And record it as ,make ;

[0167] (3) Since the decimal value need Binary bit representation, It can be considered as erasure In the binary representation of The following bits, according to definition 9, , so erase of The mantissa digits after that can directly calculate the optimal mantissa prefix number ;

[0168] (4) Since the minimum value of a double-precision floating-point number is approximately , direct storage ,need⌈ ⌉ = 9 binary digits, while the decimal digits of a double-precision floating point number are Not more than 17 years old, and It can be calculated as follows:

[0169] ;

[0170] In order to improve the compression rate and facilitate calculation ,right Modify it as follows:

[0171] .

[0172] (5) To ensure that compression achieves positive gain, the following conditions must be met simultaneously: 1) ; 2) ; 3) ;use replace , and uses 4 bits to store , used to restore .otherwise, No substitution is made.

[0173] Algorithm 7: Recursive Paring: Replace the most frequent pairs of adjacent elements in the original sequence with new elements, re-evaluate the frequencies of all pairs of elements relative to the now expanded set of elements, and repeat the process until no pair of adjacent elements appears twice, as shown in the following example of a CSRV representation of a compressed matrix based on a syntax compression algorithm:

[0174]

[0175] in:

[0176] (1) Input: CSRV representation of the matrix stored in the RDD of the matrix block M;

[0177] (2) For array S, identify elements a and b, and ab is the most frequently occurring pair of adjacent elements in the original sequence. If no pair of elements appears more than once, then end;

[0178] (3) Introducing a new element , replace all occurrences of ab with , that is, generate a rule ;

[0179] (4) Repeat step 2 until no pair of adjacent elements appears twice;

[0180] (5) Compress the CSRV representation used to store the matrix in the RDD of the matrix block M into a generated set of rules R and string sequence C, where each is a non-terminal symbol, and its expansion represents the sequence of non-zero matrix element pairs in the jth row. The expansion of the string C represents the sequence S:

[0181] .

[0182] Algorithm 8: Decompression

[0183] (1) Input: RDD of matrix block M (whose matrix representation has been compressed);

[0184] (2) Scan the rule set R in reverse order from back to front. For each rule ,use Replace all occurrences of C in string , restore the string C to CSRV representation;

[0185] (3) In the CSRV representation of the matrix, for the non-zero values ​​in the array V (Just an example) Perform the decompression process in sequence:

[0186] a. Read Corresponding and ;

[0187] b. If ,but ,in ;

[0188] c. If , then according to the following formula from and recover :

[0189] ;

[0190] .

[0191] (4) Scan the sequence S in the CSRV representation one by one, according to the two-tuple<l,j> Uniquely determine the corresponding non-zero matrix elements, wrap when encountering $, and the remaining element values ​​are 0, generating a two-dimensional array representation of the matrix;

[0192] (5) Restore the compressed representation of the matrix stored in the RDD of the matrix block M into a two-dimensional array.

[0193] 7. Each Worker node executes its own workload according to the GPU workload ratio (Step 4) The matrix cuboid consisting of the matrix blocks of input matrices A and B Divide into two sub-cubic blocks along the k axis and , used for CPU and GPU to perform matrix multiplication respectively:

[0194] (1) Use the CPU to perform matrix multiplication, running directly in memory;

[0195] (2) Using the GPU to perform matrix multiplication is done in three steps:

[0196] a. Copy from memory to GPU memory;

[0197] b. Perform matrix multiplication in GPU memory;

[0198] c. Copy the calculation results from GPU memory back to main memory;

[0199] (3) Aggregate the calculation results of step (1) and step (2) to generate an intermediate block of matrix C.

[0200] In order to maximize the utilization of heterogeneous resources (CPU and GPU) in Spark nodes, GPU utilization is checked regularly using GPU monitor and its results are read by the adaptive scheduler. When GPU utilization is low, the adaptive scheduler increases the number of GPU containers to fully utilize the GPU.

[0201] The intermediate blocks of matrix C (reference Figure 5 The aggregation operation is performed on t0 and t1 in t0 and t1 to obtain the final output block of matrix C.

[0202] In summary, the method provided in this embodiment can reasonably allocate matrix calculation tasks and make full use of different hardware resources, reduce data communication costs through lossless floating-point matrix compression, improve the computational efficiency of large-scale distributed matrix multiplication, and solve the technical problems of insufficient memory due to high memory usage of each task and performance degradation due to high communication costs in traditional solutions. In addition, the method can not only speed up task execution by combining CPU and GPU, but also effectively utilize the computing resources available in heterogeneous systems, solving the technical problem of hardware acceleration without multi-core accelerators in traditional solutions.

[0203] Example 3

[0204] Based on Example 1, the embodiment of the present invention further provides a matrix multiplication system for distributed heterogeneous computing nodes, referring to Figure 6 As shown, the system includes a matrix data reading module, a matrix block acquisition module, an intermediate block acquisition module and a final output matrix acquisition module, wherein: the matrix data reading module is used to read matrix data from a distributed file system; the matrix block acquisition module is used to quantify the CPU or GPU computing performance on each working node in the cluster; the matrix data is divided into blocks according to the performance to obtain matrix blocks; lossless floating point number compression is performed on the matrix blocks and transmitted to each working node; the intermediate block acquisition module is used to allocate matrix blocks to CPU or GPU tasks in each working node according to the computing performance ratio of the hardware, and in each task, the input matrix blocks are multiplied to generate intermediate blocks of the output matrix blocks; the final output matrix acquisition module is used to aggregate the intermediate blocks of the output matrix blocks to generate the final output matrix.

[0205] Each module described in the above system has a detailed introduction to its corresponding function in Example 1, which will not be repeated here.

[0206] Example 4

[0207] The embodiment of the present invention also provides a matrix multiplication device for distributed heterogeneous computing nodes, see Figure 7 , which shows that it is possible to execute Figure 1 The hardware structure of the matrix multiplication method for distributed heterogeneous computing nodes.

[0208] The matrix multiplication device for distributed heterogeneous computing nodes includes: at least one processor 11; and a memory 12 connected to the at least one processor 11 for communication. Figure 7 The memory 12 stores instructions that can be executed by the at least one processor 11, and the instructions are executed by the at least one processor 11 so that the at least one processor 11 can perform the above Figure 1 The matrix multiplication method for distributed heterogeneous computing nodes. The processor 11 and the memory 12 can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0209] The memory 12, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the matrix multiplication method for distributed heterogeneous computing nodes in the embodiment of the present invention. The processor 11 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, implements the matrix multiplication method for distributed heterogeneous computing nodes in the above method embodiment.

[0210] The memory 12 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created by the use of a matrix multiplication system for distributed heterogeneous computing nodes, etc. In addition, the memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the matrix multiplication system for distributed heterogeneous computing nodes via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0211] The one or more modules are stored in the memory 12, and when executed by the one or more processors 11, execute the matrix multiplication method for distributed heterogeneous computing nodes in any of the above method embodiments, for example, execute the above described Figure 1 method steps.

[0212] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0213] The embodiment of the present invention further provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 method steps.

[0214] The embodiment of the present invention further provides a computer program product, including a computing program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the matrix multiplication method for distributed heterogeneous computing nodes in any of the above method embodiments, for example, executing the above described Figure 1 method steps.

[0215] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A matrix multiplication method for distributed heterogeneous computing nodes, characterized in that: include: Deploy the Spark system to a heterogeneous distributed computer cluster, where each computer node has a different CPU or GPU configuration; use Spark as a distributed computing execution engine to read matrix data from a distributed file system; Quantify the CPU or GPU computing performance on each worker node in the cluster. Specifically, based on the input matrix A, let α and 1-α be the workload ratios assigned to the GPU and CPU, respectively, where 0≤α≤1; use Tg(α) and Tc(1-α) to represent the time spent by the GPU and CPU threads to update the elements in α˙A and (1-α)˙A, respectively; use fg(α) and fc(1-α) to represent the cost functions of the GPU and CPU threads and The estimate of , denoted as fg(α) ,fc(1-α) ; Divide the matrix data into blocks according to performance to obtain matrix blocks. Specifically, the total time for each working node to run the matrix multiplication test program for: ;in, Represents the number of CPU threads, represents the number of GPUs; i represents the i-th worker node; according to the calculated cost function, when the load between resources is balanced, the total running time is minimized, and the following optimization problem is solved to obtain , and obtain : ; Set weights based on the different computing performance of the working nodes : ; According to the weight ratio of each node, call the cuboid block algorithm to block the matrices A and B; Cuboid block algorithm: divide the three-dimensional model space into multiple cuboid-shaped volume element blocks, and make the size of each cuboid meet the maximum value that the task memory can accommodate; Input: matrices A, B; Block the three-dimensional model space constructed by the input matrix, and generate a total of P˙Q˙R cuboids, where P, Q, R represent the number of partitions on the i-axis, j-axis, and k-axis, respectively, 0<P≤I, 0<Q≤J, 0<R≤K, use Dp,q,r to represent the index of a specific cuboid, 0≤p<P, 0≤q<Q, 0≤r<R; The optimal parameters of P, Q, R By solving the following optimization problem: ;in, Indicates the amount of memory that each task can use, function Indicates that each task uses parameters Memory usage at: ;function represents the communication cost and is defined as follows: represents the amount of data copying of A and B in the matrix block step, Represents the amount of data in C during the matrix aggregation step: ; The RDD of the input matrix is ​​repartitioned into matrix blocks; the matrix blocks are represented using the data structure Key-ValueRDD, where the key is the row and column index of the block, the value is the element in the matrix, and a label is stored to identify the index of the matrix block; return the RDD of the matrix A and B blocks; perform lossless floating point compression on the matrix blocks and transmit them to each working node; Each working node allocates matrix blocks to CPU or GPU tasks according to the computing performance ratio of the hardware. In each task, the input matrix blocks are multiplied to generate an intermediate block of the output matrix block; The intermediate blocks of the output matrix blocks are aggregated to generate the final output matrix.

2. The matrix multiplication method for distributed heterogeneous computing nodes according to claim 1, characterized in that: The performing lossless floating point compression on the matrix block and transmitting it to each working node comprises: Use compressed sparse row representation to represent matrix blocks; Perform a lossless floating point compression algorithm on the non-zero elements of the matrix in the array V produced by the compressed sparse row representation; Compressed sparse row representation of compressed matrix blocks; The compressed matrix blocks are transmitted to the corresponding working nodes, and then the decompression algorithm is executed.

3. The matrix multiplication method for distributed heterogeneous computing nodes according to claim 1, characterized in that: The intermediate block for multiplying the input matrix block to generate the output matrix block specifically includes: Based on GPU workload ratio , the matrix cuboid consisting of the matrix blocks of the input matrix Divide into two sub-cubic blocks along the k axis and , used for CPU and GPU to perform matrix multiplication, respectively, where: Use the CPU to perform matrix multiplication, running directly in memory; Use GPU to perform matrix multiplication. Specifically: Copy from main memory to GPU memory; perform matrix multiplication in GPU memory; copy the result from GPU memory back to main memory; Will Use the CPU to perform matrix multiplication and The calculation results of matrix multiplication performed using the GPU are aggregated to generate an intermediate block of output matrix blocks.

4. The matrix multiplication method for distributed heterogeneous computing nodes according to claim 1, characterized in that: Also includes: Use the GPU monitor to periodically check GPU utilization and read its results by the adaptive scheduler; When the GPU utilization is lower than the preset value, the number of GPU containers is increased through the adaptive scheduler to fully utilize the GPU.

5. The matrix multiplication method for distributed heterogeneous computing nodes according to claim 1, characterized in that: The step of aggregating the intermediate blocks of the output matrix blocks to generate the final output matrix comprises: An aggregation operation is performed on the intermediate blocks of output matrix blocks from the same group of worker nodes to obtain the final output block of the final output matrix.

6. A matrix multiplication system for distributed heterogeneous computing nodes, used to implement the matrix multiplication method for distributed heterogeneous computing nodes as described in any one of claims 1 to 5, characterized in that: It includes a matrix data reading module, a matrix block acquisition module, an intermediate block acquisition module and a final output matrix acquisition module, wherein: The matrix data reading module is used to read matrix data from a distributed file system; The matrix block acquisition module is used to quantify the CPU or GPU computing performance on each working node in the cluster; divide the matrix data into blocks according to the performance to obtain matrix blocks; perform lossless floating point compression on the matrix blocks and transmit them to each working node; The intermediate block acquisition module is used to allocate matrix blocks to CPU or GPU tasks according to the computing performance ratio of the hardware at each working node, and in each task, multiply the input matrix blocks to generate the intermediate blocks of the output matrix blocks; The final output matrix acquisition module is used to aggregate the intermediate blocks of the output matrix blocks to generate the final output matrix.

7. A matrix multiplication device for distributed heterogeneous computing nodes, characterized in that: It includes at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the matrix multiplication method for distributed heterogeneous computing nodes described in any one of claims 1-5.

8. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the matrix multiplication method for distributed heterogeneous computing nodes as described in any one of claims 1 to 5.

Images