A method and system for reducing single-node memory usage based on distributed reading of ultra-large-scale sparse matrices

Abstract

This invention relates to a method and system for reducing single-node memory usage based on distributed reading of ultra-large-scale sparse matrices. The ultra-large-scale sparse matrix is ​​stored in Rutherford-Boeing format, including: (1) converting the Rutherford-Boeing format to row compression for convenient matrix operations and parallel processing, and storing the data in row compression format; (2) each node calculates the row pointer, column index, and non-zero element value of its respective block matrix row compression; (3) each node writes the calculated row pointer, column index, and non-zero element value to its respective file in row compression format; (4) when the application needs the ultra-large-scale sparse matrix, the node reads the data from its respective file, and the process ends. This invention achieves fast reading speeds when multiple nodes read in parallel, eliminates the need for data distribution between nodes, avoids communication congestion, and improves overall performance.

Description

Technical Field

[0001] This invention relates to a method and system for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrices, belonging to the technical field of numerical computation and computer applications. Background Technology

[0002] In a matrix, if the number of zero elements far exceeds the number of non-zero elements, and the non-zero elements are irregular (generally, the number of non-zero elements is less than 5% of the total matrix elements, or even less than 1%), then the matrix is ​​called a sparse matrix.

[0003] Large-scale sparse matrices require storage during the solution of equations. Storing sparse matrices using the same methods as storing ordinary dense matrices would consume a significant amount of storage space. Even if storage were possible on a cluster, most stored elements would be zero, resulting in wasted memory. Therefore, reducing the storage size of large-scale sparse matrices and minimizing communication overhead during numerical computation is essential. There are four basic storage formats for sparse matrices: COO (Coordinate List), CSR (Compress Sparse Row), CSC (Compress Sparse Column), and ELLPACK. CSR and CSC are based on COO, compressing rows and columns respectively. The Rutherford-Boeing format shares the same storage method as CSC but stores additional information.

[0004] Most matrices are stored in Rutherford-Boeing format. Specifically:

[0005] Line 1: Format (A72, A8), where A72 is a character-based explanation of the file header, such as the stiffness matrix or mass matrix title; A8 is the matrix name or ID number.

[0006] Line 2: Format (I14, 3(1X, I13)), which represents the total number of lines in the file (excluding the file header), the total number of rows of matrix column pointers, the total number of rows of matrix row indices, and the total number of rows of matrix element values, respectively.

[0007] Line 3: Format (A3, 11X, 4(1X, I13)), where A3 is the matrix type (the first character can be: R—real matrix, C—complex matrix, I—integer matrix, P—matrix structure only (no element values), Q—matrix structure (element values ​​are in the relevant file); the second character can be: S—symmetric matrix (symmetric matrix only stores the lower triangular elements), U—asymmetric matrix, H—Hermitian matrix, Z—ill-symmetric matrix; R—banded matrix; the third character can be: A—column compressed form, E—basic form), and the others are the number of matrix rows, the number of matrix columns, the total number of non-zero elements, and the number of matrix elements, respectively.

[0008] Line 4: Format (2A16, A20), representing column pointer format, row index format, and matrix value format, respectively.

[0009] After row 5: Matrix column pointer value, row index value, and non-zero element value.

[0010] Row compression stores a matrix row-wise. It uses three arrays—nonzval, rowptr, and colind—to store the sparse matrix. The index in array rowptr corresponds to the row index offset of the non-zero element in the matrix. Array colind stores the column index of each non-zero element in the original sparse matrix. Array nonzval stores the values ​​of all non-zero elements in the sparse matrix. If the matrix has dimensions n×n and the number of non-zero elements is nnz, a single node can be used to store the matrix. The size of nonzval and colind is nnz, and the size of rowptr is n+1.

[0011] The problem of storing large-scale sparse matrices is a crucial one in solving large-scale sparse linear equation systems, which are widely used in electromagnetics, mechanics, atmospheric modeling, biology, and other computational science and engineering fields. As the complexity of problems in these fields increases, the computational scale grows ever larger, making it even more important to accelerate the solution speed of large-scale sparse linear equation systems, with faster data retrieval being a critical aspect. Ultra-large-scale sparse matrices are exceptionally large; even with row compression, storage sizes can reach tens of gigabytes when dealing with matrices with tens of millions of dimensions. Using a single node to read the data is time-consuming, and network bandwidth limitations can prevent data transfer when that node tries to transmit matrix data to other nodes. Therefore, multiple nodes are needed for collaborative data retrieval. Summary of the Invention

[0012] To address the shortcomings of existing technologies, this invention provides a method for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrices. This method is used to solve or alleviate MPI communication problems encountered in certain high-performance applications and improve the overall performance of the applications.

[0013] The present invention also provides a system for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrices.

[0014] Terminology Explanation:

[0015] If the number of zero elements in a matrix far exceeds the number of non-zero elements, and the non-zero elements are irregular (generally, the number of non-zero elements is less than 5% of the total matrix elements, or even less than 1%), it is generally called a sparse matrix. When the dimension of the matrix reaches millions or tens of millions of dimensions, it is generally called a very large-scale sparse matrix.

[0016] The technical solution of this invention is as follows:

[0017] A method for reducing single-node memory usage based on distributed reading of a massive sparse matrix, wherein the massive sparse matrix is ​​stored in Rutherford-Boeing format, includes the following steps:

[0018] (1) To facilitate matrix operations and make them more parallel, the Rutherford-Boeing format is converted to row compression and the data is stored in row compression format;

[0019] (2) The matrix is ​​stored in row compression format. Each node calculates the row pointer, column index and non-zero element value of its own block matrix row compression.

[0020] (3) Each node writes its calculated row pointer, column index and non-zero element value to its own file in a row-compressed format.

[0021] (4) When the application requires a very large sparse matrix, each node reads data from its own file. This avoids a single node reading all the data and also speeds up the data reading process.

[0022] According to a preferred embodiment of the present invention, the specific implementation process of step (1) includes:

[0023] A. Calculate the number of non-zero elements in each row of a very large sparse matrix stored in Rutherford-Boeing format, as shown in equations (I) and (II):

[0024] j=colptr[i+1]-colptr[i] (Ⅰ)

[0025] k = rowind[j] (Ⅱ)

[0026] In equations (Ⅰ) to (Ⅱ), i refers to different values ​​from 0 to the number of columns n of the matrix, i = 0, 1, ..., n, n×n refers to the dimension of the matrix, j is the number of non-zero elements in each column, colptr is the column pointer stored in RB, k is the row number of the non-zero element, and rowind is the row index stored in RB.

[0027] After obtaining the row number of each non-zero element in the ultra-large-scale sparse matrix using equations (I) and (II), marker[k] is incremented by 1, indicating that the number of non-zero elements in the row containing k is incremented by 1.

[0028] Continue executing equations (Ⅰ) and (Ⅱ) until i = n, and calculate the number of non-zero elements in all rows of the ultra-large-scale sparse matrix;

[0029] B. Set the row pointer rowptr in CSR format and initialize it to 0, as shown in equations (III) and (IV):

[0030] rowptr[i+1]=rowptr[i]+marker[i]i=0,1....n (Ⅲ)

[0031] marker[i] = rowptr[i] (Ⅳ)

[0032] Set the column index colind and non-zero numerator nonzval in CSR format as shown in equations (V) to (VIII):

[0033] row = rowind[j] (V)

[0034] repos = marker[row] (Ⅵ)

[0035] colind[repos] = i (Ⅶ)

[0036] nonzval[repos]=nzval[j] (VIII)

[0037] In equations (V) to (VIII), j = colptr[i+1] - colptr[i], i = 0, 1, ..., n, row represents the row number, repos represents which non-zero element is the nth non-zero element in the entire matrix, and nzval represents the non-zero element value in RB format.

[0038] By using the colptr and rowind of the RB format, each non-zero element in the first column of the statistical matrix is ​​converted into its position in the entire matrix through row compression. This yields the column coordinate (colind) of the first non-zero element in each row during row compression. Then, the column coordinates of the other non-zero elements in each row are obtained sequentially. The storage of non-zero elements is also changed from column-based to row-based, which is equivalent to converting the Rutherford-Boeing format into row compression.

[0039] According to a preferred embodiment of the present invention, in step (2), the number of nodes depends on the memory size and the matrix size. Assuming there are p nodes, the size of the ultra-large-scale sparse matrix is ​​n×n, the number of non-zero elements is nnz, the storage capacity of the ultra-large-scale sparse matrix is ​​M, and the memory of each node is M. p Then the number of nodes is at least p. Otherwise, each node's memory would be insufficient to store the chunked data.

[0040] Suppose there are p nodes, P0, P1, P2, ..., Pn. i ....P p-1 Then node P i The number of lines processed, n_loc i Calculated using equation (IX):

[0041]

[0042] If the rows can be distributed evenly, then each node will process n_loc rows. i yes If there are any remaining nodes, they are from the last node P. p-1 If we process it, then the number of rows processed by the last node is...

[0043] Node P i The starting row index fst_row for processing very large sparse matrices is calculated by equation (X):

[0044] fst_row = P i *n_loc i (X)

[0045] The calculation of each node's row pointer rowptr_loc is shown in equation (XI):

[0046] rowptr_loc[m]=rowptr[m+fst_row]-rowptr[fst_row]m=0,1,2....n_loc(Ⅺ)

[0047] The calculation of the column index colind_loc for each node is shown in equation (XII):

[0048] colind_loc[m]=colind[j] (XII)

[0049] In equation (XII), j = rowptr[fst_row], ..., rowptr[fst_row + n_loc], and colind is the column index that stores the non-zero elements of the entire CSR format sparse matrix.

[0050] The calculation of the number of non-zero elements nnz_loc for each node is shown in equation (XIII):

[0051] nnz_loc = rowptr_loc[n_loc] i -rowptr_loc[0] (XIII)

[0052] The calculation of each node's nonzero element nonzval_loc is shown in equation (XIV):

[0053] nonzval_loc[m]=nonzval[j] (XIV)

[0054] In equation (XIV), j = rowptr[fst_row], ..., rowptr[fst_row + n_loc], and nonzval stores the non-zero elements of the entire sparse matrix in CSR format.

[0055] A system for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrices is provided to implement the above-mentioned method for reducing memory based on distributed reading of ultra-large-scale sparse matrices. The system includes a format conversion module, a calculation module, a writing module, and an application module. The format conversion module is used to implement step (1); the calculation module is used to implement step (2); the writing module is used to implement step (3); and the application module is used to implement step (4).

[0056] The beneficial effects of this invention are as follows:

[0057] When multiple nodes read data in parallel, the reading speed is fast, and there is no need to distribute data between nodes, which avoids communication congestion and improves the overall performance of the application. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the process of a method for reducing single-node memory based on distributed reading of a large-scale sparse matrix according to the present invention.

[0059] Figure 2 A time comparison chart of reading RB format matrix data and distributed reading of CSR format data.

[0060] Figure 3 This is a schematic diagram of storing the matrix exemplified in Example 3 using Rutherford-Boeing format;

[0061] Figure 4 This is a schematic diagram of storing the matrix exemplified in Example 3 in CSR format. Figure 1 ;

[0062] Figure 5This is a schematic diagram of storing the matrix exemplified in Example 3 in CSR format. Figure 2 ;

[0063] Figure 6 This is a schematic diagram of storing the matrix exemplified in Example 3 in CSR format. Figure 3 ;

[0064] Figure 7 This is a schematic diagram of storing the matrix exemplified in Example 3 in CSR format. Figure 4 . Detailed Implementation

[0065] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0066] Example 1

[0067] A method for reducing single-node memory usage based on distributed reading of ultra-large-scale sparse matrices, where the ultra-large-scale sparse matrix is ​​stored in Rutherford-Boeing format, such as... Figure 1 As shown, the steps are as follows:

[0068] (1) To facilitate matrix operations and make them more parallel, the Rutherford-Boeing format is converted to row compression and the data is stored in row compression format;

[0069] (2) The matrix is ​​stored in row compression format. Each node calculates the row pointer, column index and non-zero element value of its own block matrix row compression.

[0070] (3) Each node writes its calculated row pointer, column index and non-zero element value to its own file in a row-compressed format.

[0071] Each node writes the row pointer, column index and non-zero element value of its own CSR format calculated in step (2) to its own file. Each node is only responsible for storing the compressed matrix part of its own row.

[0072] (4) When the application requires a very large sparse matrix, the nodes read data from their respective files. This avoids a single node reading all the data and also speeds up the data reading process.

[0073] Example 2

[0074] The method for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrices as described in Example 1 differs in that:

[0075] The specific implementation process of step (1) includes:

[0076] A. Calculate the number of non-zero elements in each row of a very large sparse matrix stored in Rutherford-Boeing format, as shown in equations (I) and (II):

[0077] j=colptr[i+1]-colptr[i] (Ⅰ)

[0078] k = rowind[j] (Ⅱ)

[0079] In equations (I) and (II), i refers to different values ​​from 0 to the number of columns n of the matrix, i = 0, 1, ..., n, n×n refers to the matrix dimension, j is the number of non-zero elements in each column, colptr is the column pointer stored in RB, k is the row index of the non-zero element in each column, and rowind is the row index stored in RB. After obtaining the row index k of the non-zero element in each column of the ultra-large-scale sparse matrix through equations (I) and (II), marker[k] is incremented by 1, indicating that the number of non-zero elements in the row containing k is incremented by 1. Equations (I) and (II) are continued until i = n, to obtain the number of non-zero elements in all rows of the ultra-large-scale sparse matrix.

[0080] B. Set the row pointer rowptr in CSR format and initialize it to 0, as shown in equations (III) and (IV):

[0081] rowptr[i+1]=rowptr[i]+marker[i]i=0,1....n (Ⅲ)

[0082] marker[i] = rowptr[i] (Ⅳ)

[0083] Set the column index colind and non-zero numerator nonzval in CSR format as shown in equations (V) to (VIII):

[0084] row = rowind[j] (V)

[0085] repos = marker[row] (Ⅵ)

[0086] colind[repos] = i (Ⅶ)

[0087] nonzval[repos]=nzval[j] (VIII)

[0088] In equations (V) to (VIII), row represents the row number, repos represents which non-zero element is the nth non-zero element in the entire matrix, and nzval refers to the non-zero element in RB format;

[0089] By using the colptr and rowind of the RB format, each non-zero element in the first column of the statistical matrix is ​​converted into its position in the entire matrix through row compression. This yields the column coordinate (colind) of the first non-zero element in each row during row compression. Then, the column coordinates of each non-zero element in each row during row compression are obtained sequentially. The storage of non-zero elements is also changed from column-based to row-based, which is equivalent to converting the Rutherford-Boeing format into row compression.

[0090] Example 3

[0091] The method for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrices as described in Example 2 differs in that:

[0092] In step (2), the number of nodes depends on the memory size and the matrix size. Assuming there are p nodes, the size of the ultra-large-scale sparse matrix is ​​n×n, the number of non-zero elements is nnz, the storage requirement of the ultra-large-scale sparse matrix is ​​M, and the memory of each node is M. p Then the number of nodes is at least p. Otherwise, each node's memory would be insufficient to store the chunked data.

[0093] Suppose there are p nodes: P0, P1, P2, ..., P i ....P p-1 Then node P i The number of lines processed, n_loc i Calculated using equation (IX):

[0094]

[0095] If the rows can be distributed evenly, then each node will process n_loc rows. i yes If there are any remaining nodes, they are from the last node P. p-1 If we process it, then the number of rows processed by the last node is...

[0096] Node P i The starting row index fst_row for processing very large sparse matrices is calculated by equation (X):

[0097] fst_row = P i *n_loc i (X)

[0098] In equation (X), the size of colind and nonzval is rowptr[n_loc]-rowptr[0]; colind stores the column index of each non-zero element in the original sparse matrix, and nonzval stores the non-zero elements;

[0099] The calculation of each node's row pointer rowptr_loc is shown in equation (XI):

[0100] rowptr_loc[m]=rowptr[m+fst_row]-rowptr[fst_row]m=0,1,2....n_loc(Ⅺ)

[0101] The calculation of the column index colind_loc for each node is shown in equation (XII):

[0102] colind_loc[m]=colind[j] (XII)

[0103] In equation (XII), j = rowptr[fst_row], ..., rowptr[fst_row + n_loc], and colind is the column index that stores the non-zero elements of the entire CSR format sparse matrix.

[0104] The formula for calculating the number of non-zero elements nnz_loc for each node is shown in equation (XIII):

[0105] nnz_loc = rowptr_loc[n_loc] i -rowptr_loc[0] (XIII)

[0106] The formula for calculating the nonzero element nonzval_loc for each node is shown in equation (XIV):

[0107] nonzval_loc[m]=nonzval[j] (XIV)

[0108] In equation (XIV), j = rowptr[fst_row], ..., rowptr[fst_row + n_loc]. nonzval stores the non-zero elements of the entire sparse matrix in CSR format.

[0109] Take the following matrix as an example:

[0110]

[0111] The sparse matrix is ​​stored in Rutherford-Boeing format: the index positions in the array `colptr` correspond to the relative offsets of the column indices of the non-zero elements in the matrix. The second number 1 and the first number 0 indicate that there is one non-zero element in column 0, the third number 3 and the second number 1 indicate that there are two non-zero elements in column 1, and so on. The array `rowind` stores the row indices of each non-zero element in the original sparse matrix. The first number 0 indicates that the row index of the first non-zero element is 0, the second number 1 indicates that the row index of the second non-zero element is 1, and so on. The array `nzval` stores all non-zero elements in the sparse matrix. The first number 1 represents a non-zero element with a value of 1 in column 0 and row 0, the second number 3 represents a non-zero element with a value of 3 in column 1 and row 1, the third number 5 represents a non-zero element with a value of 5 in column 1 and row 3, and so on. Figure 3 As shown.

[0112] The matrix is ​​stored in CSR format, with the index positions in the `rowptr` array corresponding to the relative offsets of the row numbers of the non-zero elements in the matrix. The second number 2 and the first number 0 indicate that there are two non-zero elements in row 0, the third number 3 and the second number 2 indicate that there is one non-zero element in row 1, and so on. The `colind` array stores the column numbers of each non-zero element in the CSR sparse matrix. The first number 0 indicates that the column number of the first non-zero element is 0, the second number 2 indicates that the column number of the second non-zero element is 2, and so on. The `nonzval` array stores all non-zero elements in the sparse matrix. The first number 1 represents a non-zero element with a value of 1 in row 0 and column 0, the second number 2 represents a non-zero element with a value of 2 in row 0 and column 2, the third number 3 represents a non-zero element with a value of 3 in row 1 and column 1, and so on. Figure 4 As shown.

[0113] Assuming there are two nodes, node 0 is responsible for rows 0 and 1 of the matrix, and node 1 is responsible for rows 2 and 3 of the matrix. Figure 5 As shown.

[0114] Stored in CSR format, each matrix is ​​stored as follows, taking node 1 as an example:

[0115] fst_row=2, rowptr_loc[0]=0, rowptr_loc[1]=rowptr[3]-rowptr[2]=4-3=1, rowptr_loc[2]=rowptr[4]-rowptr[2]=6-3=3. colind_loc[0]=colind[3]=2, colind_loc[1]=colind[4]=1, colind_loc[2]=colind[5]=3, nonzval_loc[0]=nonzval[3]=4, nonzval_loc[1]=nonzval[4]=5, nonzval_loc[2]=nonzval[5]=6, such as Figure 6 , Figure 7 As shown.

[0116] The test data for this example comes from the University of Florida's sparse matrix collection https: / / sparse.tamu.edu / . Figure 2 As shown, Data 1 is a directed weighted graph with a matrix dimension of 5154859×5154859 and 99199551 non-zero elements. Data 2 is a structural problem with a matrix dimension of 381689×381689 and 37464962 non-zero elements. Data 3 is a semiconductor process problem with a matrix dimension of 2146677×2146677 and 65129037 non-zero elements.

[0117] Compared to the single-node reading of RB, the CSR block reading method improves the speed of data 1 by 64.026%, data 2 by 72.506%, and data 3 by 48.718%.

[0118] This invention is designed for large-scale sparse matrices and employs a distributed reading method, which improves speed to varying degrees compared to the previous single-node reading method. Taking the experimental data in this embodiment as an example, the speed is improved by more than 40%.

[0119] Example 4

[0120] A system for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrix is ​​used to implement the method for reducing single-node memory based on distributed reading of ultra-large-scale sparse matrix as described in any of Examples 1-3. The system includes a format conversion module, a calculation module, a writing module and an application module. The format conversion module is used to implement step (1); the calculation module is used to implement step (2); the writing module is used to implement step (3); and the application module is used to implement step (4).

Claims

1. A method for reducing single-node memory based on distributed reading of a super-large sparse matrix, comprising the following steps: (1) The super-large sparse matrix is mostly stored in Rutherford-Boeing format, and the Rutherford-Boeing format is converted into row compression for the convenience of matrix operation and parallelism, and the data is stored in the row compression format; (2) The matrix is stored in the row compression format, and each node calculates the row pointer, column index and non-zero element value of the respective sub-block matrix row compression; (3) Each node writes the respective row pointer, column index and non-zero element value calculated in the row compression format into a respective file; (4) When the super-large sparse matrix is required by an application, the node reads data from the respective file; The specific implementation process of step (1) comprises: A. The number of non-zero elements of each row of the super-large sparse matrix stored in the Rutherford-Boeing format is calculated; as shown in formula (I) and formula (II): j = colptr[i + 1] - colptr[i] (I) k = rowind[j] (II) In formula (I) to formula (II), i refers to different values from 0 to the number of rows n of the matrix, i = 0, 1,..., n, n x n refers to the dimension of the matrix, j refers to the number of non-zero elements in each column, colptr refers to the column pointer stored in RB, k refers to the row index of the non-zero element, and rowind refers to the row index stored in RB; after the row index of the non-zero element in each column of the super-large sparse matrix is calculated by formula (I) and formula (II) each time, marker[k] is incremented by 1, indicating that the number of non-zero elements in the row where k is located is incremented by 1; Continue to execute formula (I) and formula (II) until i = n, and the number of non-zero elements of all rows of the super-large sparse matrix is calculated; B. The row pointer rowptr of the CSR format is set to 0, and initialized as shown in formula (III) and formula (IV): rowptr[i + 1] = rowptr[i] + marker[i] i = 0, 1,..., n (III) marker[i] = rowptr[i] (IV) The column index colind and the non-zero element nonzval of the CSR format are set as shown in formula (V) to formula (VIII): row = rowind[j] (V) repos = marker[row] (VI) colind[repos] = i (VII) nonzval[repos] = nzval[j] (VIII) In formula (V) to formula (VIII), j = colptr[i + 1] - colptr[i], i = 0, 1,..., n, row represents the row index, repos represents the number of non-zero elements in the entire matrix, and nzval represents the non-zero element value in the RB format. The first column of each non-zero element of the matrix RB format is compressed in the position of the entire matrix by the colptr and rowind of the RB format, and the column coordinates of the first non-zero element of each row in the row compression are obtained, and the column coordinates of other non-zero elements of other rows in the row compression are obtained in turn, and the storage of the non-zero elements is also changed from column storage to row storage, that is, the Rutherford-Boeing format is converted into row compression.

2. The method for reducing memory of a single node based on distributed reading of an ultra-large sparse matrix according to claim 1, characterized in that, In step (2), assuming p nodes, the size of the super large sparse matrix is n x n, the number of non-zero elements is nnz, the storage of the super large sparse matrix is M, and the memory of each node is M p Therefore, the number of nodes is at least p, Otherwise, the memory of each node cannot store the data of the block. Assume p nodes are: P0, P1, P2...P i ....P p-1 , then node P i The number of rows n_loc i calculated by formula (IX) If equally distributed, the number of rows processed by each node n_loc i is If there is a remainder, processed by the last node P p-1 , the number of rows processed by the last node is Node P i The starting row index fst_row for processing the super large sparse matrix is calculated by formula (X): fst_row = P i *n_loc i (X) In formula (X), the size of colind and nonzval is rowptr[n_loc]-rowptr[0]; colind stores the column index of each non-zero element in the original sparse matrix, and nonzval stores the non-zero element.

3. The method for reducing memory of a single node based on distributed reading of an ultra-large sparse matrix according to claim 2, characterized in that, In step (2), the calculation formula of the row pointer rowptr_loc of each node is shown in formula (XI): rowptr_loc[m]=rowptr[m+fst_row]-rowptr[fst_row]m=0,1,2....n_loc (XI).

4. The method for reducing memory based on distributed reading of ultra-large sparse matrix according to claim 2, characterized in that, In step (2), the calculation formula of the column index colind_loc of each node is shown in formula (XII): colind_loc[m]=colind[j] (XII) In formula (XII), j=rowptr[fst_row],......rowptr[fst_row+n_loc], and colind stores the column index of each non-zero element in the entire CSR format sparse matrix.

5. The method of claim 2, wherein, In step (2), the calculation formula of the number of non-zero elements nnz_loc of each node is shown in formula (XIII): nnz_loc = rowptr_loc [n_loc i ] - rowptr_loc [0] (XIII) In formula (XIII), rowptr_loc refers to the row pointer, and the calculation of the non-zero element nonzval_loc of each node is shown in formula (XIV): nonzval_loc[m]=nonzval[j] (XIV) In formula (XIV), j=rowptr[fst_row],......rowptr[fst_row+n_loc], nonzval stores the non-zero element in the entire CSR format sparse matrix, m=0,1,2....n_loc, and n_loc refers to the number of rows processed by the node.

6. A system for reducing the memory of a single node based on distributed reading of a super-large sparse matrix, for implementing the method for reducing the memory of a single node based on distributed reading of a super-large sparse matrix according to any one of claims 1-5, characterized in that, The application comprises a format conversion module, a calculation module, a writing module and an application module, the format conversion module is used to realize the step (1); the calculation module is used to realize the step (2); the writing module is used to realize the step (3); and the application module is used to realize the step (4).

Images