Large-scale sparse matrix multiplication method based on mapreduce

A sparse matrix and multiplication operation technology, applied in the field of large-scale sparse matrix multiplication operation, can solve problems such as low execution performance, large dimensions, and inability to execute, and achieve the effect of reducing operating resource constraints, reducing computation, and improving operating speed.

Inactive Publication Date: 2013-05-15
FUJIAN TQ DIGITAL
View PDF1 Cites 17 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The invention converts the problem of large matrix multiplication into an operation suitable for mapreduce, and solves the problem of low execution performance or even inability to perform large-scale matrix multiplication due to resource constraints in a stand-alone environment due to excessive dimensions.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Large-scale sparse matrix multiplication method based on mapreduce

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] The present invention is realized like this: a kind of method based on mapreduce large-scale sparse matrix multiplication operation, suppose large-scale sparse matrix is ​​A and B, the product matrix of A and B is C,

[0037] A={(i,k,A ik )︱i∈[1,2…m],k∈[1,2…n],A ik ≠0},

[0038] B={(k,j,B kj )︱k∈[1,2…n],j∈[1,2…l],B kj ≠0},

[0039] Find the matrix C={(i,j,C ij )︱i∈[1,2…m],j∈[1,2…l],C ij ≠0},

[0040] make i=1,2,...,m; j=1,2,...,l; wherein the present invention is particularly suitable for large-scale sparse matrix E-level dimension matrix, that is, the value of m, n, l can be in tens of millions.

[0041] The methods include:

[0042] Step 10, a mapreduce Job completes the transposition matrix A, and outputs the matrix A'; the mapreduce process is as follows:

[0043] Step 11. The map function reads the record position and ik >Constituted key-value pair, and then output the column number k and the corresponding sparse vector partColumnVectorik > form the key...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a large-scale sparse matrix multiplication method based on mapreduce. Suppose that the large-scale sparse matrixes are A and B, and a multiplication matrix of A and B is C. The method includes the following steps: step 10, a mapreduce Job finishes transposing of the matrix A and outputting of a matrix A'; step 20, a transformational matrix B converts a storage mode using the matrix B as a coordinate point into a storage mode of sparse vector, and outputs a matrix B'; step 30, connecting the matrix A' and the matrix B', calculating a product component, obtaining the product component of a column number K on the matrix A and column number K on the matrix B of Cij; step 40, merging the product components, calculating Cij through accumulation of the product components Cij_k. The large-scale sparse matrix multiplication is converted into basic operations of transposition and transformation, connection and merging and the like which are suitable for mapreduce calculation, and the problem of resource limit of a single machine large-scale sparse matrix multiplication is solved.

Description

technical field [0001] The invention relates to a large-scale sparse matrix multiplication operation method based on mapreduce. Background technique [0002] Matrix multiplication is one of the common problems in linear algebra, and many numerical calculation problems include the calculation of matrix multiplication. Therefore, the problem of improving the running speed of the matrix multiplication algorithm has attracted great attention of algorithm researchers for many years. In the research community, there are mainly two ways to achieve the goal: improving the algorithm to reduce the algorithm complexity of matrix multiplication and parallelizing the algorithm of matrix multiplication. [0003] The research results of the first method show that when the size of the matrix is ​​small, the benefits of reducing the algorithm complexity are very obvious. For the matrix multiplication operation of n×n and n×n, the algorithm complexity of the optimal method is close to the lo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/16G06F7/523
Inventor 刘德建陈宏展吴拥民刘飞荣
Owner FUJIAN TQ DIGITAL
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap