Super-large scale sparse matrix multiplication method based on mapreduce frame

A technology of sparse matrix and multiplication operation, applied in the field of matrix multiplication, which can solve problems such as low execution performance, large memory and calculation amount, and large dimension, and achieve the effect of reducing requirements, operation steps and time

Active Publication Date: 2015-03-25
阿里巴巴(北京)软件服务有限公司
View PDF2 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the prior art CN201310033884.6, a method of converting the large matrix multiplication problem into an operation suitable for mapreduce is proposed to solve the large-scale matrix multiplication operation because the dimension is too large, and the execution performance is low or even impossible to execute in a single machine environment due to resource constraints. The problem
However, this operation requires 4 mapreduce jobs to complete, and still occupies a large amount of memory and calculation. Therefore, how to reduce the amount of calculation and complete the operation of matrix multiplication more quickly and effectively has become an urgent problem to be solved in the existing technology. technical problem

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Super-large scale sparse matrix multiplication method based on mapreduce frame

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, only some structures related to the present invention are shown in the drawings but not all structures.

[0020] The present invention is applied to the method of large-scale sparse matrix multiplication operation under the mapreduce framework, that is, to find the matrix C, so that C=A*B, wherein the storage format of A is (i, k, A ik ), the storage format of B is (k, j, B kj ), the storage format of C is (i,j,C ij ), where 1≤i≤m, 1≤k≤n, 1≤j≤l. The whole algorithm is completed by two mapreduce jobs:

[0021] Step 1: The first job, which requires two mappers and one reduce to complete:

[0022] (i) Generate mapper...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

Disclosed is a super-large scale sparse matrix multiplication method based on a mapreduce frame. The algorithm is realized through two mapreduce jobs, elements in a matrix A and elements in a matrix B are grouped correctly to enable the elements in the i row of the matrix A and the elements in the k line of the matrix B to belong to a group of the same reduce, and multiplication is conducted on each element from the matrix A and each element from the matrix B in the group. Due to the fact that super-large scale sparse matrix multiplication can be achieved through two mapreduce jobs, the number of steps of the algorithm is reduced, time is shortened, and the requirement for the memory of a machine is reduced as long as each row of the matrix A can be stored by the machine with a hashmap.

Description

technical field [0001] This application relates to a matrix multiplication, in particular, to a method for super-large-scale sparse matrix multiplication based on the mapreduce framework. Background technique [0002] Matrix multiplication is one of the common problems in linear algebra, and many numerical calculation problems include the calculation of matrix multiplication. Therefore, the problem of improving the running speed of the matrix multiplication algorithm has attracted great attention of algorithm researchers for many years. In the prior art CN201310033884.6, a method of converting the large matrix multiplication problem into an operation suitable for mapreduce is proposed to solve the large-scale matrix multiplication operation because the dimension is too large, and the execution performance is low or even impossible to execute in a single machine environment due to resource constraints. The problem. However, this operation requires 4 mapreduce jobs to compl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/16
Inventor 蒋伟姚键潘柏宇卢述奇
Owner 阿里巴巴(北京)软件服务有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap