A method for multiplication of super-large-scale sparse matrix based on mapreduce framework

A technology of sparse matrix and multiplication operation, applied in the field of matrix multiplication, which can solve the problems of low execution performance, inability to execute, large memory and calculation amount, etc., and achieve the effect of reducing operation steps and time, and reducing requirements

Active Publication Date: 2017-07-21
阿里巴巴(北京)软件服务有限公司
View PDF1 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the prior art CN201310033884.6, a method of converting the large matrix multiplication problem into an operation suitable for mapreduce is proposed to solve the large-scale matrix multiplication operation because the dimension is too large, and the execution performance is low or even impossible to execute in a single machine environment due to resource constraints. The problem
However, this operation requires 4 mapreduce jobs to complete, and still occupies a large amount of memory and calculation. Therefore, how to reduce the amount of calculation and complete the operation of matrix multiplication more quickly and effectively has become an urgent problem to be solved in the existing technology. technical problem

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method for multiplication of super-large-scale sparse matrix based on mapreduce framework

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, only some structures related to the present invention are shown in the drawings but not all structures.

[0020] The present invention is applied to the method of large-scale sparse matrix multiplication operation under the mapreduce framework, that is, to find the matrix C, so that C=A*B, wherein the storage format of A is (i, k, A ik ), the storage format of B is (k, j, B kj ), the storage format of C is (i, j, C ij ), where 1≤i≤m, 1≤k≤n, 1≤j≤l. The whole algorithm is completed by two mapreduce jobs:

[0021] Step 1: The first job, which requires two mappers and one reduce to complete:

[0022] (i) Generate mapp...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

A large-scale sparse matrix multiplication method based on the mapreduce framework. The algorithm is completed by two mapreduce jobs, and the elements of matrix A and matrix B are correctly grouped, so that the elements of the ith column of matrix A are the same as The elements of the kth row of the matrix B enter the same reduce group, and do a product of each element from A and element from B in the group. The present invention only needs 2 mapreduce operations to complete the multiplication of the super-large-scale coefficient matrix, reducing the operation steps and time of the algorithm, and the present invention reduces the requirement for the machine memory, and only needs the machine to store each row of the matrix A with a hashmap conduct.

Description

technical field [0001] This application relates to a matrix multiplication, in particular, to a method for super-large-scale sparse matrix multiplication based on the mapreduce framework. Background technique [0002] Matrix multiplication is one of the common problems in linear algebra, and many numerical calculation problems include the calculation of matrix multiplication. Therefore, the problem of improving the running speed of the matrix multiplication algorithm has attracted great attention of algorithm researchers for many years. In the prior art CN201310033884.6, a method of converting the large matrix multiplication problem into an operation suitable for mapreduce is proposed to solve the large-scale matrix multiplication operation because the dimension is too large, and the execution performance is low or even impossible to execute in a single machine environment due to resource constraints. The problem. However, this operation requires 4 mapreduce jobs to comple...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/16
Inventor 蒋伟姚键潘柏宇卢述奇
Owner 阿里巴巴(北京)软件服务有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap