Unlock instant, AI-driven research and patent intelligence for your innovation.

Code distributed calculation method based on Partition structure

A distributed computing and coding technology, applied in the field of network communication, which can solve the problems of inability to carry out practical applications, large number of input files and output functions, etc.

Active Publication Date: 2021-05-07
GUANGXI NORMAL UNIV
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In S.Li, M.A. Maddah-Ali, Q. Yu, and A.S. Avestimehr, “Afundamental tradeoff between computation and communication in distributed computing,” IEEE Trans. Inf. Theory, vol.64, no.1, pp.109–128, In the scheme proposed by Jan.2018., the number of input files and output functions is too large to be practically applied, so we hope to sacrifice part of the communication load L to reduce the number of input files N and the number of output functions Q

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Code distributed calculation method based on Partition structure
  • Code distributed calculation method based on Partition structure
  • Code distributed calculation method based on Partition structure

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0045] For the number s of calculations of each output function, most of the literature is researched on the case of s=1, and only a few articles mention the case of s>1, and in this case, the focus is on Solve the problem of how to reduce the number N of input files and the number Q of output functions when s>1.

[0046] An encoding distributed computing method based on the Partition structure. In encoding distributed computing, for N input files, K distributed computing nodes are used to calculate the values ​​of Q output functions. The entire calculation process is carried out in three stages: In Map stage, Shuffle stage and Reduce stage, K nodes jointly calculate Q functions s times distributed computing method, including the following steps:

[0047] A.Map stage:

[0048] Each node passes the Map function n∈{1,...,N}, T∈N calculates the local stored files to generate intermediate values ​​v corresponding to all Q functions q,n :

[0049]

[0050] in n∈{1,...,N} ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a coding distributed calculation method based on a Partition structure, in coding distributed calculation, K distributed calculation nodes are used for calculating values of Q output functions for N input files, the whole calculation process is carried out in three stages: a Map stage, a Shuffle stage and a Reduce stage, and the K nodes are used for jointly calculating the Q functions for s times. The method comprises the following steps: A, a Map stage; b, a Shuffle stage; and C, a Reduce stage. The communication load realized by the method is almost approximately equal to the optimal communication load, and on the basis, the minimum requirements of the number N of required input files and the number Q of required output functions are obviously reduced compared with the scheme of Ali.

Description

technical field [0001] The invention belongs to network communication technology, in particular to a coding distributed computing method based on a Partition structure. Background technique [0002] Driven by the rapid development of machine learning and data science, modern computing paradigms have shifted from traditional single-processor systems to large-scale distributed computing systems, and the MapReduce framework is a popular framework for distributed computing. Distributed computing shows its great advantages when dealing with large-scale data, so it has become a popular research direction in the past two years. [0003] Distributed computing is a computing method, which is relative to centralized computing. Distributed computing refers to the distribution of centralized computing on one device to distributed computing on multiple devices in the network, thus speeding up the computing process. It can handle large-scale data analysis tasks, such as machine learning...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L1/00
CPCH04L1/0076H04L1/0057
Inventor 钟逸云蒋静曲凌晓周玲玲王文汉董洪岩姬生鹏
Owner GUANGXI NORMAL UNIV