Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and device for processing data

A technology for processing data and data, which is applied in the field of data processing and can solve problems such as high chain building overhead

Inactive Publication Date: 2020-04-28
HUAWEI TECH CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the replication process in the Shuffle stage of the MapReduce system still needs to establish a large number of transmission links, and the network link establishment costs a lot

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for processing data
  • Method and device for processing data
  • Method and device for processing data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0031] The data processing method provided by the embodiment of the present invention can be applied to any system that performs parallel computing on data. The distributed file system (Distributed File System, DFS) in the embodiment of the present invention can be Hadoop distributed file system (Hadoop Distributed File System, HDFS), can be network file system (NetworkFile System, be called for short NFS), can be The Google File System (Google File Sy...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data processing method, and the method is executed in a system which comprises at least one calculation node and at least one simplifying node. Each calculation node is provided with K transmission links, and K simplifying tasks are operated on the simplifying node. The K transmission links are corresponding to K simplifying tasks in a one-to-one manner, and the K simplifying tasks are corresponding to K data formats in a one-to-one manner. Each simplifying task is used for simplifying the data in the corresponding data format, wherein K is not less than two. The method comprises the steps that the calculation nodes obtain to-be-processed data, wherein the to-be-processed data is generated by at least two calculation tasks operated in the calculation nodes, and comprises at least two types of subdata with different data formats; the calculation nodes transmits the first subdata according to the data format of the first subdata, wherein the first transmission link for transmitting the first subdata is corresponding to the data format of the first subdata.

Description

technical field [0001] The present invention relates to the field of data processing, and more particularly, to a method and device for processing data. Background technique [0002] In the parallel computing technology of big data processing, the MapReduce system plays a very important role. The data processing flow of MapReduce can be divided into two phases: mapping Map phase and simplifying Reduce phase. Generally, the process from the output of the Map to the input of the Reduce is also called the Shuffle stage. [0003] figure 1 It is a working principle diagram of the MapReduce system in the prior art. like figure 1 As shown, in the MapReduce system, a job (Job) will be divided into a large number of tasks to execute in parallel. The first is the Map stage. Each Map task will read a piece of data (that is, a fragment) from the distributed file system (Hadoop Distributed File System) as input, and the output will be stored in the memory buffer (buffer) after being...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08
CPCH04L67/10
Inventor 王朱珍
Owner HUAWEI TECH CO LTD