Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and device for distributed system

A distributed system and data processing technology, applied in the field of distributed systems, can solve problems such as inability to fully utilize computing resources

Inactive Publication Date: 2017-07-14
INT BUSINESS MASCH CORP
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

It can be seen that when a large block is used, only 200 map tasks are used in the second round, and the available computing resources cannot be fully utilized.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device for distributed system
  • Data processing method and device for distributed system
  • Data processing method and device for distributed system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0018] Preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0019] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, that is: it can be complete hardware, it can also be complete software (including firmware, resident software, microcode, etc.), and it can also be a combination of hardware and software. Called a "circuit", "module" or "s...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a data processing method and device for a distributed system. In one embodiment, the technical solution includes: in response to a request for writing a data file, storing multiple copies of the data file on the multiple secondary storage nodes, each of which is divided into data files of the same size block, where the data blocks divided by at least two copies are of different sizes; and the distribution information of the multiple copies is stored. By adopting the technical solution of the present application, when storing multiple backups of data files in a distributed system, the advantages brought by storage with different data block sizes can be combined.

Description

technical field [0001] The present invention relates to a distributed system, and more specifically, to a data processing method and device for a distributed system. Background technique [0002] HDFS (Hadoop Distributed File System) is a typical example of a distributed file system. Next, take HDFS as an example to illustrate the characteristics and shortcomings of the existing distributed file system. [0003] HDFS adopts a master-slave architecture (Master / Slave). An HDFS cluster includes a name node (NameNode) and multiple data nodes (DataNode). The name node is the main storage node, which manages the namespace of the entire file system and the access requests of clients. In the name node, you can perform operations such as opening / closing / renaming files or directories. The data node is a slave storage node, which is used to receive read / write requests from the client, and at the same time complete the establishment, deletion and replication of file blocks according t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/178G06F11/2094G06F16/122G06F16/182
Inventor 陈冠诚李严李欣滕启明李剑
Owner INT BUSINESS MASCH CORP