Hadoop cluster system and data processing method

A Hadoop cluster and data technology, applied in the field of distributed file systems, can solve problems such as the overall system service bottleneck, and achieve the effect of improving security, improving data throughput, and enhancing data processing capabilities

Active Publication Date: 2016-08-03
HANDAN BRANCH OF CHINA MOBILE GRP HEBEI COMPANYLIMITED
View PDF2 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In MapReduce, the JobTracker is also a single point. When there are a large number of client co

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hadoop cluster system and data processing method
  • Hadoop cluster system and data processing method
  • Hadoop cluster system and data processing method

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0053] Embodiment one

[0054] An embodiment of the present invention provides a Hadoop cluster system. figure 2 It is a schematic diagram of the composition structure of the Hadoop cluster system of the embodiment of the present invention; as figure 2 As shown, the system includes: a gateway node 24, at least two master nodes 22 and at least two slave nodes 23; the gateway node 24, the at least two master nodes 22 and the at least two slave nodes 23 A broadcast link and a data transmission link are respectively established between every two nodes; where,

[0055] The gateway node 24 is configured to broadcast the service request to the at least two master nodes 22 through the broadcast link after receiving the service request from the client 21, and the master node 22 that first receives the response signaling As the first master node 22; also used to establish the connection between the client 21 and the first master node 22;

[0056] The master node 22 is configured to...

Example Embodiment

[0081] Embodiment two

[0082] The embodiment of the present invention also provides a data processing method, and the data method is applied to the Hadoop cluster system described in the first embodiment. The Hadoop cluster system includes: a gateway node, at least two master nodes and at least two slave nodes; image 3 It is a schematic flow diagram of a data processing method in an embodiment of the present invention; as image 3 As shown, the data processing method includes:

[0083] Step 301: After receiving the service request from the client, the gateway node broadcasts the service request to the at least two master nodes through a broadcast link.

[0084] In this embodiment, the service request may include multiple types, and specifically may be a data storage request, a data deletion request, or a data calculation request. Correspondingly, broadcasting the service request to the at least two master nodes through a broadcast link includes: broadcasting the data stor...

Example Embodiment

[0110] Embodiment Three

[0111] An embodiment of the present invention provides a data processing method. Figure 4 It is a first detailed schematic flow chart of the data processing method of the embodiment of the present invention; as Figure 4 As shown, the data processing method includes:

[0112] Step 1001: The client sends out a data storage request, and the data storage request includes file name information and capacity requirements.

[0113] Here, the client sends the data storage request through a data transmission link.

[0114] Step 1002: After receiving the data storage request, the gateway node sends the data storage request to each master node in the form of broadcast signaling; Figure 4 Only masternode 1, masternode 2, and masternode 3 are listed in , not limited to the above three masternodes.

[0115] Step 1003: Each master node judges whether there is a slave node satisfying the data storage request according to the remaining storage space of each slav...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the present invention discloses a Hadoop cluster system and a data processing method; the system includes: a gateway node, at least two master nodes and at least two slave nodes; the gateway node is configured to receive a service request from a client , broadcasting the service request to the at least two master nodes through a broadcast link, and using the master node that received the response signaling first as the first master node; and also used to establish the relationship between the client and the first A connection between master nodes; the master node is configured to judge whether the service request is satisfied according to its own resource conditions after receiving the service request broadcast by the gateway node; determine that its own resource conditions satisfy the service request After the request, send a response signaling to the gateway node; it is also used to allocate a slave node that meets the requirements for the service request of the client after establishing a connection with the client through the gateway node, and instruct the slave node to execute The service request.

Description

technical field [0001] The invention relates to the field of distributed file systems, in particular to a Hadoop cluster system and a data processing method. Background technique [0002] Hadoop architecture is widely used as an important architecture of cloud computing. Hadoop consists of two main parts: Distributed File System (HDFS, Hadoop Distribute File System) and MapReduce. Among them, HDFS mainly provides storage for massive data; while MapReduce is mainly used for computing massive data. figure 1 It is a schematic diagram of the architecture of the distributed file system; such as figure 1 As shown, the distributed file system includes: a client (Client) 11 , a name node (NameNode) 12 and multiple data nodes (Datanode) 13 . The distributed file system divides the data to be stored into blocks, stores the file blocks on a certain data node (Datanode) 13, and stores multiple backups of the file blocks in other different data nodes (Datanode) 13 in order to Improve...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30H04L29/08
Inventor 李秀清张聚广苏彦志曹英卓李清铎
Owner HANDAN BRANCH OF CHINA MOBILE GRP HEBEI COMPANYLIMITED
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap