Map reduce computing method under internet environment

A computing method and environment technology, applied in the direction of resource allocation, multi-programming devices, etc., can solve the problem of limited application range of voluntary computing mode, achieve the effect of realizing large-scale scalability system and reducing difficulty

Inactive Publication Date: 2013-05-22
贺海武 +1
View PDF0 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This determines that the scope of application of the voluntary computing model is limited, not all applications can run on the voluntary computing platform

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Map reduce computing method under internet environment
  • Map reduce computing method under internet environment
  • Map reduce computing method under internet environment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The embodiment of the present invention provides a method and a system for performing MapReduce calculation using a desktop PC under the Internet environment. Embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0038] Such as figure 2 As shown, the MapReduce system in the Internet environment mainly includes the following three different types of nodes.

[0039] (1) Client (Master): Submit the parallel processing job written by the user to the service node, monitor the job status through the service node, and automatically obtain the final execution result.

[0040] (2) Service node (server): manages all work nodes, and automatically decomposes user jobs into Map tasks and Reduce tasks, and schedules tasks to work nodes for execution. Specifically, it includes four service components:

[0041] ① Metadata server: manage and store metadata;

[0042] ② Data storage server: HTTP, FTP server, or Tracker...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a map reduce computing method under an internet environment. The method utilizes available computation and storage resources of an internet desktop personal computer (PC) to carry out map reduce computation. The desktop PC is added to a system in a volunteer form and forms a work node. A user as a master uploads to-be-processed data to a server. The server schedules map and reduce tasks to the work node for execution. When all the tasks are finished, final results are fed back to the user automatically. On the basis of metadata management, a data driving way is adopted, task scheduling is converted into data scheduling, the problems that node dynamic fails easily and influence is caused are solved, and therefore a fault-tolerant map reduce computing model and the method are simple and effective under the internet environment. Due to the method, an existing desktop PC is utilized by campuses, enterprises and governments to construct a map reduce computation platform with private purposes for carrying out mass data analysis, or the method is used for improving a volunteer computing system to support the map reduce computing model and lowering difficulty of application development of a volunteer commutating system.

Description

technical field [0001] The invention belongs to the field of distributed computing models, and in particular relates to a new MapReduce computing method and system in the Internet environment, and a Map and Reduce task scheduling method in the system. Background technique [0002] Following high-performance cluster computing and grid computing, cloud computing is receiving strong attention in the field of large-scale computing. As the core technology of cloud computing, MapReduce has also received extensive attention. MapReduce is a parallel programming model proposed by Google in 2004 to process large-scale data sets to simplify the programming of distributed systems. Application programmers only need to focus on the application itself, while complex processes related to cluster processing, including reliability, scalability, task parallelization, data distribution storage, load balancing, and node communication, are shielded, and users do not need to You need to care abo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50
Inventor 贺海武唐兵
Owner 贺海武
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap