File migration method and file migration equipment of Map Reduce distributed system

A distributed system and file migration technology, applied in the direction of program startup/switching, multi-program device, etc., can solve the problem of low migration efficiency, achieve the effect of improving migration efficiency and shortening migration time

Active Publication Date: 2013-06-26
BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
View PDF5 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, in the file migration process of the MapReduce distributed system, the file is used as the unit to migrate in the same task, and the migration efficiency is not high.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • File migration method and file migration equipment of Map Reduce distributed system
  • File migration method and file migration equipment of Map Reduce distributed system
  • File migration method and file migration equipment of Map Reduce distributed system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0039] In addition, the term "and / or" in this article is only an association relationship describing associated objects, which means that there may be three relationships, for example, A and / or B, which may mean: A exists alone, A and B exist at the same time, There are three cases of B alone. In addition, the character " / " in this article genera...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a file migration method of a Map Reduce distributed system and file migration equipment of the Map Reduce distributed system. Migration operation which is used for migrating a target file is started. The migration operation at least comprises a first Map task and a second Map task which are executed in a parallel mode, and a Reduce task which corresponds to the first Map task and the second Map task so that in the Reduce task, metadata, in the target Map Reduce distributed system, of the target file can be generated. Due to the fact that the migration task which enables the target file to be migrated at least comprises the first Map task and the second Map task, and the first Map task and the second Map task are executed in the parallel mode, migration time of the target file can be shortened, and accordingly migration efficiency of the target is improved.

Description

【Technical field】 [0001] The invention relates to file migration technology, in particular to a file migration method and equipment for a MapReduce distributed system. 【Background technique】 [0002] In recent years, with the rapid development of broadband network technology and parallel computing theory, a more simplified distributed system, the MapReduce (MapReduce) distributed system, emerged to provide services for various applications, for example, for search engines Provide services. In the MapReduce distributed system, it can also become a MapReduce distributed cluster, for example, in the Hadoop system, a data processing process is called a job (Job). After the Job is submitted, the data to be processed is divided into N parts, each to be processed The data is processed by a mapping (Map) task. The Map task runs on a node device in the MapReduce distributed system. One or more Map tasks can be run on a node device; the output results of all Map tasks are aggregated ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48
Inventor 潘瑾瑜
Owner BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap