Implementation method and device of Hadoop submitter based on object storage

A technology for object storage and implementation methods, which is applied to instruments, file/folder operations, calculations, etc., and can solve problems such as low efficiency of file storage mechanisms

Active Publication Date: 2021-10-01
北京星辰天合科技股份有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The main purpose of the present invention is to provide an implementation method and device based on object storage Hadoop submitter, to solve the technical problem of low efficiency of the file storage mechanism in the related art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Implementation method and device of Hadoop submitter based on object storage
  • Implementation method and device of Hadoop submitter based on object storage

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] It should be noted that, in the case of no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and examples.

[0025] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0026] It should be noted that the terms "first...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and a device for implementing a Hadoop submitter based on object storage. The invention includes: reading one or more files, wherein the custom metadata of the files is used to represent the description information of the files; using the file instant combination function to merge one or more files to generate a new file; storing the new file Go to the target directory, where the new file is used as a soft link to point to the corresponding source data file. The present invention solves the technical problem in the related art that the file storage efficiency is low due to the submission protocol adopted in the file storage.

Description

technical field [0001] The invention relates to the field of file storage, in particular to an implementation method and device for an object-based storage Hadoop submitter. Background technique [0002] In related technologies, MapReduce in Hadoop splits the job submitted by the user into multiple single operation tasks (respectively map task and reduce task) and executes these tasks on multiple nodes. After the execution of the task is completed, the execution result will be output Stored in the final result directory through the output commit protocol. Any job-side submitted work will be executed across nodes in the cluster, and may occur outside of critical parts of the job execution. However, unless the output commit protocol requires all tasks to wait for a signal from the job driver, task commits cannot instantiate their output in the final directory, which can be used to promote the output of successful tasks to a state where the job can be submitted, solving specul...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/16G06F16/172G06F16/182
CPCG06F16/16G06F16/172G06F16/182
Inventor 战策张旭明王豪迈胥昕
Owner 北京星辰天合科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap