Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and system for realizing mutual access of hadoop file system hdfs and object storage s3 data

A technology of object storage and file system, which is applied in the direction of file system, electronic digital data processing, digital data information retrieval, etc., to achieve the effect of expanding the scope of the field and promoting development

Pending Publication Date: 2020-09-25
SHENZHEN VIRTUAL CLUSTERS INFORMATION TECH
View PDF8 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] However, hdfs and s3 are two completely different styles and use completely different file access and reading interfaces. There are isolation barriers for mutual reading and access of files between different file systems. Therefore, there are problems in the existing technology and need to be further improved.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for realizing mutual access of hadoop file system hdfs and object storage s3 data
  • Method and system for realizing mutual access of hadoop file system hdfs and object storage s3 data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0023] refer to figure 1 , the present invention provides a method for realizing data exchange between Hadoop file system hdfs and object storage s3, comprising the steps of:

[0024] S11, configure the hadoop big data environment containing the file system hdfs, and the distributed storage software ceph environment containing the object storage s3; both sides of the hadoop big data environment and the distributed storage software ceph environment communicate with the ceph-mon node through the namenode node;

[0025] S12. Connect to the file system hdfs through the namenode node, and connect to the object storage s3 through the ceph-mon node;

[0026] S13. Obtain an external data access instruction;

[0027] S14. Perform data access between ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and a system for realizing mutual access of hadoop file system hdfs and object storage s3 data. The method comprises the following steps: configuring a hadoop big dataenvironment containing the file system hdfs and a distributed storage software ceph environment containing the object storage s3, wherein both the hadoop big data environment and the distributed storage software ceph environment communicate with the ceph-mon node through the namenode node; docking a file system hdfs through a namenode node, and docking an object storage s3 through a ceph-mon node; obtaining an external data access instruction; and performing data access between the corresponding file system hdfs and the object storage s3 according to the data access instruction. According tothe invention, the isolation barrier that files in different file systems are mutually read and accessed is broken through; mutual access reading and coexistence of data of the hadoop file system hdfsand the object storage s3 are realized, the advantages of the hdfs and the object storage s3 complement each other to jointly promote development of big data, and the application field range of the big data is expanded.

Description

technical field [0001] The invention relates to the field of computer data interaction, in particular to a method and system for realizing data mutual access between Hadoop file system HDFS and object storage S3. Background technique [0002] Hadoop is an open source big data framework developed by the Apache Foundation. It is a software platform for developing and operating large-scale data processing. Its three core components are distributed file system hdfs, job scheduling and cluster resource management framework yarn, and distributed computing programming framework MapReduce. They respectively solve the three core problems of the Hadoop big data framework, how to solve the storage of massive data, how to solve the scheduling of computing resource tasks, and how to solve the computing tasks of massive data. Among them, hdfs is a distributed file system specially designed based on streaming data access and processing massive large files and does not strictly abide by th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/16G06F16/182
CPCG06F16/164G06F16/182Y02D10/00
Inventor 张涛刘毅林涛睿林鹏程
Owner SHENZHEN VIRTUAL CLUSTERS INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com