A data management method and hadoop distributed file system

A distributed file and data management technology, applied in the computer field, can solve problems such as low efficiency, and achieve the effect of improving read/write efficiency

Active Publication Date: 2021-08-31
ZHEJIANG UNIVIEW TECH CO LTD
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But the current HDFS only provides unified access to existing data. For clients that need to write, HDFS still has multiple data centers; in addition, to find a file, in the worst case, you need to find all The NameNode node in the data center is relatively inefficient

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data management method and hadoop distributed file system
  • A data management method and hadoop distributed file system
  • A data management method and hadoop distributed file system

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0039] The embodiment of the present invention provides a Hadoop distributed file system 200, which is applied to the server 100 and used to realize more unified and efficient management of the data center. see figure 2 , is a functional block diagram of the Hadoop distributed file system 200 provided by the embodiment of the present invention. The Hadoop distributed file system 200 includes load balancing nodes 210 , proxy nodes 220 , metadata nodes 230 and data nodes 240 .

[0040] Wherein, the load balancing node 210 is configured to receive a data processing request initiated by a client.

[0041] It should be noted that the data processing request includes, but is not limited to, a write request, a read request, and the like. When the client initiates a write request to the server 100, it indicates that the user wants to write data to the server 100 through the client, which can be understood as new data; when the client initiates a read request to the server 100, it i...

no. 2 example

[0083]The embodiment of the present invention provides a data management method, which is applied to the server 100 and runs on the cluster nodes of the Hadoop distributed file system 200 . It should be noted that the basic principles and technical effects of the data management method provided by this embodiment are the same as those of the above-mentioned embodiments. Corresponding content. see Figure 5 , is a flowchart of a data management method provided by an embodiment of the present invention. This approach to data management includes:

[0084] Step S501: Receive a data processing request initiated by a client through the proxy node 220 .

[0085] see Figure 6 , is a specific flowchart of step S501. This step S501 includes:

[0086] Sub-step S5011: Receive the data processing request initiated by the client through the load balancing node 210.

[0087] It should be noted that the data processing request includes, but is not limited to, a write request, a read r...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

Embodiments of the present invention provide a data management method and a Hadoop distributed file system, which relate to the field of computer technology. The data management method and the Hadoop distributed file system receive a data processing request initiated by a client through an agent node, and determine a metadata node corresponding to the data processing request on the agent node based on a pre-established hash model and the data processing request, Send a data processing request to the metadata node through the proxy node, and finally respond to the data processing request through the metadata node and feed back the message to the client; by unifying the command space of the metadata node, multiple independent metadata nodes are unified as a whole The namespace realizes unified access to data; in addition, metadata nodes corresponding to data processing requests can be quickly queried based on a pre-established hash model, improving read / write efficiency.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a data management method and a Hadoop distributed file system. Background technique [0002] When faced with big data application requirements, when stand-alone performance faces bottlenecks and relational database cluster systems can no longer meet the needs of big data applications, using distributed storage systems to store data and adopting standard user interfaces based on SQL language has become a solution to large-scale structures. An effective way to simplify data retrieval problems. [0003] In the prior art, a Hadoop Distributed File System (Hadoop Distributed File System, HDFS) is usually used to store data files, and the HDFS files and their contents are mapped into a table structure by maintaining a piece of metadata. But the current HDFS only provides unified access to existing data. For clients that need to write, HDFS still has multiple data centers; in additio...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/13G06F16/182
CPCG06F16/137G06F16/182
Inventor 张期莲
Owner ZHEJIANG UNIVIEW TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap