Data reading method, device, electronic device and readable storage medium

A data reading and data technology, applied in the field of data processing, can solve the problems of low reading and writing efficiency and occupying bandwidth, etc.

Active Publication Date: 2020-10-27
BEIJING SANKUAI ONLINE TECH CO LTD
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

HDFS usually occupies a large amount of bandwidth in the computer room when reading and writing data, resulting in low read and write efficiency.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data reading method, device, electronic device and readable storage medium
  • Data reading method, device, electronic device and readable storage medium
  • Data reading method, device, electronic device and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0073] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

[0074] refer to figure 1 , figure 1 It is a flowchart of a data reading method proposed by an embodiment of the present application, and the data reading method can be applied to the file distribution system HDFS.

[0075] refer to Figure 2A , Figure 2A It is a structural block diagram of an HDFS system, which includes: a management node NN, multiple data storage nodes DN, and multiple client clients. Wherein, the client client may be located in the same host as the da...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a data reading method and device, electronic equipment and a readable storage medium. The method comprises the following steps: setting a machine room identifier for each data storage node DN in the HDFS, and setting a copy distribution attribute for to-be-backed-up data; determining a target machine room corresponding to the machine room identifier, and selecting a data storage node DN with the copy number from the target machine room; storing the data to be backed up in the selected DN; when a data reading request which is sent by a client and aims attarget data is received through an NN, sending DN identifiers of all DNs storing the target data to the client through the NN; determining whether the machine room identifier of each DN storing the target data is the same as the machine room identifier of the client through the client; and reading the target data from the target DN under the condition that the client determines that all the DNs comprise the target DN of which the machine room identifier is the same as the machine room identifier of the client.

Description

technical field [0001] The embodiments of the present application relate to the technical field of data processing, and in particular to a data reading method, device, electronic device, and readable storage medium. Background technique [0002] The file distributed system infrastructure Hadoop includes two important service systems, namely the file distributed system HDFS (Hadoop Distributed File System) and the resource management system YARN (Yet Another Recourse Negotiator). Among them, HDFS is based on a master-slave structure (master / slaver), including at least one management node NameNode (NN) as a master and multiple data storage nodes DataNode (DN) as slavers. Files written to HDFS are first divided into several file blocks, each file block is backed up into multiple copies, and each copy is stored in a DataNode. When HDFS reads and writes data, it usually occupies a large amount of bandwidth in the computer room, resulting in low read and write efficiency. Conte...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08G06F11/14
CPCG06F11/1448G06F11/1451H04L67/1097
Inventor 刘俊宏万畅李建斌张世龙
Owner BEIJING SANKUAI ONLINE TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap