HDFS API calling method and device, equipment and storage medium

A technology of preset data and target files, applied in the field of data communication, can solve problems such as HDFS API cannot be called efficiently, achieve the effect of asynchronous transmission optimization and improve data transmission efficiency

Pending Publication Date: 2021-05-07
SHENZHEN IPANEL TECH LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the present invention provides a HDFS API calling method, device, equipment and storage medium to solve the technical problem that the current HDFS API cannot be efficiently called in a high concurrency environment

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • HDFS API calling method and device, equipment and storage medium
  • HDFS API calling method and device, equipment and storage medium
  • HDFS API calling method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0052] OpenResty is a high-performance web platform based on Nginx and Lua. It integrates a large number of excellent Lua libraries, third-party modules and most dependencies. It can be used to easily build dynamic Web applications, Web services and dynamic gateways.

[0053] OpenResty makes full use of Nginx's non-blocking I / O model, while Nginx adopts the master-worker model. A master process manages multiple worker processes. Basic event processing is place...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

According to the HDFS API calling method and device, the equipment, and the storage medium, the server receives the file reading request sent by the client through multiple threads, the interface docking module is used for docking the HDFS API to obtain the target file identifier and send the target file identifier to the client, support is provided for a distributed and high-concurrency scene, and efficient calling of the HDFS API in the high-concurrency environment is achieved; a data reading request which is sent by a client and carries a preset deviation value, a preset data block size and a target file identifier is received; part of data of the target file is obtained according to the data reading request; a corresponding memory sharing buffer area is distributed for storage; the stored part of data is sent to the client according to a queuing sequence; and the data reading request sent by the client is received again until the file reading completion indication sent by the client is received, so that asynchronous transmission of the large file is optimized on the basis of efficient calling of the HDFS API, and the data transmission efficiency is improved.

Description

technical field [0001] The present invention relates to the technical field of data communication, and more specifically, relates to an HDFS API calling method, device, equipment and storage medium. Background technique [0002] Hadoop Distributed File System (HDFS, Hadoop Distributed File System) is an implementation of the Hadoop abstract file system. The Hadoop abstract file system can be integrated with local systems, Amazon S3, etc., and can even be operated through the web protocol (webhsfs). HDFS files are distributed on cluster machines, and copies are provided for fault tolerance and reliability assurance. [0003] At present, the Java / C API provided by HDFS is a return handle Handle interface, that is, a blocking interface. For example, if you need to read 10M data, you must wait for all the data to be read before returning, but this will lead to calling Threads are blocked, and nginx is a single-threaded working model in a process, which cannot complete high-con...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/182G06F16/172G06F16/13H04L12/26H04L29/06H04L29/08
CPCG06F16/13G06F16/172G06F16/182H04L43/10H04L67/1097H04L69/162
Inventor 徐佳宏陈华兵
Owner SHENZHEN IPANEL TECH LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap