Method and apparatus for realizing distributed pipeline in distributed file system
A distributed file and distributed technology, applied in the computer field, can solve the problems of complex system architecture, limited scalability, and high use cost of application systems
Active Publication Date: 2017-04-26
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF3 Cites 5 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] However, the implementation of the above-mentioned distributed pipeline has the following problems in data transmission / publishing / subscribing: the system must also consider complex issues such as availability, transmission efficiency, data reliability and consistency, which makes the system architecture very complicated , the flexibility is limited, the scale is small, and the scalability is limited; in the case of distributed inter-process communication, due to the complexity of the technology itself and the need to provide a special programming interface, the cost of using the application system is relatively high
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0034] The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, rather than to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.
[0035] It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.
[0036] figure 1 It shows an exemplary system architecture 100 to which embodiments of the method for implementing a distributed pipeline in a distributed file system or the embodiment of an apparatus for implementing a distributed pipeline in a distributed file system can be...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a method and an apparatus for realizing a distributed pipeline in a distributed file system. The method comprises the steps that a writing interface receives a data writing request of a writing session and sends the data writing request comprising a writing length to a meta-information manager; the meta-information manager notifies the writing interface to send a forward cutoff notification to a cutoff interface in response to the facts that the sum of the writing length and the existing length of a pipeline file exceeds a limited threshold of the pipeline and a preset data clearing policy in a pipeline file attribute allows automatic clearing; the cutoff interface sends a cutoff request to the meta-information manager in response to the fact that the forward cutoff notification is received, wherein the cutoff request comprises a cutoff offset; and the meta-information manager compares whether the cutoff offset in the cutoff request is greater than an initial offset of the pipeline file or not in response to the fact that the cutoff request is received, and if yes, the meta-information manager notifies a data server corresponding to a pipeline file copy to execute forward cutoff operation. Therefore, the writing efficiency and the stability of the distributed pipeline are remarkably improved.
Description
technical field [0001] The present application relates to the field of computer technology, specifically to the field of Internet technology, and in particular to a method and device for realizing distributed pipelines in a distributed file system. Background technique [0002] All mainstream operating systems support an inter-process communication method called "pipelines". The principle is that the writing end pushes data into the pipeline, while the reading end obtains data from the pipeline, and the flow of data follows the first-in-first-out rule. [0003] In the case of a distributed file system, such as data transmission / publish / subscribe, or as a distributed inter-process communication infrastructure for a cluster operating system, pipelines still have great application requirements: in data transmission / publishing In the case of / subscription: the upper layer system of the distributed file system is based on distributed pipelines, which can realize data transmissio...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/17G06F16/183
Inventor 郭波
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD