Sample data/parameter transmission method in big data distributed calculation process
A distributed computing and sample data technology, applied in the field of sample data/parameter communication, can solve the communication bottleneck of TCP channel cluster data transmission, the application program cannot respond in time, and the data copy overhead time is not significant, etc. Effect
Pending Publication Date: 2022-03-01
北京瀚海云星科技有限公司 +1
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
In a low-speed network (the network protocol stack is usually the aforementioned TCP / IP / Ethernet), its data copy overhead is not significant relative to the time of network hardware transmission. However, when the general bandwidth reaches 40Gb / s- When 300Gb / s network equipment is applied to a commercial cluster for big data processing, the TCP channel becomes the communication bottleneck for cluster data transmission
In addition, even if RDMA is considered to be more efficient to replace the traditional TCP protocol, and it is assumed that it can be implemented in engineering practice, considering the characteristics of the RDMA communication bypass kernel, even if it is directly accessing related data objects, receiving sample data The application on one end of the / parameter also fails to respond in a timely manner
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0023] The technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings of the embodiments of the present invention. Apparently, the described embodiments are only part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0024] In the process of big data distributed computing, it is undoubtedly necessary to generate a large amount of cross-terminal communication, which may not only involve the cross-terminal communication of sample data, but also involve the calculation results of the intermediate process in the distributed computing process (generally expressed as a certain format parameter form) communication. Taking the MapReduce distributed computing model as an example,...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a sample data / parameter transmission method in a big data distributed calculation process. The invention relates to a sample data / parameter transmission method in a big data distributed computing process and a computer readable storage medium related to a big data distributed computing system based on the sample data / parameter transmission method. An efficient RDMA channel is provided for sample data / parameters between a task scheduling engine and a task calculation engine or between task calculation engines by providing efficient and reusable RDMA channel connection and the like, and a completion notification message is transmitted through a TCP channel after transmission is completed, so that the efficiency of the distributed calculation system is further improved.
Description
technical field [0001] The present invention relates to the technical field of communication of sample data / parameters in a big data distributed computing process, and in particular, to a method for transferring sample data / parameters in a big data distributed computing process. Background technique [0002] Today's society is a rapidly developing society, with advanced technology, information flow, people's communication is getting closer and closer, and life is becoming more and more convenient. Big data is the product of this high-tech era. Based on this, some people even assert that the future era will not be the IT era, but the DT era, and DT is Data Technology. If the data is compared to a coal mine, then, just as coal is divided into coking coal, fat coal, lean coal, and lean coal according to its degree of combustion, coal mines are divided into open-pit coal mines and underground coal mines according to their mining difficulty. (Obviously, the cost of mining raw co...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/54H04L67/14H04L69/163
CPCG06F9/541G06F9/546G06F9/4806H04L67/14H04L69/163
Inventor 李杨张翔宇张曼妮孙军欢
Owner 北京瀚海云星科技有限公司