MapReduce-oriented data transmission method and device, equipment and storage medium
A data transmission method and data transmission technology, which are applied in the fields of MapReduce-oriented data transmission methods, devices, equipment and storage media, can solve problems such as insufficient data transmission performance, improve data transmission efficiency, improve data computing performance and efficiency, Realize the effect of decoupling communication and computing
Inactive Publication Date: 2021-11-02
ORIENTAL MIND (WUHAN) COMPUTING TECH CO LTD
View PDF0 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0006] The main purpose of the present invention is to provide a MapReduce-oriented data transmission method, device, equipment and storage medium, aiming to solve the technical problem of insufficient data transmission performance of the prior art shuffling mechanism
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0049] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.
[0050] refer to figure 1 , figure 1 It is a schematic structural diagram of a MapReduce-oriented data transmission device in the hardware operating environment involved in the solution of the embodiment of the present invention.
[0051] Such as figure 1 As shown, the MapReduce-oriented data transmission device may include: a processor 1001 , such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless in...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a MapReduce-oriented data transmission method and device, equipment and a storage medium, and solves the technical problem of low data transmission efficiency based on Ethernet and TCP / IP (Transmission Control Protocol / Internet Protocol) under a MapReduce storage and calculation separation architecture. The method comprises the following steps: searching a corresponding data fragment in a first memory buffer area of a fragment in a preset RDMA communication memory on a node according to first meta-information in a received data pulling request, generating second meta-information from the queried data fragment and a corresponding storage position, and feeding back the second meta-information to a Reduce process on a Reduce node, so that the Reduce process allocates a second memory buffer area in a preset RDMA communication memory on a Reduce node according to the second meta-information, and pulling the data fragments from the first memory buffer area to the second memory buffer area according to the second meta-information.
Description
technical field [0001] The invention relates to the technical field of computer communication, in particular to a MapReduce-oriented data transmission method, device, equipment and storage medium. Background technique [0002] In recent years, big data analysis and processing technology has been widely used in industries represented by the Internet. Among many typical data analysis and processing methods, the MapReduce computing model is the most popular. MapReduce is a programming model for parallel computing of large-scale data sets. It has strong versatility and scalability, and can well meet the needs of multiple algorithms and various types of data executed on computing resources of different scales. need. [0003] The calculation process of the MapReduce computing model is divided into two stages: Map and Reduce. In the Map stage, the model completes the classification and organization of data according to the fields that users care about; in the Reduce stage, the m...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50G06F16/182G06F16/2458
CPCG06F9/5072G06F16/182G06F16/2471
Inventor 林健洪志刚
Owner ORIENTAL MIND (WUHAN) COMPUTING TECH CO LTD