A data processing method and device for a distributed memory computing engine cluster
By employing a remote memory direct access module and the CXL protocol in a distributed memory computing engine cluster, the data transmission process is optimized, solving the problem of poor data transmission and processing efficiency in existing technologies, and achieving more efficient data processing and improved cluster performance.
Patent Information
- Application Number
- CN202411206368.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The data transmission and processing efficiency of existing distributed memory computing engine clusters is poor, mainly due to the limitations of network transmission and disk operations during the Shuffle process.
By employing a remote direct memory access module and the CXL protocol, the data transfer process is optimized by directly accessing the memory extension module between the mapping task node and the simplification task node. This includes storing result data during the mapping task phase, querying and retrieving result status objects during the simplification task phase, and using the remote direct memory access module for data transfer.
It improves the data transmission and processing efficiency of the distributed memory computing engine cluster, thereby enhancing cluster performance.
Smart Images

Figure CN118708368B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device for a distributed memory computing engine cluster, and also to an electronic device, a non-transitory computer-readable storage medium and a computer program product. Background Art
[0002] With the rapid development of computer technology, the application of distributed memory computing engine clusters is becoming more and more widespread. Spark applications (i.e., applications) of distributed memory computing engine clusters are usually composed of a driver and several jobs during operation. During the operation process, jobs are usually divided into directed acyclic graphs (DAGs) according to different processing stages of resilient distributed datasets (RDDs). The DAG scheduling stage divides a job into multiple stages. The upstream stage performs data mapping tasks (i.e., Map tasks) and the downstream stage performs data simplification tasks (i.e., Reduce tasks), which is similar to the distributed computing framework MapReduce. Shuffle is a bridge connecting data mapping task nodes (i.e., Map task nodes) and data simplification task nodes (i.e., Reduce task nodes). It maps the output of Map task nodes to the input of Reduce task nodes. However, the calculations of Reduce task nodes and Map task nodes are usually not on the same server, which requires the Shuffle stage to usually require cross-network and some disk read and write operations. Therefore, the performance of Shuffle directly affects the performance and throughput of the entire program. Currently, the communication between distributed memory computing engine clusters is conventional Ethernet, and the network transmission uses the TCP / IP protocol. During the operation of the distributed memory computing engine cluster, the shuffle process of different map tasks and reduce tasks in the cluster uses conventional Ethernet connections, and its performance and throughput are affected by the Ethernet speed, making the data transmission and processing efficiency of the distributed memory computing engine cluster poor. Therefore, how to design a more efficient data processing solution for distributed memory computing engine clusters has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] The present invention provides a data processing method for a distributed memory computing engine cluster, which is used to solve the defects of poor data transmission and processing efficiency of the distributed memory computing engine cluster in the prior art.
[0004] The present invention provides a data processing method for a distributed memory computing engine cluster, comprising:
[0005] When the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, obtaining the execution result data of the mapping task stage, storing the execution result data in the first memory extension module configured in the mapping task node, and obtaining the mapping task result state object;
[0006] Uploading the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and saving the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object;
[0007] When the simplification task phase of the data simplification task node in the distributed memory computing engine cluster is started, a query request for obtaining a mapping task result status object is sent to the mapping output tracking master control object in the memory engine driver to obtain an address query result from the database management object; based on the address query result, the mapping task result status object is obtained from the first memory extension module using a remote memory direct access module.
[0008] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, when the mapping task stage of a data mapping task node in the distributed memory computing engine cluster is completed, the execution result data of the mapping task stage is obtained, the execution result data is stored in a first memory expansion module configured in the mapping task node, and a mapping task result state object is obtained, which specifically includes:
[0009] When the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is completed, the address of the execution result data of the mapping task phase is recorded based on the first mapping tracking work object in the data mapping task node, and the execution result data is written into the first memory extension module in the data mapping task node based on the first block management slave node object in the data mapping task node;
[0010] In the case where the memory space of the first memory extension module is insufficient, based on the first block management slave node object, writing the execution result data exceeding the memory space of the first memory extension module to the first local disk file in the data mapping task node;
[0011] When the writing of the execution result data is completed, encapsulation processing is performed based on the address storing the execution result data in the first memory expansion module and the address storing the execution result data in the first local disk file to obtain a mapping task result state object.
[0012] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, when the simplification task phase of a data simplification task node in the distributed memory computing engine cluster is started, a query request for obtaining a mapping task result state object is sent to a mapping output tracking master control object in the memory engine driver to obtain an address query result from the database management object, specifically comprising:
[0013] When the simplification task phase of the data simplification task node in the distributed memory computing engine cluster is started, the second mapping tracking work object in the data simplification task node is used to find the mapping task result state object from the second block management node object in the data simplification task node;
[0014] In the case that the mapping task result status object is not found, sending a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver through the second block management slave node object;
[0015] When the mapping output tracking master control object in the memory engine driver receives the query request for obtaining the mapping task result status object, the mapping output tracking master control object searches for the address of the mapping task result status object corresponding to the mapping task stage in the block management master control object through the mapping output tracking master control object to obtain the address query result from the database management object;
[0016] The data reduction task returns the address query result to the data reduction task node.
[0017] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, the method of obtaining a mapping task result state object from the first memory extension module using a remote memory direct access module according to the address query result specifically includes:
[0018] In the case where the data simplification task of the data simplification task node obtains the address query result, triggering the start of mapping task result data shuffling, and calling the remote memory direct access module configured in the data simplification task node and the remote memory direct access module configured in the data mapping task node to obtain the data of the corresponding mapping task result status object from the data mapping task node; wherein the data of the mapping task result status object includes the server address, data block address and disk file address of the execution result data of the mapping task stage;
[0019] Calling a preset memory driver through the second block management slave node object to write the received data of the mapping task result state object into a local second memory extension module, and when the memory space of the second memory extension module is full, writing the data exceeding the memory space of the second memory extension module into a local second local disk file;
[0020] After the data of the mapping task result status object is written, the address of the data of the mapping task result status object stored in the second memory extension module and the data of the mapping task result status object stored in the second local disk file are encapsulated to obtain a new mapping task result status object, and the new mapping task result status object is sent to the mapping output tracking master control object in the memory engine driver through the second mapping tracking work object.
[0021] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, the remote memory direct access module configured in the data reduction task node and the remote memory direct access module configured in the data mapping task node are called to obtain data of a corresponding mapping task result state object from the data mapping task node, specifically comprising:
[0022] Calling the remote memory direct access module configured in the data reduction task node, and sending a data transmission request to the remote memory direct access module configured in the data mapping task node;
[0023] When the remote memory direct access module configured in the data mapping task node receives the data transmission request, returning the work queue element in the sending queue pre-configured in the data mapping task node to the receiving queue pre-configured in the data mapping task node;
[0024] The data of the mapping task result status object corresponds to a plurality of different data blocks, and the sending queue stores a plurality of work queue elements to be transmitted which are obtained by pre-processing the data blocks.
[0025] A data processing method for a distributed memory computing engine cluster provided by the present invention also includes:
[0026] Before the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is executed, it also includes: pre-allocating a corresponding first memory area for a first memory extension module in the data mapping task node, and configuring the first memory extension module in the first memory area; and pre-allocating a corresponding second memory area for a second memory extension module in the data reduction task node, and configuring the second memory extension module in the second memory area.
[0027] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, before the mapping task phase of a data mapping task node in the distributed memory computing engine cluster is executed, the method further includes: pre-configuring a first remote memory direct access module supporting a remote memory direct access protocol in the data mapping task node, and configuring a second remote memory direct access module supporting a remote memory direct access protocol in the data reduction task node;
[0028] The first remote memory direct access module and the second remote memory direct access module are used to transmit data in the memory extension module through a remote memory direct access protocol.
[0029] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, after configuring a second remote memory direct access module supporting a remote memory direct access protocol in the data reduction task node, the method further includes:
[0030] Using the second remote memory direct access module of the data reduction task node to send a link channel establishment request to the first remote memory direct access module in the data mapping task node;
[0031] After receiving the link channel establishment request, the first remote memory direct access module in the data mapping task node returns corresponding response information for establishing the link channel;
[0032] According to the response information of establishing the link channel, a link channel for remotely sending data is constructed between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node.
[0033] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, after constructing a link channel for remotely sending data between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node, the method further includes:
[0034] Using the second remote memory direct access module of the data reduction task node to send a data transmission request to the first remote memory direct access module in the data mapping task node;
[0035] When the first remote memory direct access module in the data mapping task node receives the data transmission request, the data of the mapping task result status object stored in the first memory extension module of the data mapping task node is sent to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel.
[0036] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes:
[0037] Configuring a corresponding sending queue in the data mapping task node;
[0038] Processing the data of the mapping task result state object to obtain at least one data block;
[0039] The at least one data block is processed to obtain at least one work queue element corresponding to each data block, and the at least one work queue element is pre-stored in the sending queue.
[0040] According to a data processing method of a distributed memory computing engine cluster provided by the present invention, before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes:
[0041] A receiving queue corresponding to the second memory expansion module is pre-configured in the data reduction task node; wherein the receiving queue is used to store at least one work queue element sent by a corresponding sending queue configured in the data mapping task node.
[0042] The present invention also provides a data processing device for a distributed memory computing engine cluster, comprising:
[0043] A data mapping task processing unit is used to obtain the execution result data of the mapping task stage when the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, store the execution result data in the first memory extension module configured in the mapping task node, and obtain a mapping task result state object;
[0044] The engine driver management unit is used to upload the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and save the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object;
[0045] A data simplification task processing unit is used to send a query request for obtaining a mapping task result status object to a mapping output tracking master control object in the memory engine driver when the simplification task phase of a data simplification task node in a distributed memory computing engine cluster is started, so as to obtain an address query result from the database management object; and obtain the mapping task result status object from the first memory extension module based on the address query result and using a remote memory direct access module.
[0046] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the data mapping task processing unit is specifically used for:
[0047] When the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is completed, the address of the execution result data of the mapping task phase is recorded based on the first mapping tracking work object in the data mapping task node, and the execution result data is written into the first memory extension module in the data mapping task node based on the first block management slave node object in the data mapping task node;
[0048] In the case where the memory space of the first memory extension module is insufficient, based on the first block management slave node object, writing the execution result data exceeding the memory space of the first memory extension module to the first local disk file in the data mapping task node;
[0049] When the writing of the execution result data is completed, encapsulation processing is performed based on the address storing the execution result data in the first memory expansion module and the address storing the execution result data in the first local disk file to obtain a mapping task result state object.
[0050] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the data reduction task processing unit is specifically used for:
[0051] When the simplification task phase of the data simplification task node in the distributed memory computing engine cluster is started, the second mapping tracking work object in the data simplification task node is used to find the mapping task result state object from the second block management node object in the data simplification task node;
[0052] In the case that the mapping task result status object is not found, sending a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver through the second block management slave node object;
[0053] When the mapping output tracking master control object in the memory engine driver receives the query request for obtaining the mapping task result status object, the mapping output tracking master control object searches for the address of the mapping task result status object corresponding to the mapping task stage in the block management master control object through the mapping output tracking master control object to obtain the address query result from the database management object;
[0054] The data reduction task returns the address query result to the data reduction task node.
[0055] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the data reduction task processing unit is specifically used for:
[0056] In the case where the data simplification task of the data simplification task node obtains the address query result, triggering the start of mapping task result data shuffling, and calling the remote memory direct access module configured in the data simplification task node and the remote memory direct access module configured in the data mapping task node to obtain the data of the corresponding mapping task result status object from the data mapping task node; wherein the data of the mapping task result status object includes the server address, data block address and disk file address of the execution result data of the mapping task stage;
[0057] Calling a preset memory driver through the second block management slave node object to write the received data of the mapping task result state object into a local second memory extension module, and when the memory space of the second memory extension module is full, writing the data exceeding the memory space of the second memory extension module into a local second local disk file;
[0058] After the data of the mapping task result status object is written, the address of the data of the mapping task result status object stored in the second memory extension module and the data of the mapping task result status object stored in the second local disk file are encapsulated to obtain a new mapping task result status object, and the new mapping task result status object is sent to the mapping output tracking master control object in the memory engine driver through the second mapping tracking work object.
[0059] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the calling of the remote memory direct access module configured in the data reduction task node and the remote memory direct access module configured in the data mapping task node to obtain data of the corresponding mapping task result state object from the data mapping task node specifically includes:
[0060] Calling the remote memory direct access module configured in the data reduction task node, and sending a data transmission request to the remote memory direct access module configured in the data mapping task node;
[0061] When the remote memory direct access module configured in the data mapping task node receives the data transmission request, returning the work queue element in the sending queue pre-configured in the data mapping task node to the receiving queue pre-configured in the data mapping task node;
[0062] The data of the mapping task result status object corresponds to a plurality of different data blocks, and the sending queue stores a plurality of work queue elements to be transmitted which are obtained by pre-processing the data blocks.
[0063] A data processing device of a distributed memory computing engine cluster provided by the present invention further includes:
[0064] Memory expansion module configuration unit for:
[0065] Before the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is executed, it also includes: pre-allocating a corresponding first memory area for a first memory extension module in the data mapping task node, and configuring the first memory extension module in the first memory area; and pre-allocating a corresponding second memory area for a second memory extension module in the data reduction task node, and configuring the second memory extension module in the second memory area.
[0066] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, before the mapping task phase of a data mapping task node in the distributed memory computing engine cluster is executed, the device further comprises:
[0067] Remote memory direct access module configuration unit for:
[0068] Pre-configuring a first remote memory direct access module supporting a remote memory direct access protocol in the data mapping task node, and configuring a second remote memory direct access module supporting a remote memory direct access protocol in the data reduction task node;
[0069] The first remote memory direct access module and the second remote memory direct access module are used to transmit data in the memory extension module through a remote memory direct access protocol.
[0070] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, after configuring a second remote memory direct access module supporting a remote memory direct access protocol in the data reduction task node, the device further comprises:
[0071] Link channel building unit, used to:
[0072] Using the second remote memory direct access module of the data reduction task node to send a link channel establishment request to the first remote memory direct access module in the data mapping task node;
[0073] After receiving the link channel establishment request, the first remote memory direct access module in the data mapping task node returns corresponding response information for establishing the link channel;
[0074] According to the response information of establishing the link channel, a link channel for remotely sending data is constructed between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node.
[0075] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, after constructing a link channel for remotely sending data between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node, the device further includes:
[0076] Remote data transmission unit, used for:
[0077] Using the second remote memory direct access module of the data reduction task node to send a data transmission request to the first remote memory direct access module in the data mapping task node;
[0078] When the first remote memory direct access module in the data mapping task node receives the data transmission request, the data of the mapping task result status object stored in the first memory extension module of the data mapping task node is sent to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel.
[0079] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes:
[0080] Send queue construction and storage unit, used for:
[0081] Configuring a corresponding sending queue in the data mapping task node;
[0082] Processing the data of the mapping task result state object to obtain at least one data block;
[0083] The at least one data block is processed to obtain at least one work queue element corresponding to each data block, and the at least one work queue element is pre-stored in the sending queue.
[0084] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes:
[0085] Receive queue construction and storage unit, used for:
[0086] A receiving queue corresponding to the second memory expansion module is pre-configured in the data reduction task node; wherein the receiving queue is used to store at least one work queue element sent by a corresponding sending queue configured in the data mapping task node.
[0087] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method of the distributed memory computing engine cluster as described in any one of the above items is implemented.
[0088] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data processing method of the distributed memory computing engine cluster as described in any of the above items is implemented.
[0089] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method of the distributed memory computing engine cluster as described in any of the above items.
[0090] The data processing method of the distributed memory computing engine cluster provided by the present invention obtains execution result data when the mapping task phase of the data mapping task node is completed, stores the execution result data in a first memory expansion module configured in the mapping task node, obtains a mapping task result status object, uploads the address of the mapping task result status object to a mapping output tracking master control object in a memory engine driver, and sends a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver when the simplification task phase of the data simplification task node is started, so as to obtain an address query result from the database management object, and obtains the mapping task result status object from the first memory expansion module according to the address query result, which can effectively improve the data transmission and processing efficiency and cluster performance of the distributed memory computing engine cluster. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0092] Figure 1 This is one of the flow charts of the data processing method of the distributed memory computing engine cluster provided by the present invention.
[0093] Figure 2 This is the second flow chart of the data processing method of the distributed memory computing engine cluster provided by the present invention.
[0094] Figure 3 This is the third flow chart of the data processing method of the distributed memory computing engine cluster provided by the present invention.
[0095] Figure 4 It is a schematic diagram of block data management in the data processing method of the distributed memory computing engine cluster provided by the present invention.
[0096] Figure 5 It is a flow chart of a data processing method of a distributed memory computing engine cluster in the prior art provided by the present invention.
[0097] Figure 6 It is a structural schematic diagram of a data processing device of a distributed memory computing engine cluster provided by the present invention.
[0098] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0099] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0100] Combine the following Figure 1-Figure 7 The data processing method and device of the distributed memory computing engine cluster of the present invention are described, and its embodiments are described in detail.
[0101] The following is a detailed description of an embodiment of the data processing method of the distributed memory computing engine cluster according to the present invention. Figure 1 As shown, it is one of the flow charts of the data processing method of the distributed memory computing engine cluster provided by the present invention, and the specific implementation process includes the following steps:
[0102] Step 101: When the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, the execution result data of the mapping task stage is obtained, and the execution result data is stored in the first memory extension module configured in the mapping task node to obtain a mapping task result status object.
[0103] In an embodiment of the present invention, when the mapping task stage of a data mapping task node in a distributed memory computing engine cluster is completed, the address of the execution result data of the mapping task stage is recorded based on the first mapping tracking work object in the data mapping task node, and the execution result data is written to the first memory expansion module in the data mapping task node based on the first block management slave node object in the data mapping task node. When the memory space of the first memory expansion module is insufficient, the execution result data exceeding the memory space of the first memory expansion module is written to the first local disk file in the data mapping task node based on the first block management slave node object. When the execution result data is written, encapsulation processing is performed based on the address of the execution result data stored in the first memory expansion module and the address of the execution result data stored in the first local disk file to obtain a mapping task result status object. Wherein, the distributed memory computing engine cluster is a Spark cluster, and the Spark cluster includes multiple worker nodes (WorkerNode), such as multiple data mapping task nodes and multiple data simplification task nodes. The data mapping task node is a Map task node (for example, Figure 3 Node M1 or Figure 2Computer node 1 in the diagram). Accordingly, the mapping task stage is the Map task (i.e., MapTask) stage. The first mapping tracking work object, i.e., the first Map tracking work object (MapCXLTrackerWorker) interface or CXL-Map tracking work object that supports CXL, is a program module preset in the Map task node for recording the address of the local execution result data of the Map task, such as managing the execution result data of the Map task in the form of data blocks and corresponding addresses. The first block management slave node object, i.e., the block management slave node object (BlockManagerCXLSlave) that supports CXL, is a program module preset in the Map task node for managing the local execution result data of the Map task and its address, such as managing the execution result data of the Map task in the form of data blocks. The first memory extension module is a CXL memory device pre-configured in the Map task node for storing the execution result data and its address of the Map task, which can be specifically CXL.mem (i.e., memory device). The first local disk file is a disk file pre-configured in the Map task node for storing the execution result data of the Map task and its address, which can be specifically Figure 3 The mapping task result status object (MapStatus) is encapsulated, and the mapping task result status object in the data mapping task node is the first Map task result status object. It should be noted that the Figure 3 The lines marked with 1, 2, 3, ..., 7 are link channels connecting the node M1, the node R1, the memory engine driver (Driver), and its internal modules.
[0104] That is, when the Map task stage in the Map task node is executed, the address of the execution result data of the Map task stage is recorded based on the first Map tracking work object in the Map task node, and the execution result data of the Map task stage is written to the first memory expansion module in the Map task node based on the first block management slave node object. When the memory space of the first memory expansion module is insufficient, the execution result data exceeding the memory space of the first memory expansion module is written to the first local disk file in the Map task node based on the first block management slave node object. When the execution result data is written, the address of the first memory expansion module and the address of the first local disk file are encapsulated into the first Map task result status object, and the first Map task result status object is uploaded to the Map output tracking master control object in the memory engine driver through the first Map tracking work object, and the first Map task result status object is saved to the database management object in the block management master control object through the Map output tracking master control object.
[0105] Step 102: upload the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and save the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object.
[0106] Specifically, the mapping task result status object (i.e., Map result status object) is uploaded to the memory engine driver ( Figure 3 The mapping output tracking master object in the Driver) is the Map output tracking master object (MapOutputTrackerMaster), and the Map result status object is saved to the block management master object, which is the database management object of the block management control object (BlockManagerMaster), and the database management object is the CXL block management object (BlockMasterCXL).
[0107] It should be noted that the Map output tracking master control object is a program module preset in the memory engine driver for managing the execution result data and addresses of the Map tasks in all working nodes. The distributed memory computing engine cluster includes multiple working nodes, and the working nodes are data reduction task nodes ( Figure 2 Computer Node 2 or Figure 3 Node R1 in the Node Manager or a data mapping task node.
[0108] The block management control object includes a CXL block management object, etc., which is a program module preset in the memory engine driver for managing and controlling the data block corresponding to the execution result data of the Map task and the address of the data block. The CXL block management object is a program module preset in the memory engine driver for managing and controlling the CXL data block corresponding to the execution result data of the Map task and the address of the CXL data block.
[0109] Step 103: When the simplification task phase of the data simplification task node in the distributed memory computing engine cluster is started, a query request for obtaining a mapping task result status object is sent to the mapping output tracking master control object in the memory engine driver to obtain an address query result from the database management object; based on the address query result, the mapping task result status object is obtained from the first memory extension module using a remote memory direct access module.
[0110] In an embodiment of the present invention, when the reduction task phase (i.e., Reduce task phase) of a data reduction task node (i.e., Reduce task node) in a distributed memory computing engine cluster is started, the second mapping tracking work object (i.e., second Map tracking work object) in the data reduction task node is used to search for a mapping task result status object from a second block management slave node object (i.e., second CXL block management slave node object) in the data reduction task node. If the local mapping task result status object (i.e., second Map task result status object) is not found, a query request for obtaining the mapping task result status object is sent to a mapping output tracking master control object (i.e., Map output tracking master control object) in the memory engine driver through the second block management slave node object. When the mapping output tracking master control object in the memory engine driver receives the query request for obtaining the mapping task result status object, the address of the mapping task result status object corresponding to the mapping task phase is searched in the block management master control object through the mapping output tracking master control object to obtain the address query result from the database management object. The address query result is returned to the data reduction task of the data reduction task node. Wherein, the Reduce task is ReduceTask. The second mapping tracking work object is a CXL-Map tracking work object (MapCXLTrackerWorker) local to the Reduce task node, that is, a second Map tracking work object. The second block management slave node object is a CXL block management slave object (BlockManagerCXLSlave) local to the Reduce task node, that is, a second CXL block management slave node object. The mapping task result status object search is a Map status object (MapStatus) local to the Reduce task node, that is, a second Map task result status object. If the required data is not available locally, a request to obtain Map status (getMapStatuses) is sent to the Map output tracking master object (MapOutputTrackerMaster) of the Driver to query the Map status object (MapStatus) of the MapTask running result. That is, a query request to obtain the mapping task result status object is sent to the mapping output tracking master object in the memory engine driver through the second block management slave node object to obtain the address query result from the database management object.After the Map output tracking master object (MapOutputTrackerMaster) in the Driver receives the GetMapStatuses request from the Reduce task (ReduceTask), it will find the first Map task result status object (MapStatus) corresponding to the Map task in the block management master object (BlockManagerMaster), that is, the Map task result status object (first Map task result status object), and return the queried result to the Reduce task (ReduceTask). MapStatus includes the server address, data block address, and disk file address of the Map task result status object in CXL.mem, and the server address, data block address, and disk file address of the Map task result are used as the address query result.
[0111] In the process of obtaining the mapping task result status object from the first memory extension module according to the address query result and using the remote memory direct access module, it is necessary to trigger the start of the mapping task result data shuffling when the data reduction task of the data reduction task node obtains the address query result, call the request function protocol transmission interface for obtaining CXL data, and call the remote memory direct access module configured in the data reduction task node and the remote memory direct access module configured in the data mapping task node to obtain the data of the corresponding mapping task result status object (i.e., the data corresponding to the first Map task result status object) from the data mapping task node. The data of the mapping task result status object includes the server address, data block address, and disk file address where the execution result data of the mapping task stage is located. The preset memory driver is called through the second block management slave node object (i.e., the second CXL block management slave node object) to write the received data of the mapping task result status object into the local second memory extension module. When the memory space of the second memory extension module is full, the data exceeding the memory space of the second memory extension module is written into the local second local disk file (i.e., Figure 3 After the data of the mapping task result state object is written, the address of the data of the mapping task result state object stored in the second memory extension module and the data of the mapping task result state object stored in the second local disk file are encapsulated to obtain a new mapping task result state object, and the new mapping task result state object is sent to the mapping output tracking master control object in the memory engine driver through the second mapping tracking work object.
[0112] Among them, in the process of calling the remote memory direct access module configured in the data simplification task node and the remote memory direct access module configured in the data mapping task node to obtain the data of the corresponding mapping task result status object from the data mapping task node, specifically, it is necessary to call the remote memory direct access module configured in the data simplification task node, send a data transmission request to the remote memory direct access module configured in the data mapping task node, and when the remote memory direct access module configured in the data mapping task node receives the data transmission request, the work queue element (that is, including) in the sending queue pre-configured in the data mapping task node is sent. Figure 2 The queue element 0, queue element 1, and queue element 2 of the sending queue in the data mapping task node are returned to the receiving queue pre-configured in the data mapping task node. The data of the mapping task result state object corresponds to multiple different data blocks, and the sending queue stores multiple work queue elements to be transmitted obtained by pre-processing the data blocks. The queue element 0 is WQE0, the queue element 1 is WQE1, and the queue element 2 is WQE2.
[0113] In addition, in an embodiment of the present invention, it also includes: before the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is executed, a corresponding first memory area is pre-allocated to the first memory extension module in the data mapping task node, the first memory extension module is configured in the first memory area, and a first remote memory direct access module that supports the remote memory direct access protocol is configured in the data mapping task node. A corresponding second memory area is pre-allocated to the second memory extension module in the data reduction task node, the second memory extension module is configured in the second memory area, and a second remote memory direct access module that supports the remote memory direct access protocol is configured in the data reduction task node. The first remote memory direct access module and the second remote memory direct access module are used to transmit data in the memory extension module through the remote memory direct access protocol. The remote memory direct access module includes the first remote memory direct access module (such as Figure 2 The RDMA network card RNIC in the computer node 1 shown in the figure, that is, the network card supporting the RDMA protocol is configured on the data mapping task node of the distributed memory computing cluster, and an interface for calling the RDMA protocol is added to the data shuffling process interface of the distributed memory computing engine) and the second remote memory direct access module (such as Figure 2The RDMA network card RNIC in the computer node 2 shown in the figure, that is, the network card supporting the RDMA protocol is configured on the data reduction task node of the distributed memory computing cluster, and an interface for calling the RDMA protocol is added to the data shuffling process interface of the distributed memory computing engine). The first remote memory direct access module and the second remote memory direct access module need to be pre-configured in the data mapping task node and the data reduction task node respectively. RDMA is a new direct memory access technology. RDMA allows computers to directly access the memory of other computers without being processed by the processor. In terms of technical implementation, RDMA is a remote memory direct high-speed access technology with fully optimized intelligent network card and software architecture. By solidifying the RDMA protocol on the hardware (i.e., the network card) and supporting zero copy (Zero-copy) and kernel bypass (Kernel bypass) two ways to achieve its high-performance remote direct data access goal. The RDMA data transmission process does not require CPU intervention (No CPU involvement), and the application can access the remote host memory without consuming any CPU in the remote host. Computer node 1 is a data mapping task node, and each computer node 1 may contain both a sending queue and a receiving queue. Computer node 2 is a data reduction task node, and each computer node 2 may include both a sending queue and a receiving queue. The application main process is the Spark application main process or Application. WR represents an interface.
[0114] It should be noted that the second map tracking work object, i.e., the second Map tracking work object (MapCXLTrackerWorker) or CXL-Map tracking work object configured in the Reduce task node and supporting CXL, is a program module preset in the Reduce task node for recording the address of the local execution result data of the Map task, such as managing the execution result data of the Map task in the form of data blocks and corresponding addresses. The second block management slave node object, i.e., the block management slave node object (BlockManagerCXLSlave) supporting CXL configured in the Reduce task node, is a program module preset in the Reduce task node for managing the local execution result data of the Map task and its address, such as managing the execution result data of the Map task in the form of data blocks. The second memory extension module is a CXL memory device pre-configured in the Reduce task node for storing the execution result data and its address of the Map task, which can be specifically CXL.mem (i.e., an extended memory device). The second local disk file is a disk file pre-configured in the Reduce task node for storing the execution result data and its address of the Map task, which can be specifically Figure 3 The File disk file in.
[0115] It should be noted that Compute Express Link (CXL) is an industry-supported cache-coherent open interconnect protocol for processors, memory extensions, and accelerators. The CXL protocol contains three sub-protocols: CXL.io, CXL.cache, and CXL.mem. CXL defines three types of devices, such as CXL.mem memory devices that support the CXL.mem protocol.
[0116] In the embodiment of the present invention, the CXL block management object (BlockManagerCXL) of the slave working node in the distributed memory computing engine cluster is registered to the block management control object (BlockManagerMaster) through the network. The logical relationship between the block management control object and the CXL block management object is as follows: Figure 4 As shown. The CXL block management object (BlockManagerCXL) mainly consists of three parts. The CXL memory block storage area (i.e., CXLMemoryStore object) implements the reading and writing of data blocks of the CXL.mem memory device, calls the CXL RcxlmemDriver local data writing interface, and reads and writes data according to the address of the data block BlockCXLMemoryEndpointRef (i.e., CXL block reference breakpoint); the MemoryStore object (i.e., memory storage area) implements the processing of data blocks in memory according to the address of BlockMemoryEndpointRef (memory block reference breakpoint); the DiskStore object (disk storage area) implements the processing of disk data blocks through the address of BlockMemoryStorageEndpoint (block storage endpoint). The block management control object (BlockManagerMaster) stores the CXL block address (CXLBlockLocations) of the Map task result data, the memory block address (MemBlockLocations), the shuffle block address (shuffleBlockLocations) in the Reduce task, and maintains the heartbeat data (BlockManagerHeartbeatEndpoint) with each block storage node.
[0117] It should be noted that before executing step 101, it is necessary to configure CXL.mem memory devices (which can also be CXL.io memory devices) in the data mapping task nodes and data reduction task nodes in advance, that is, to develop a work node that supports CXL.io or CXL.mem Type3 memory devices for the distributed memory computing engine cluster, and to configure the remote cxl.mem driver software RcxlmemDriver that supports the registration, memory space allocation, and read and write operation functions of CXL.mem memory devices for the memory management engine of the distributed memory computing engine. The specific technical implementation method is as follows: develop and configure more than one CXL root port on the system bus of the CPU of the work node of the distributed memory computing engine cluster, and connect the CXL.mem memory device through the root port, where the CXL.mem memory device type is a CXL Type3 memory device endpoint. The working node calls the kernel driver on the operating system to start PCIe transactions to scan the base address register (BAR) and internal memory (Host-managed Device Memory, HDM) of the CXL.mem memory device; registers the CXL.mem memory device in the node according to the scan information, allocates address space for BAR and HDM in the reserved memory space, and initializes the memory of the CXL.mem memory device and allocates address space; then synchronizes and maps the logical address space data of the CXL.mem memory device with the physical address space in the CXL.mem memory device; writes the information and address space data of the CXL memory device into the node operating system configuration file; when the working node restarts, it reads the CXL memory device configuration information of the operating system, and when receiving a data read / write operation request, it accesses the CXL.mem device HDM memory through the load / store instruction, and the request is passed to the corresponding root port, which converts the request into a CXL link control unit packet (CXL flit) and calls the CXL protocol to send it to the CXL Type3 device. After that, the CXL controller in the CXL memory device parses the CXL flit, converts the incoming address, and sends the converted request to the memory controller to realize the reading and writing of the memory resources of the CXL memory device. By pre-configuring the CXL.mem memory device in the data mapping task node and the data reduction task node, the remote direct data transmission between the data mapping task node and the data reduction task node based on the remote memory direct access module can be realized, thereby improving the data transmission efficiency between the data mapping task node and the data reduction task node.
[0118] In addition, before executing step 101, the RCXL protocol process for data transmission between the Map task node and the Reduce task node in the distributed memory computing engine cluster needs to be implemented in advance. The logical unit of data transmission operation of the RCXL protocol is the RDMA device work queue element WQE (Work Queue Elements), which is the "task description" issued by the software to the hardware RDMA host channel adapter HCA, and contains the task that the software requires the hardware to do and detailed information about the task. The WQE contains a pointer, and the memory buffer pointed to by the pointer is used to cache the data to be sent or stored. At the same time, the RCXL protocol also defines the completion queue element CQE (Completion Queue Element) corresponding to the WQE. The WQE can be regarded as the "task book" issued by the software to the hardware, and the CQE can be regarded as the "task report" returned to the software by the hardware after completing the task. The CQE describes the relevant information of whether a task is correctly executed. If the task fails, the report contains relevant information of the failure. The work queue WQ (Work Queue) is used to store the queue of WQE, and the WQ contains many WQEs. Queue Pair (QP) is a pair of WQs, including a send queue (SQ) and a receive queue (RQ), which are used for logical operations on both ends of the communication process. The Map task node creates a WQE to send data and puts it in its send queue (SQ), and the Reduce task node creates a WQE to receive data and puts it in the receive queue (RQ). The completion queue (CQ) is similar to the WQ. The CQ queue stores the first-in-first-out queue of CQE elements.
[0119] Specifically, the nodes where the Map task and the Reduce task are located create a data operation task queue pair (QP) and a completion queue (CQ), and register the corresponding CXL.mem memory region (Memory Region) for the upcoming RCXL transmission. At the same time, the Reduce task node links with the Map task node by sending a remote cxl direct data request (getRcxlDateRequest) to create a link channel and prepare resources for receiving RDMA data transmission: allocate CXL.mem storage and link resources, and obtain the remote cxl direct data request (getRcxlDateRequest) including the Map result status (MapStatus) object (data source address, data size, data destination address information, etc.) and the RDMA transmission channel work queue element WQE, etc. After receiving the remote cxl direct data request (getRcxlDateRequest), the Map task node parses the request parameters; registers the RDMA and CXL transmission channels with the Map task node; the Map task node creates a WQE and places it in its send queue (SQ). The pointer in the WQE points to the memory buffer segment to be sent to the Reduce task node. Map the data block address to the CXL.mem memory, and mark the CXL.mem memory segment where the data node to be transferred to the Reduce task is located (including the disk file address where the data is located, etc.). Then the Reduce task remote interface receives the permissions of the network card and CXL.mem, and transmits data to the CXL memory device of the Reduce task node. The host channel adapter network card (i.e., the network card that supports the RDMA protocol) on the Map task node cyclically queries whether there is a WQE in the send queue. The host channel adapter HCA will execute the WQE from the Map task, and then convert the data in the cache area into a data stream and send it to the cache of the Reduce task node. When the data stream begins to reach the Reduce task node, the host channel adapter HCA on the node executes the WQE in the receive queue, and then puts the data into the allocated CXL.mem buffer. In this way, data is transmitted from the Map task node to the Reduce task node through the RDMA high-speed channel, and zero copy of data is achieved during the data stream transmission process.
[0120] The present invention uses RDMA as the network protocol for shuffling data transmission between the Executor node of the Map task and the Executor node of the Reduce task. Executor is an executor, which is a process running on the worker node (WorkerNode), responsible for running Map tasks, Reduce tasks, and storing data for applications. It is a process that executes partitioned computing tasks. Compared with the current TCP / IP network technology, the data transmission rate between nodes is improved. Through the RDMA protocol, data is sent from the Map task node to the Reduce task node, and directly from the user-state cache to the cache area of the target worker node, thereby improving transmission efficiency. In addition, the present invention develops and uses worker nodes that support the CXL technology protocol CXL.mem memory device in the data shuffling cluster nodes, and realizes the memory computing engine to write and read data to the CXL.mem memory device. In addition, the following is reconstructed: Figure 3 The shuffling technology solution of the distributed memory computing engine shown in FIG. 1 is that the new distributed memory computing engine shuffling implements support for the RCXL protocol, and realizes direct transmission of shuffled data from the Map task node to the Reduce task node through the RCXL protocol.
[0121] like Figure 5As shown in the figure, the current Spark cluster uses the traditional TCP / IP protocol network (i.e., network communication protocol). Traditional TCP Socket data transmission requires many steps: data is first copied from the Buffer (i.e., the cache of the engine executor) of the source Spark Map job to the Sockets cache (i.e., the cache of the network programming interface) of the current host; data is copied from the Sockets cache to the TransportProtocol Driver cache (i.e., the cache of the transport protocol driver); data is copied from the TransportProtocol Driver cache to the NIC Driver (i.e., the network card driver), and finally the NIC (i.e., the network card) sends the data to the NIC of the target end through the Ethernet network (i.e., the network communication protocol). The target host then transmits the data to the application memory of the target host through the above steps. From the above data flow, it can be seen that the data is transmitted in the TCP / IP network, and it is copied three times on each server, and at least three copies are stored, which seriously occupies the transmission time and affects the efficiency of data transmission between networks. A large amount of memory is required in the process of big data shuffling. Currently, it is limited by memory capacity and the number of interfaces, which cannot meet the memory requirements in shuffling. The present invention adopts a method that combines structured pruning and unstructured pruning. When the mapping task stage of the data mapping task node is completed, the execution result data is obtained, the execution result data is stored in the first memory extension module configured in the mapping task node, and a mapping task result status object is obtained. The address of the mapping task result status object is uploaded to the mapping output tracking master control object in the memory engine driver. When the simplification task stage of the data simplification task node is started, a query request for obtaining the mapping task result status object is sent to the mapping output tracking master control object in the memory engine driver to obtain the address query result from the database management object, and the mapping task result status object is obtained from the first memory extension module according to the address query result. This can effectively improve the data transmission and processing efficiency and cluster performance of the distributed memory computing engine cluster.
[0122] The following is a description of the data processing device of the distributed memory computing engine cluster provided by the present invention. The data processing device of the distributed memory computing engine cluster described below and the data processing method of the distributed memory computing engine cluster described above can be referred to each other. Figure 6 As shown, it is a schematic diagram of the structure of the data processing device of the distributed memory computing engine cluster provided by the present invention. The data processing device of the distributed memory computing engine cluster of the present invention specifically includes the following parts:
[0123] The data mapping task processing unit 601 is used to obtain the execution result data of the mapping task stage when the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, store the execution result data in the first memory extension module configured in the mapping task node, and obtain the mapping task result status object.
[0124] The engine driver management unit 602 is used to upload the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and save the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object.
[0125] The data simplification task processing unit 603 is used to send a query request for obtaining a mapping task result status object to the mapping output tracking master control object in the memory engine driver when the simplification task phase of the data simplification task node in the distributed memory computing engine cluster is started, so as to obtain an address query result from the database management object; based on the address query result, and using the remote memory direct access module, obtain the mapping task result status object from the first memory extension module.
[0126] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the data mapping task processing unit is specifically used for:
[0127] When the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is completed, the address of the execution result data of the mapping task phase is recorded based on the first mapping tracking work object in the data mapping task node, and the execution result data is written into the first memory extension module in the data mapping task node based on the first block management slave node object in the data mapping task node;
[0128] In the case where the memory space of the first memory extension module is insufficient, based on the first block management slave node object, writing the execution result data exceeding the memory space of the first memory extension module to the first local disk file in the data mapping task node;
[0129] When the writing of the execution result data is completed, encapsulation processing is performed based on the address storing the execution result data in the first memory expansion module and the address storing the execution result data in the first local disk file to obtain a mapping task result state object.
[0130] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the data reduction task processing unit is specifically used for:
[0131] When the simplification task phase of the data simplification task node in the distributed memory computing engine cluster is started, the second mapping tracking work object in the data simplification task node is used to find the mapping task result state object from the second block management node object in the data simplification task node;
[0132] In the case that the mapping task result status object is not found, sending a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver through the second block management slave node object;
[0133] When the mapping output tracking master control object in the memory engine driver receives the query request for obtaining the mapping task result status object, the mapping output tracking master control object searches for the address of the mapping task result status object corresponding to the mapping task stage in the block management master control object through the mapping output tracking master control object to obtain the address query result from the database management object;
[0134] The data reduction task returns the address query result to the data reduction task node.
[0135] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the data reduction task processing unit is specifically used for:
[0136] In the case where the data simplification task of the data simplification task node obtains the address query result, triggering the start of mapping task result data shuffling, and calling the remote memory direct access module configured in the data simplification task node and the remote memory direct access module configured in the data mapping task node to obtain the data of the corresponding mapping task result status object from the data mapping task node; wherein the data of the mapping task result status object includes the server address, data block address and disk file address of the execution result data of the mapping task stage;
[0137] Calling a preset memory driver through the second block management slave node object to write the received data of the mapping task result state object into a local second memory extension module, and when the memory space of the second memory extension module is full, writing the data exceeding the memory space of the second memory extension module into a local second local disk file;
[0138] After the data of the mapping task result status object is written, the address of the data of the mapping task result status object stored in the second memory extension module and the data of the mapping task result status object stored in the second local disk file are encapsulated to obtain a new mapping task result status object, and the new mapping task result status object is sent to the mapping output tracking master control object in the memory engine driver through the second mapping tracking work object.
[0139] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, the calling of the remote memory direct access module configured in the data reduction task node and the remote memory direct access module configured in the data mapping task node to obtain data of the corresponding mapping task result state object from the data mapping task node specifically includes:
[0140] Calling the remote memory direct access module configured in the data reduction task node, and sending a data transmission request to the remote memory direct access module configured in the data mapping task node;
[0141] When the remote memory direct access module configured in the data mapping task node receives the data transmission request, returning the work queue element in the sending queue pre-configured in the data mapping task node to the receiving queue pre-configured in the data mapping task node;
[0142] The data of the mapping task result status object corresponds to a plurality of different data blocks, and the sending queue stores a plurality of work queue elements to be transmitted which are obtained by pre-processing the data blocks.
[0143] A data processing device of a distributed memory computing engine cluster provided by the present invention further includes:
[0144] Memory expansion module configuration unit for:
[0145] Before the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is executed, it also includes: pre-allocating a corresponding first memory area for a first memory extension module in the data mapping task node, and configuring the first memory extension module in the first memory area; and pre-allocating a corresponding second memory area for a second memory extension module in the data reduction task node, and configuring the second memory extension module in the second memory area.
[0146] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, before the mapping task phase of a data mapping task node in the distributed memory computing engine cluster is executed, the device further comprises:
[0147] Remote memory direct access module configuration unit for:
[0148] Pre-configuring a first remote memory direct access module supporting a remote memory direct access protocol in the data mapping task node, and configuring a second remote memory direct access module supporting a remote memory direct access protocol in the data reduction task node;
[0149] The first remote memory direct access module and the second remote memory direct access module are used to transmit data in the memory extension module through a remote memory direct access protocol.
[0150] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, after configuring a second remote memory direct access module supporting a remote memory direct access protocol in the data reduction task node, the device further comprises:
[0151] Link channel building unit, used to:
[0152] Using the second remote memory direct access module of the data reduction task node to send a link channel establishment request to the first remote memory direct access module in the data mapping task node;
[0153] After receiving the link channel establishment request, the first remote memory direct access module in the data mapping task node returns corresponding response information for establishing the link channel;
[0154] According to the response information of establishing the link channel, a link channel for remotely sending data is constructed between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node.
[0155] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, after constructing a link channel for remotely sending data between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node, the device further includes:
[0156] Remote data transmission unit, used for:
[0157] Using the second remote memory direct access module of the data reduction task node to send a data transmission request to the first remote memory direct access module in the data mapping task node;
[0158] When the first remote memory direct access module in the data mapping task node receives the data transmission request, the data of the mapping task result status object stored in the first memory extension module of the data mapping task node is sent to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel.
[0159] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes:
[0160] Send queue construction and storage unit, used for:
[0161] Configuring a corresponding sending queue in the data mapping task node;
[0162] Processing the data of the mapping task result state object to obtain at least one data block;
[0163] The at least one data block is processed to obtain at least one work queue element corresponding to each data block, and the at least one work queue element is pre-stored in the sending queue.
[0164] According to a data processing device of a distributed memory computing engine cluster provided by the present invention, before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes:
[0165] Receive queue construction and storage unit, used for:
[0166] A receiving queue corresponding to the second memory expansion module is pre-configured in the data reduction task node; wherein the receiving queue is used to store at least one work queue element sent by a corresponding sending queue configured in the data mapping task node.
[0167] The data processing device of the distributed memory computing engine cluster provided by the present invention obtains execution result data when the mapping task phase of the data mapping task node is completed, stores the execution result data in the first memory extension module configured in the mapping task node, obtains the mapping task result status object, uploads the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and sends a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver when the simplification task phase of the data simplification task node is started, so as to obtain the address query result from the database management object, and obtains the mapping task result status object from the first memory extension module according to the address query result, which can effectively improve the data transmission and processing efficiency and cluster performance of the distributed memory computing engine cluster.
[0168] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device (ie, electronic device) may include: a processor (processor) 701, a communication interface (Communications Interface) 704, a memory (memory) 702 and a communication bus 703, wherein the processor 701, the communication interface 704, and the memory 702 communicate with each other through the communication bus 703. The processor 701 can call the logic instructions in the memory 702 to execute the data processing method of the distributed memory computing engine cluster, which includes: when the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, obtaining the execution result data of the mapping task stage, storing the execution result data in the first memory extension module configured in the mapping task node, and obtaining the mapping task result status object; uploading the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and saving the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object; when the simplification task stage of the data simplification task node in the distributed memory computing engine cluster is started, sending a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver to obtain the address query result from the database management object; according to the address query result, and using the remote memory direct access module, obtaining the mapping task result status object from the first memory extension module.
[0169] In addition, the logic instructions in the above-mentioned memory 702 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.
[0170] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data processing method of the distributed memory computing engine cluster provided by the above methods, the method comprising: when the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, obtaining the execution result data of the mapping task stage, storing the execution result data in the first memory extension module configured in the mapping task node, and obtaining a mapping task result status object; uploading the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and saving the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object; when the simplification task stage of the data simplification task node in the distributed memory computing engine cluster is started, sending a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver to obtain the address query result from the database management object; according to the address query result, and using the remote memory direct access module, obtaining the mapping task result status object from the first memory extension module.
[0171] On the other hand, the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein the program executes the data processing method of the distributed memory computing engine cluster provided by the above methods when it is run, the method comprising: when the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, obtaining the execution result data of the mapping task stage, storing the execution result data in the first memory extension module configured in the mapping task node, and obtaining a mapping task result status object; uploading the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and saving the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object; when the simplification task stage of the data simplification task node in the distributed memory computing engine cluster is started, sending a query request for obtaining the mapping task result status object to the mapping output tracking master control object in the memory engine driver to obtain the address query result from the database management object; according to the address query result, and using the remote memory direct access module, obtaining the mapping task result status object from the first memory extension module.
[0172] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A data processing method for a distributed memory computing engine cluster, characterized in that: include: When the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, obtaining the execution result data of the mapping task stage, storing the execution result data in the first memory extension module configured in the mapping task node, and obtaining the mapping task result state object; Uploading the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and saving the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object; When the reduction task phase of the data reduction task node in the distributed memory computing engine cluster is started, a query request for obtaining the mapping task result state object is sent to the mapping output tracking master control object in the memory engine driver to obtain the address query result from the database management object; According to the address query result, and using the remote memory direct access module, obtain the mapping task result state object from the first memory extension module; The method of sending a query request for obtaining a mapping task result status object to a mapping output tracking master object in the memory engine driver in the case where the reduction task phase of the data reduction task node in the distributed memory computing engine cluster is started, so as to obtain an address query result from the database management object, specifically comprises: in the case where the mapping task result status object is not found through the second mapping tracking work object in the data reduction task node to the second block management slave node object in the data reduction task node, sending a query request for obtaining a mapping task result status object to the mapping output tracking master object in the memory engine driver through the second block management slave node object; When the mapping output tracking master control object receives a query request for obtaining a mapping task result status object, the mapping output tracking master control object searches for an address of the mapping task result status object corresponding to the mapping task stage in the block management master control object to obtain an address query result; The data reduction task returns the address query result to the data reduction task node.
2. The data processing method of the distributed memory computing engine cluster according to claim 1, characterized in that: When the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, obtaining the execution result data of the mapping task stage, storing the execution result data in the first memory expansion module configured in the mapping task node, and obtaining the mapping task result state object specifically includes: When the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is completed, the address of the execution result data of the mapping task phase is recorded based on the first mapping tracking work object in the data mapping task node, and the execution result data is written into the first memory extension module in the data mapping task node based on the first block management slave node object in the data mapping task node; In the case where the memory space of the first memory extension module is insufficient, based on the first block management slave node object, writing the execution result data exceeding the memory space of the first memory extension module to the first local disk file in the data mapping task node; When the writing of the execution result data is completed, encapsulation processing is performed based on the address storing the execution result data in the first memory expansion module and the address storing the execution result data in the first local disk file to obtain a mapping task result state object.
3. The data processing method of the distributed memory computing engine cluster according to claim 1, characterized in that: The acquiring the mapping task result state object from the first memory extension module using the remote memory direct access module according to the address query result specifically includes: In the case where the data simplification task of the data simplification task node obtains the address query result, triggering the start of mapping task result data shuffling, and calling the remote memory direct access module configured in the data simplification task node and the remote memory direct access module configured in the data mapping task node to obtain the data of the corresponding mapping task result status object from the data mapping task node; wherein the data of the mapping task result status object includes the server address, data block address and disk file address of the execution result data of the mapping task stage; Calling a preset memory driver through the second block management slave node object to write the received data of the mapping task result state object into a local second memory extension module, and when the memory space of the second memory extension module is full, writing the data exceeding the memory space of the second memory extension module into a local second local disk file; After the data of the mapping task result status object is written, the address of the data of the mapping task result status object stored in the second memory extension module and the data of the mapping task result status object stored in the second local disk file are encapsulated to obtain a new mapping task result status object, and the new mapping task result status object is sent to the mapping output tracking master control object in the memory engine driver through the second mapping tracking work object.
4. The data processing method of the distributed memory computing engine cluster according to claim 3 is characterized in that: The calling of the remote memory direct access module configured in the data reduction task node and the remote memory direct access module configured in the data mapping task node to obtain data of the corresponding mapping task result state object from the data mapping task node specifically includes: Calling the remote memory direct access module configured in the data reduction task node, and sending a data transmission request to the remote memory direct access module configured in the data mapping task node; When the remote memory direct access module configured in the data mapping task node receives the data transmission request, returning the work queue element in the sending queue pre-configured in the data mapping task node to the receiving queue pre-configured in the data mapping task node; The data of the mapping task result status object corresponds to a plurality of different data blocks, and the sending queue stores a plurality of work queue elements to be transmitted which are obtained by pre-processing the data blocks.
5. The data processing method of the distributed memory computing engine cluster according to claim 1, characterized in that: Before the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is executed, it also includes: pre-allocating a corresponding first memory area for a first memory extension module in the data mapping task node, and configuring the first memory extension module in the first memory area; and pre-allocating a corresponding second memory area for a second memory extension module in the data reduction task node, and configuring the second memory extension module in the second memory area.
6. The data processing method of the distributed memory computing engine cluster according to claim 1, characterized in that: Before the mapping task phase of the data mapping task node in the distributed memory computing engine cluster is executed, it also includes: pre-configuring a first remote memory direct access module supporting a remote memory direct access protocol in the data mapping task node, and configuring a second remote memory direct access module supporting a remote memory direct access protocol in the data reduction task node; The first remote memory direct access module and the second remote memory direct access module are used to transmit data in the memory extension module through a remote memory direct access protocol.
7. The data processing method of the distributed memory computing engine cluster according to claim 6, characterized in that: After configuring the second remote memory direct access module supporting the remote memory direct access protocol in the data reduction task node, the method further includes: Using the second remote memory direct access module of the data reduction task node to send a link channel establishment request to the first remote memory direct access module in the data mapping task node; After receiving the link channel establishment request, the first remote memory direct access module in the data mapping task node returns corresponding response information for establishing the link channel; According to the response information of establishing the link channel, a link channel for remotely sending data is constructed between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node.
8. The data processing method of the distributed memory computing engine cluster according to claim 7, characterized in that: After constructing a link channel for remotely sending data between the first remote memory direct access module in the data mapping task node and the second remote memory direct access module in the data reduction task node, the method further includes: Using the second remote memory direct access module of the data reduction task node to send a data transmission request to the first remote memory direct access module in the data mapping task node; When the first remote memory direct access module in the data mapping task node receives the data transmission request, the data of the mapping task result status object stored in the first memory extension module of the data mapping task node is sent to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel.
9. The data processing method of the distributed memory computing engine cluster according to claim 8, characterized in that: Before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes: Configuring a corresponding sending queue in the data mapping task node; Processing the data of the mapping task result state object to obtain at least one data block; The at least one data block is processed to obtain at least one work queue element corresponding to each data block, and the at least one work queue element is pre-stored in the sending queue.
10. The data processing method of the distributed memory computing engine cluster according to claim 8, characterized in that: Before sending the data of the mapping task result state object stored in the first memory extension module of the data mapping task node to the receiving queue corresponding to the second memory extension module of the data reduction task node through the sending queue based on the link channel, it also includes: A receiving queue corresponding to the second memory expansion module is pre-configured in the data reduction task node; wherein the receiving queue is used to store at least one work queue element sent by a corresponding sending queue configured in the data mapping task node.
11. A data processing device for a distributed memory computing engine cluster, characterized in that: include: A data mapping task processing unit is used to obtain the execution result data of the mapping task stage when the mapping task stage of the data mapping task node in the distributed memory computing engine cluster is completed, store the execution result data in the first memory extension module configured in the mapping task node, and obtain a mapping task result state object; The engine driver management unit is used to upload the address of the mapping task result status object to the mapping output tracking master control object in the memory engine driver, and save the address of the mapping task result status object to the database management object in the block management master control object through the mapping output tracking master control object; A data reduction task processing unit is used to send a query request for obtaining a mapping task result state object to a mapping output tracking master control object in the memory engine driver when the reduction task phase of the data reduction task node in the distributed memory computing engine cluster is started, so as to obtain an address query result from the database management object; According to the address query result, and using the remote memory direct access module, obtain the mapping task result state object from the first memory extension module; The data reduction task processing unit is specifically configured to send a query request for obtaining a mapping task result status object to a mapping output tracking master object in the memory engine driver through the second block management slave node object when no mapping task result status object is found through the second mapping tracking work object in the data reduction task node to the second block management slave node object in the data reduction task node; When the mapping output tracking master control object receives a query request for obtaining a mapping task result status object, the mapping output tracking master control object searches for an address of the mapping task result status object corresponding to the mapping task stage in the block management master control object to obtain an address query result; The data reduction task returns the address query result to the data reduction task node.
12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data processing method of the distributed memory computing engine cluster as described in any one of claims 1 to 10 is implemented.
13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method of the distributed memory computing engine cluster as claimed in any one of claims 1 to 10 is implemented.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method of the distributed memory computing engine cluster as claimed in any one of claims 1 to 10 is implemented.
