Data processing system and method
By introducing a memory pool in a distributed computing system and using CXL or UB protocols to directly exchange data in the memory pool, the resource waste caused by limited memory space of the computing node is solved, and computing efficiency and performance are improved.
Patent Information
- Application Number
- PCT/CN2024/089750
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-04-25
- Publication Date
- 2024-11-07
AI Technical Summary
In distributed computing, due to the limited memory storage space of the computing node, unnecessary data exchange between memory and hard disk, resulting in wasted resources.
By introducing memory pools in the data processing system, using Computation Quick Link (CXL) or Unified Bus (UB) protocols, the processor of the computing node directly exchanges data with the memory pool, avoiding the amount of data between local memory and hard disk Change in and out times to improve data acquisition efficiency.
Reduces resource waste, improves the global Shuffle performance of the MapReduce process, accelerates the computing process, reduces storage hardware costs, and improves the access efficiency of computing nodes.
Smart Images

Figure CN2024089750_07112024_PF_FP_ABST
Abstract
Description
Data processing system and method
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 4, 2023, with application number 202310492750.4 and application name “A Data Processing System and Method”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of computer technology, and in particular to a data processing system and method. Background Art
[0004] Distributed computing is a science that divides engineering data requiring extensive computation into smaller pieces, has multiple computing nodes independently compute the results, and then consolidates these results to draw conclusions. Furthermore, because each computing node can only process a portion of the data for a task, when a particular stage relies on the results of all previous stages, these results must be reassembled.
[0005] However, due to the limited storage space of the computing node's memory, there will be multiple data exchanges between the memory and the hard disk during the reintegration operation. For example, the first computing node is used to generate data, and the second computing node is used to read the data in the first computing node. Since the storage space of the local memory of the first computing node is limited, the first computing node needs to write the generated data from the local memory of the first computing node to the hard disk of the first computing node; correspondingly, due to the limited storage space of the local memory of the second computing node, the second computing node needs to write the data read from the hard disk of the first computing node from the local memory of the second computing node to the hard disk of the second computing node; and before processing the data, the second computing node needs to first read the data from the hard disk of the second computing node to the local memory of the second computing node.
[0006] Thus, in distributed computing, there is unnecessary data exchange between the memory and the hard disk in a computing node.
[0007] Summary of the Invention
[0008] The present application provides a data processing system and method for solving the problem of unnecessary data exchange between memory and hard disk in computing nodes due to limited memory storage space in distributed computing, thereby reducing resource waste.
[0009] In a first aspect, the present application provides a data processing system, comprising: a first computing node, comprising a first processor, the first processor being connected to a memory pool via a communication bus of the first computing node; a second computing node, comprising a second processor, the second processor being connected to the memory pool via a communication bus of the second computing node; the first processor being configured to generate M data, and storing the M data in the memory pool via a cache of the first processor according to a protocol corresponding to the communication bus; the second processor being configured to obtain N data from the memory pool to a cache of the second processor according to a protocol corresponding to the communication bus, and processing the N data, wherein the M data include N data, and M and N are both positive integers. The storage space of the memory pool is larger than the storage space of the local memory of the first computing node. In one possible implementation, the communication bus is a compute express link (CXL) bus or a unified bus (UB).
[0010] In the above technical solution, the first computing node is prevented from writing M data into the local memory of the first computing node, and due to the limited storage space of the local memory of the first computing node, the M data are swapped in and out between the local memory of the first computing node and the hard disk; and / or the second computing node is prevented from reading N data into the local memory of the second computing node, and due to the limited storage space of the local memory of the second computing node, the N data are swapped in and out between the local memory of the second computing node and the hard disk, which helps to reduce resource waste. In addition, in the existing distributed computing process, the first processor includes multiple processor cores, and the M data generated by the multiple processor cores are respectively stored in the memory space corresponding to each processor core. Therefore, when the second processor needs to obtain the N data it needs, it needs to obtain it N times. In the present application, since the M data will be written into the memory pool at the same time, the second processor can obtain the N data it needs at one time, which helps to improve the efficiency of data acquisition.
[0011] In one possible implementation, the first processor is specifically configured to generate M data during the execution of a Map task and store these M data in a memory pool during the execution of a Shuffle task. The second processor, during the execution of a Reduce task, retrieves the N data required for the Reduce task from the memory pool. This technical solution provides a data interaction method within MapReduce, which helps improve global Shuffle performance and thus accelerates the computation process.
[0012] In one possible implementation, the first computing node further includes a local memory, which is also connected to the communication bus of the first computing node. The local memory of the first computing node is used to store metadata for M data. When the second processor retrieves N data from the memory pool and stores them in the cache of the second processor, the second processor is specifically configured to: retrieve the metadata for the N data from the local memory of the first computing node, and then retrieve the N data from the memory pool based on the metadata for the N data. The metadata for each data item may include, but is not limited to, one or more of the following: the address of the data item in the storage space, the data length, and the data backup information.
[0013] In one possible implementation, the M data include first data. The first processor is configured to write the first data to a target address in a memory pool via a protocol corresponding to a communication bus, and to write the target address to the local memory of the first computing node. The second processor is configured to retrieve the target address from the local memory of the first computing node, and to read the first data from the target address in the memory pool via the protocol corresponding to the communication bus. In the above technical solution, the first computing node stores the target address in its local memory, which helps to increase the speed of data processing. The target address also occupies a small amount of storage space, which does not cause storage pressure on the local memory of the first computing node.
[0014] The data processing system can be a storage-computing integrated architecture or a storage-computing separated architecture.
[0015] When the data processing system is a storage-computing integrated architecture:
[0016] In one example, the memory pool is the first extended memory in a first computing node. The first computing node also includes a first switch, and the first processor, the first extended memory, and the first switch are all connected to the communication bus of the first computing node. The second computing node includes a second switch, and the second processor and the second switch are connected to the communication bus of the second computing node. The second switch is connected to the first switch, and the second processor can connect to the first extended memory through the first switch and the second switch. In the above technical solution, each computing node (or the processor in each computing node) manages its own extended memory, decoupling the extended memory of each computing node from each other and improving the stability of system operation.
[0017] In another example, the memory pool includes a first extended memory in a first computing node and a second extended memory in a second computing node; the first computing node also includes a first switch, and the first processor, the first extended memory, and the first switch are all connected to the communication bus of the first computing node; the second computing node also includes a second switch, and the second processor, the second extended memory, and the second switch are all connected to the communication bus of the second computing node; the second switch is connected to the first switch, and the second processor is connected to the first extended memory through the first switch and the second switch, and the first processor is connected to the second extended memory through the first switch and the second switch. In the above technical solution, forming a memory pool by combining the extended memories of each computing node helps to expand the storage space of the memory pool.
[0018] In one possible implementation, before writing the first data to a target address in the extended memory using a protocol corresponding to the communication bus, the first processor is further configured to determine, based on records of data stored in the memory pool, the address of a storage space in the memory pool where no data has been written, and further determine the target address based on the address of the storage space in the memory pool where no data has been written. In this way, the first processor determines the location in the memory pool to which the first data is to be written.
[0019] When the data processing system adopts a storage-computing separation architecture:
[0020] In one possible implementation, the memory pool is a storage node; a first computing node includes a first switch, and a first processor and the first switch are both connected to the communication bus of the first computing node; a second computing node includes a second switch, and a second processor and the second switch are both connected to the communication bus of the second computing node; the storage node is connected to the first computing node via the first switch, and the storage node is connected to the second computing node via the second switch. Setting up a separate storage node as the memory pool decouples computing and storage, helping to improve both computing and storage performance.
[0021] In one example, the system also includes a management node, which is used to manage the memory pool; before the first processor writes the first data to the target address in the memory pool through the protocol corresponding to the communication bus, it is also used to: send an address request to the management node; and receive the target address from the management node in response to the address request. This storage and computing separation architecture can be applied to scenarios with a large number of storage nodes (such as high performance computing, HPC). In the above technical solution, the management node has strong computing power and network input and output (IO) capabilities, and transfers the interaction between the computing node and the storage node to the interaction between the computing node and the management node, which helps to reduce the hardware cost of the storage node and improve the access efficiency of the computing node.
[0022] In another example, a storage node manages a memory pool. Before writing first data to a target address in the memory pool via a protocol corresponding to a communication bus, the first processor is further configured to: send an address request to the storage node; and receive a target address from the storage node in response to the address request. In the above technical solution, the storage node manages its own memory pool, which is suitable for scenarios with a small number of storage nodes and helps reduce hardware deployment costs.
[0023] In one possible implementation, the first processor is further configured to determine that the first data cannot be stored in the memory pool; upon determining that storage space for storing the first data exists in the local memory of the first computing node, the first data is written to the local memory of the first computing node; and / or upon determining that storage space for storing the first data does not exist in the local memory of the first computing node, the first data is written to the hard disk of the first computing node. It is understood that since the storage space of the memory pool is larger than the storage space of the local memory of the first computing node, the first processor and the second processor transfer data based on the memory pool, which helps to reduce unnecessary data exchange; further, since the access latency of the local memory of the first computing node is lower than the access latency of the hard disk of the first computing node, when the first processor determines that there is insufficient storage space in the memory pool to store the first data, the first processor preferentially stores the first data in the local memory of the first computing node, thereby helping to reduce access latency.
[0024] In a second aspect, the present application provides a data processing method applicable to a data processing system including a first computing node and a second computing node; the first computing node includes a first processor, the first processor being connected to a memory pool via a communication bus of the first computing node; the second computing node includes a second processor, the second processor being connected to the memory pool via a communication bus of the second computing node; the method comprising: the first processor generating M data, and storing the M data in the memory pool via a cache of the first processor according to a protocol corresponding to the communication bus; the second processor obtaining N data from the memory pool to a cache of the second processor according to a protocol corresponding to the communication bus, and processing the N data, wherein the M data include N data, and M and N are both positive integers. The storage space of the memory pool is larger than the storage space of the local memory of the first computing node.
[0025] In a possible implementation, the protocol corresponding to the communication bus is the CXL protocol or the UB protocol.
[0026] In one possible implementation, a first processor generates M data and stores them in a memory pool via the first processor's cache according to a protocol corresponding to a communication bus. Specifically, the first processor generates M data while executing a Map task; and stores the M data in the memory pool according to a protocol corresponding to the communication bus during a Shuffle task. A second processor retrieves N data from the memory pool according to the protocol corresponding to the communication bus and stores them in the second processor's cache, and then processes the N data. Specifically, the second processor retrieves the N data required to execute the Reduce task from the memory pool according to the protocol corresponding to the communication bus and then processes the N data.
[0027] In one possible implementation, the first computing node also includes a local memory, which is also connected to the communication bus of the first computing node. The local memory of the first computing node is used to store metadata for M data. When the second processor retrieves N data from the memory pool and stores them in the cache of the second processor, the second processor may specifically retrieve the metadata for the N data from the local memory of the first computing node and then retrieve the N data from the memory pool based on the metadata for the N data. The metadata for each data item may include, but is not limited to, one or more of the following: the address of the data in the storage space, the data length, and the data backup information.
[0028] In one possible implementation, the first computing node writes the first data to the target address in the extended memory through the protocol corresponding to the communication bus, and writes the target address to the local memory of the first computing node; the second computing node obtains the target address from the local memory of the first computing node, and reads the first data from the target address in the extended memory through the protocol corresponding to the communication bus.
[0029] The data processing system can be a storage-computing integrated architecture or a storage-computing separated architecture.
[0030] When the data processing system is a storage-computing integrated architecture:
[0031] In one example, the memory pool is a first extended memory in a first computing node; in another example, the memory pool includes the first extended memory in the first computing node and the second extended memory in the second computing node. Before the first computing node writes the first data to a target address in the memory pool using a protocol corresponding to a communication bus, the first computing node also determines, based on records of stored data in the memory pool, an address of a storage space in the memory pool where no data has been written; and the first computing node determines the target address based on the address of the storage space in the memory pool where no data has been written.
[0032] When the data processing system adopts a storage-computing separation architecture:
[0033] In one example, the system further includes a management node and a storage node, wherein the storage node is a memory pool and the management node is used to manage the memory pool; before the first computing node writes the first data to the target address in the extended memory via the protocol corresponding to the communication bus, the first computing node further sends an address request to the management node; and the first computing node receives the target address from the management node in response to the address request. In another example, the system further includes a storage node, wherein the storage node is a memory pool and the storage node is used to manage the memory pool; before the first computing node writes the first data to the target address in the extended memory via the protocol corresponding to the communication bus, the first computing node further sends an address request to the storage node; and the first computing node receives the target address from the storage node in response to the address request.
[0034] In one possible implementation, the first processor may also determine that the first data cannot be stored in the memory pool; upon determining that there is storage space for storing the first data in the local memory of the first computing node, write the first data to the local memory of the first computing node; and / or, upon determining that there is no storage space for storing the first data in the local memory of the first computing node, write the first data to the hard disk of the first computing node.
[0035] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program or instruction. When the computer program or instruction is executed by a device, it implements the method of the first computing node in any possible implementation of the second aspect above, or when the computer program or instruction is executed by a device, it implements the method of the second computing node in any possible implementation of the second aspect above.
[0036] In a fourth aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a device, the method of the first computing node in any possible implementation of the second aspect above is implemented, or when the computer program or instructions are executed by a device, the method of the second computing node in any possible implementation of the second aspect above is implemented.
[0037] In a fifth aspect, the present application provides a device comprising a processor, the processor being connected to a memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the device implements the method of the first computing node in any possible implementation of the above-mentioned second aspect, or so that the device implements the method of the second computing node in any possible implementation of the above-mentioned second aspect.
[0038] The technical effects that can be achieved in any of the second to fifth aspects mentioned above can refer to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] FIG1 is a schematic diagram of a host transmitting data to a device via the PCIe protocol provided by the present application;
[0040] Figure 2 is a schematic diagram of a MapReduce process;
[0041] FIG3 is a schematic diagram of a distributed computing architecture;
[0042] FIG4 is a schematic diagram of data flow;
[0043] FIG5a is a schematic diagram of an integrated storage and computing architecture exemplarily provided in this application;
[0044] FIG5 b is a schematic diagram of the first storage and computing separation architecture provided by the present application;
[0045] FIG5c is a schematic diagram of a second storage-computing separation architecture exemplarily provided in this application;
[0046] FIG6 is a flow chart of a data processing method exemplarily provided in this application;
[0047] FIG7 is a schematic diagram of data flow provided by this application. DETAILED DESCRIPTION
[0048] In order to better explain the embodiments of this application, the relevant terms or technologies in this application are first explained:
[0049] 1. The high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIe) protocol belongs to high-speed serial point-to-point dual-channel high-bandwidth transmission. Figure 1 is a schematic diagram of a host transmitting data to a device via the PCIe protocol. Specifically, the central processing unit (CPU) in the host needs to first write the data to the host's local memory and then set a flag bit, wherein the flag bit is used to notify the device that the host has written the data to the host's local memory. Accordingly, the device queries the flag bit and determines that the host has written the data to the host's local memory based on the flag bit. Then, the device reads the data from the host's local memory and stores the read data in the device's local memory. The way in which the device transmits data to the host via the PCIe protocol is similar to the way in which the host transmits data to the device via the PCIe protocol, and will not be repeated here.
[0050] 2. Compute Express Link (CXL) protocol: The CXL protocol is a new protocol based on the PCIe protocol and is optimized for cache and memory. The CXL protocol runs on the PCIe physical layer.
[0051] The CXL protocol can be divided into the following three types of protocols:
[0052] The CXL.io protocol is an enumeration configuration protocol, primarily used for device discovery and enumeration, and error reporting.
[0053] The CXL.cache protocol enables devices to achieve memory coherence and access host memory with low latency. Specifically, the CXL.cache protocol provides fast access to host memory by maintaining cache coherence on the device side. This protocol allows devices to participate in the host CPU's coherent cache protocol. Devices can directly access host memory and the host's local cache, and the host can also retrieve data from the device's cache, eliminating the need for memory as an intermediary.
[0054] The CXL.mem protocol allows the host to access device memory as if it were its own local memory. In the CXL.mem protocol, the CPU sends requests to the device, and the device sends responses to the CPU. Requests sent by the CPU are categorized as either data-containing or data-free. Accordingly, responses from the device are categorized as either data-containing or data-free.
[0055] 3. The unified bus (UB) protocol is a communication bus protocol developed by Huawei. It is used to provide high-performance interconnection in data centers, making the interconnection of data centers like a single computer.
[0056] 4. Spark is a general-purpose computing engine designed specifically for large-scale data processing. The core idea of Spark is to reduce disk and network input / output (IO) overhead through a resilient distributed dataset (RDD) that can be operated in parallel and has a fault-tolerant mechanism. Among them, RDD is a parallel data structure that allows users to explicitly store data on the hard disk or in memory and control the partitioning of data. Spark task submission has four stages: (1) Constructing a directed acyclic graph (DAG): The job submitted by the user is converted into a series of RDDs, and a DAG is constructed based on the dependencies between the RDDs. The DAG is then submitted to the scheduling system. The scheduling system includes a DAG scheduler and a task scheduler. (2) The DAG scheduler divides the DAG into stages and sends the tasks generated in the stage to the task scheduler in the form of a task set. (3) The task scheduler schedules tasks to executors based on resource availability. (4) Executors receive tasks and pass them to the thread pool for execution.
[0057] 5. MapReduce (Map-Reduce): Used for parallel computing on large datasets. MapReduce defines the Map function interface and the Reduce function interface. The Map function transforms input data into intermediate results, while the Reduce function transforms intermediate results into the final result. Therefore, users can simply specify the Map and Reduce functions to perform computations on data using parameters.
[0058] Figure 2 shows a schematic diagram of the MapReduce process, which can be divided into the following five stages:
[0059] (1) Input stage: Obtain data, slice the data, and use the obtained slices as input for the Map stage.
[0060] (2) Map stage: The shards from the Input stage are mapped through a Map task to obtain multiple intermediate results, where the Map task is implemented through the Map function.
[0061] (3) Shuffle stage: The intermediate results output by the Map stage are used as the input of the Reduce stage. Specifically, in the Shuffle stage, multiple intermediate results of a Map stage can be distributed to multiple Reduce stages, and the intermediate results of multiple Map stages can be input into the same Reduce stage.
[0062] (4) Reduce phase: Merge the data from the Shuffle to get the final result.
[0063] (5) Output stage: Output the final result to the specified directory.
[0064] Based on the above explanations of professional terms or technologies, the embodiments of the present application are explained as follows.
[0065] FIG3 is a schematic diagram of a distributed computing architecture, which includes one or more computing nodes 200 ( FIG3 shows two computing nodes 200, but is not limited to two computing nodes 200). The two computing nodes 200 can communicate with each other. A computing node 200 is a device with both computing and storage capabilities, such as a server or desktop computer. For example, an Advanced RISC Machine (ARM) server or an X86 server can serve as a computing node 200.
[0066] In terms of hardware, as shown in Figure 3, computing node 200 includes at least a processor 201, memory 202, switch 203, and hard disk 204. Processor 201, memory 202, switch 203, and hard disk 204 are all located on the PCIe communication bus of computing node 200. That is, any two modules of processor 201, memory 202, switch 203, and hard disk 204 can be connected via the PCIe communication bus of computing node 200.
[0067] Specifically, the processor 201 is a central processing unit (CPU) that is used to process data access requests from outside the computing node 200, and is also used to process requests generated inside the computing node 200. For example, when the processor 201 receives a write data request, it temporarily stores the data in these write data requests in the memory 202. When the total amount of data in the memory 202 reaches a certain threshold, the processor 201 sends the data stored in the memory 202 to the hard disk 204 for persistent storage. In addition, the processor 201 is also used to calculate or process data, such as metadata management, virtualized storage space, and address conversion. Figure 3 shows only one processor 201. In actual applications, there are often multiple processors 201, and one processor 201 has one or more processor cores. In this application, the number of processors and the number of processor cores are not limited.
[0068] Memory 202 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very high speed, and serves as a temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be either random access memory or read-only memory (ROM). Among them, random access memory is dynamic random access memory (DRAM), storage class memory (SCM), static random access memory (SRAM), etc. Read-only memory can be programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), etc. In addition, memory 202 can also be a dual in-line memory module or a dual in-line memory module (DIMM), that is, a module composed of DRAM, or a solid state disk (SSD). In actual applications, multiple memories 202 and different types of memories 202 can be configured in the computing node 200. This application does not limit the number and type of memory 202. Furthermore, memory 202 can be configured to have a power-saving feature. This feature ensures that data stored in memory 202 is not lost even after a system power outage and power cycle. Memory with this power-saving feature is referred to as non-volatile memory. Memory 202 can be considered the local memory of compute node 200.
[0069] The hard disk 204 is used to provide storage resources, such as storing data, and can be a magnetic disk or other types of storage media, such as a solid state drive or a shingled magnetic recording hard disk.
[0070] The switch 203 is used to communicate with other computing nodes 200. The switch 203 may be a network interface card (NIC) in a remote direct memory access (RDMA) protocol.
[0071] In distributed computing (such as the above-mentioned MapReduce), there is a large amount of data exchange. For example, the distributed computing involves a first computing node and a second computing node. The first computing node and the second computing node can be the two computing nodes 200 in the distributed computing architecture diagram shown in the above-mentioned Figure 3.
[0072] See FIG4 for an exemplary flow chart of a MapReduce process:
[0073] The first computing node includes Map task a, Map task b, Reduce task a, and Reduce task b; the second computing node includes Reduce task c.
[0074] In the Map phase, Map task a generates data a through c and writes them to the local memory of the first computing node (referred to as the first local memory). Furthermore, Map task b generates data d through f and writes them to the first local memory. Furthermore, due to the limited storage space of the first local memory, Map task a writes data a through c from the first local memory to the hard disk of the first computing node (referred to as the first hard disk). Furthermore, Map task b writes data d through f from the first local memory to the first hard disk.
[0075] In the Reduce stage, Reduce task a needs to read data a and data d from the first hard disk to the first local memory, and process data a and data d; Reduce task b needs to read data b and data e from the first hard disk to the first local memory, and then process data b and data e; Reduce task c needs to read data c and data f from the first hard disk to the local memory of the second computing node (recorded as the second local memory). Due to the limited storage space of the second local memory, Reduce task c needs to store data c and data f in the second local memory to the hard disk of the second computing node (recorded as the second hard disk). When the second computing node needs to process data c and data f, it reads data c and data f from the second hard disk to the second local memory and processes data c and data f.
[0076] It can be understood that Map task a, Map task b, Reduce task a, and Reduce task b are tasks running on the processor of the first computing node (referred to as the first processor), and the first processor here can be one or more. In the Map phase, taking Map task a writing data a as an example, Map task a (i.e., the first processor) can directly write data a to the first local memory. Exemplarily, the first local memory is DRAM, and Map task a can directly write data a to the DRAM via double data rate (DDR). Furthermore, Map task a can write an instruction to the first local memory, which is used to instruct the first local memory to write data a to the first hard disk via the PCIe bus in the first computing node. In the Reduce phase, taking Reduce task a reading data a as an example, Reduce task a (i.e., the first processor) can read data a from the first hard disk to the first local memory based on the PCIe communication bus in the first computing node. Subsequently, Reduce task a reads data a from the first local memory. Exemplarily, the first local memory is DRAM, and Reduce task a can directly read data a from the DRAM via DDR.
[0077] Similarly, Reduce task c is a task running on the processor of the second computing node (referred to as the second processor). In the Reduce phase, taking Reduce task c reading data c as an example, Reduce task c (i.e., the second processor) can read data c from the first hard disk to the second local memory based on RDMA, and then write data c from the second local memory to the second hard disk based on the PCIe communication bus in the second computing node. Subsequently, Reduce task c reads data c from the second hard disk to the second local memory based on the PCIe communication bus in the second computing node, and then reads data c from the second local memory. For example, the second local memory is DRAM, and Reduce task c can read data c directly from DRAM through DDR.
[0078] In the example shown in Figure 4 above, the flow directions of data a, data b, data d, and data e are all "first local memory → first hard disk → first local memory", and the flow directions of data c and data f are all "first local memory → first hard disk → second local memory → second hard disk → second local memory". It can be seen that in distributed computing, due to the limited storage space of memory, there may be unnecessary data exchange between the memory and the hard disk in a certain computing node, resulting in the problem of resource waste.
[0079] In order to solve the problem of resource waste in the above-mentioned distributed computing, Figures 5a to 5c are schematic diagrams of three exemplary architectures provided by this application:
[0080] Figure 5a is an architectural diagram of a data processing system exemplarily provided in the present application. The data processing system can specifically be an architectural diagram of storage and computing integration, which includes one or more computing nodes 200 (two computing nodes 200 are shown in Figure 5a, but the present application is not limited to two computing nodes 200). The two computing nodes 200 are connected through a communication bus between the two computing nodes 200, that is, the two computing nodes 200 communicate with each other through the protocol corresponding to the communication bus.
[0081] In the present application, the communication bus is, for example, a compute express link (CXL) protocol bus, and correspondingly, the protocol corresponding to the communication bus is, for example, the CXL protocol; or, the communication bus is, for example, a unified bus (UB), and correspondingly, the protocol corresponding to the communication bus is, for example, the UB protocol.
[0082] In terms of hardware, the computing node 200 includes at least a processor 201, a memory 202, a switch 203, and a hard disk 204. The processor 201, the memory 202, the switch 203, and the hard disk 204 are all connected to the communication bus of the computing node 200. For specific details, please refer to the description in the relevant embodiment of Figure 3. In addition, the computing node 200 also includes an extended memory 205. The extended memory 205 can be inserted into the computing node 200 by means of a plug-in card. That is, the extended memory 205 is also connected to the communication bus of the computing node 200. The processor 201 can access the extended memory 205 based on the protocol corresponding to the communication bus. The storage space of the extended memory 205 is larger than the storage space of the memory 202 (i.e., the local memory of the computing node 200). Furthermore, when the communication bus is a CXL protocol bus, the extended memory 205 is, for example, a CXL memory expander.
[0083] Furthermore, the switches 203 of the two computing nodes 200 are interconnected to enable the two computing nodes 200 to communicate with each other via a protocol corresponding to the communication bus between the two computing nodes 200. The processor 201 in any computing node 200 can access the extended memory 205 in the other computing node 200 through the two switches 203. For example, the processor 201 in the left computing node 200 can access the extended memory 205 in the right computing node 200 through the switches 203 in the left computing node 200 and the switches 203 in the right computing node 200, respectively. Similarly, the processor 201 in the right computing node 200 can access the extended memory 205 in the left computing node 200 through the switches 203 in the right computing node 200 and the switches 203 in the left computing node 200, respectively.
[0084] In the embodiment of the present application, the extended memory 205 in any computing node 200 can be used as a memory pool of the data processing system. For example, the extended memory 205 in the left computing node 200 can be used as a memory pool of the data processing system. The processor 201 in the left computing node 200 can write data to the memory pool, and the processor 201 in the right computing node 200 can read the data from the memory pool.
[0085] In addition, in another possible manner, for any computing node 200, the extended memory 205 of other computing nodes 200 can also serve as the extended memory 205 of the computing node 200. For example, in Figure 5a, the computing node 200 on the left can access two extended memories 205 (its local extended memory 205 and the local extended memory 205 of the computing node 200 on the right) through the protocol corresponding to the communication bus within the computing node 200 and / or through the protocol corresponding to the communication bus between the two computing nodes 200. In the embodiment of the present application, the extended memories 205 of multiple computing nodes 200 in the data processing system can be combined into a memory pool of the data processing system.
[0086] FIG5b is a schematic diagram of the architecture of a data processing system exemplarily provided in the present application. The data processing system may specifically be a schematic diagram of an architecture with storage and computing separation, including computing nodes 200 (two computing nodes 200 are shown in FIG5b , but are not limited to two computing nodes 200) and storage nodes 300 (two storage nodes 300 are shown in FIG5b , but are not limited to two storage nodes 300). Two computing nodes 200, two storage nodes 300, or any computing node 200 and any storage node 300 can communicate with each other based on the protocol corresponding to the communication bus between the two.
[0087] In terms of hardware, the computing node 200 includes at least a processor 201, a memory 202, a switch 203 and a hard disk 204. The processor 201, the memory 202, the switch 203 and the hard disk 204 are all connected to the communication bus of the computing node 200. For specific details, please refer to the description in the relevant embodiment of Figure 3.
[0088] The storage node 300 includes at least a controller 301, a hard disk 302, and a switch 303. The controller 301, hard disk 302, and switch 303 are all connected to the storage node 300's communication bus. The controller 301 is used to write data to or read data from the hard disk 302 based on read / write data requests sent by the computing node 200. During the data reading and writing process, the controller 301 needs to convert the addresses carried in the read / write data requests into addresses that the hard disk can recognize. Thus, the controller 301 also has some simple computing functions. The switch 303 is used for communication between the storage node 300 and other storage nodes 300, or between the storage node 300 and the computing node 200.
[0089] Furthermore, the switch 203 in the computing node 200 is used not only for the computing node 200 to communicate with other computing nodes 200, but also for the computing node 200 to communicate with the storage node 300. Specifically, the switch 203 in the computing node 200 is connected to the switch 303 in the storage node 300. The processor 201 in the computing node 200 can access the hard disk 302 in the storage node 300 through the switch 203 in the computing node 200 and the switch 303 in the storage node 300.
[0090] It can be understood that the processor 201 in the computing node 200 can access the hard disk 302 in the storage node 300 based on the protocol corresponding to the communication bus between the computing node 200 and the storage node 300. That is, the hard disk 302 is an extended memory for the computing node 200, and the storage space of the hard disk 302 (that is, the extended memory) is larger than the storage space of the memory 202 (that is, the local memory of the computing node 200).
[0091] In the embodiment of the present application, the hard disk 302 in the storage node 300 can be used as a memory pool of the data processing system.
[0092] Figure 5b is an architectural diagram of another data processing system provided as an example in the present application. The data processing system can specifically be an architectural diagram of storage and computing separation, which includes computing nodes 200 (two computing nodes 200 are shown in Figure 5c, but are not limited to two computing nodes 200), storage nodes 300 (two storage nodes 300 are shown in Figure 5c, but are not limited to two storage nodes 300), and management nodes 400 (one management node 400 is shown in Figure 5c, but is not limited to one management node 400).
[0093] Two computing nodes 200, two storage nodes 300, or a computing node 200 and a storage node 300, or a computing node 200 and a management node 400, or a storage node 300 and a management node 400 can communicate with each other based on the protocol corresponding to the communication bus between them.
[0094] The description of the computing node 200 and the storage node 300 can be found in the description of the embodiment related to FIG5b above.
[0095] The management node 400 is used to centrally manage the storage space in the storage node 300 (specifically, the storage space of the hard disk 302), as well as the metadata corresponding to the data stored in the storage space. The metadata corresponding to the data may include the address of the data in the storage space, the data length, the data backup information, etc.
[0096] Management node 400 includes a processor 401, memory 402, and switch 403. Processor 401, memory 402, and switch 403 are all connected to a communication bus of management node 400. The communication bus can be a PCIe communication bus or a communication bus that complies with the CXL protocol (or UB protocol). For a description of processor 401, memory 402, and switch 403, see the description of the embodiment related to FIG. 3 .
[0097] For the convenience of description, this application may record the storage-computing separation architecture shown in Figure 5b as storage separation architecture 1, and the storage-computing separation architecture shown in Figure 5c as storage separation architecture 2. It can be understood that storage separation architecture 2 adds a management node 400 compared to storage separation architecture 1, and storage separation architecture 2 is suitable for scenarios with a large number of computing nodes 200 and / or storage nodes 300 (such as high performance computing (HPC) scenarios).
[0098] In the embodiment of the present application, the hard disk 302 in the storage node 300 can be used as a memory pool of the data processing system.
[0099] FIG6 is a flow chart of a data processing method exemplarily provided in the present application. The data processing method can be applied to the architecture shown in FIG5a to FIG5c , and is specifically as follows:
[0100] When the data processing method is applied to the architecture shown in FIG5 a , the data processing method includes a first computing node and a second computing node. The first computing node and the second computing node may be the two computing nodes 200 in FIG5 a , respectively.
[0101] When the data processing method is applied to the architecture shown in Figure 5b, the data processing method includes a first computing node, a second computing node and a storage node. The first computing node and the second computing node can be two computing nodes 200 in Figure 5b respectively, and the storage node can be any storage node 300 in Figure 5b.
[0102] When the data processing method is applied to the architecture shown in Figure 5c, the data processing method includes a first computing node, a second computing node, a storage node and a management node. The first computing node and the second computing node can be the two computing nodes 200 in Figure 5c respectively, the storage node can be any storage node 300 in Figure 5c, and the management node can be the management node 400 in Figure 5c.
[0103] In this application, the processor, local memory, hard disk, extended memory, and switch in the first computing node may be referred to as the first processor, the first local memory, the first hard disk, the first extended memory, and the first switch; the processor, local memory, hard disk, extended memory, and switch in the second computing node may be referred to as the second processor, the second local memory, the second hard disk, the second extended memory, and the second switch.
[0104] Explanation with reference to the flow diagram shown in FIG6 :
[0105] Step 601: The first processor generates M data, where M is an integer greater than or equal to 1.
[0106] In one possible implementation, the first processor includes one or more Map tasks. When the first processor generates M data, specifically, the first processor may generate a data set while executing a Map task, and then divide the data set into M data according to the total number M of Reduce tasks in the data processing system.
[0107] Refer to FIG7 , which shows another exemplary MapReduce process flow diagram: Map task a and Map task b, as well as Reduce task a and Reduce task b, are running on the first processor, and Reduce task c is running on the second processor. Furthermore, the first processor generates data set 1 during the execution of Map task a, and then divides the generated data set 1 into three parts (i.e., data a to data c) according to the total number of Reduce tasks 3, wherein data a to data c are the data read by Reduce tasks a to Reduce tasks c, respectively; the first processor generates data set 2 during the execution of Map task b, and divides the generated data set 2 into three parts (i.e., data d to data f), wherein data d to data f are the data read by Reduce tasks a to Reduce tasks c, respectively.
[0108] In step 602 , the first processor stores M data into a memory pool via a cache of the first processor according to a protocol corresponding to the communication bus.
[0109] In a possible implementation, the first processor stores M data in a cache of the first processor during execution of the Shuffle task, and then stores the M data in the cache of the first processor to a memory pool.
[0110] In one possible implementation, the first processor determines a target address in the memory pool for storing one of the M data (referred to as the first data). Based on different storage architectures, the first processor determines the target address in different ways:
[0111] In the storage-computing integrated architecture:
[0112] The memory pool is a first extended memory in the first computing node; or the memory pool includes the first extended memory in the first computing node and the second extended memory in the second computing node.
[0113] The first processor can be used to manage the memory pool. Specifically, the first processor maintains a record of data stored in the memory pool (referred to as a first record). The first record is used to indicate which address segments in the memory pool are occupied (i.e., have stored data) and which address segments are unoccupied (i.e., have not stored data). It is understood that the unoccupied address segments are the remaining storage space in the memory pool. Optionally, the first record is also used to indicate the identifier of the data stored in the occupied address segments.
[0114] In one possible approach, the first record includes multiple address segments and usage information for the multiple address segments. The usage information for an address segment can be either occupied or unoccupied. For example, the first record includes address segments 1 to 6, where address segments 1 to 5 are occupied, while address segment 6 is unoccupied.
[0115] Optionally, when the address segment is already occupied, the first record may also include an identifier of the data stored in the address segment. In the above example, the first record also includes identifiers of data 1 to data 5 corresponding to address segments 1 to 5, respectively. That is, data 1 is stored in address segment 1, data 2 is stored in address segment 2, and so on.
[0116] In another possible manner, the first record includes a correspondence between an address segment and a data identifier of data stored in the address segment, that is, the address not recorded in the first record has not been occupied.
[0117] For example, the first record includes address segments 1 to 5 and identifiers of data 1 to 5 corresponding to address segments 1 to 5, that is, data 1 is stored in address segment 1, data 2 is stored in address segment 2, and so on.
[0118] Furthermore, an address segment can be specifically identified by a start address and an end address. The length from the start address to the end address in the address segment is the length of the data in the address segment. For example, if data 1 is stored in address segment 1, the start address of address segment 1 is A1, the end address is A2, and the length between A1 and A2 is L, then the length of data 1 is L.
[0119] Alternatively, an address segment can be specifically identified by a starting address and an address length, where the address length is the length of the data in the segment. For example, if data 1 is stored in address segment 1, the starting address of address segment 1 is A1, and the address length is L, i.e., the length of data 1 is L. Data 1 is stored from A1 to A1+L.
[0120] The target address may specifically be a starting address in an address segment (recorded as a target address segment) used to store the first data. The target address is determined by the first processor according to the first record.
[0121] In one possible approach, the first processor determines the address of the remaining storage space in the memory pool based on the first record, that is, determines the address segment of the unstored data. The first processor then determines the target address based on the address segment of the unstored data, and then determines the target address segment based on the target address and the length of the first data. Optionally, after determining the target address and the target address segment, the first processor updates the first record of the first processor.
[0122] Optionally, the first record may be stored in a cache of the first processor, or in a first local memory. Optionally, after determining the target address and / or target address segment, the first processor generates update information for the first record and updates the first record according to the update information.
[0123] In storage separation architecture 1:
[0124] The memory pool is a storage node (such as a hard disk in a storage node). The storage node can be used to manage the memory pool, that is, a first record is maintained in the storage node. For the description of the first record, please refer to the relevant description in the above-mentioned storage and computing integrated architecture. The first record can be saved in the cache of the storage node, or saved in the local memory of the storage node.
[0125] The first processor may request the target address from the storage node in two possible ways as follows:
[0126] In one possible manner, the first processor sends an address request to the storage node, and optionally, the address request includes the length of the first data. Accordingly, the storage node receives the address request from the first processor, and in response to the address request, determines the address of the remaining storage space in the memory pool according to the first record, that is, determines the address segment of the unstored data, and the storage node then determines the target address based on the address segment of the unstored data. For the specific determination method, please refer to the relevant description in the above-mentioned storage and computing integrated architecture. After determining the target address, the storage node may send an address response to the first processor, and the address response includes the target address. Optionally, the first processor receives the address response from the storage node, obtains the target address from the address response, and determines the target address segment according to the target address and the length of the first data; or, the storage node determines the target address segment according to the target address and the length of the first data, and sends an address response to the first processor, and the address response includes the target address segment.
[0127] In another possible embodiment, a second record is maintained in the first processor, and the second record includes a record of the first processor storing data in the memory pool. Optionally, the second record includes multiple address segments, and the data stored in the multiple address segments are all written to the memory pool by the first processor. In addition, the second record may also include data identifiers of the data stored in the multiple address segments. For example, the second record includes address segment 1 and address segment 2, as well as identifiers of data 1 and data 2 corresponding to address segment 1 and address segment 2, respectively. It can be understood that data 1 is written by the first processor to address segment 1, and data 2 is written by the first processor to address segment 2. Optionally, the second record can be stored in the cache of the first processor, or in the first local memory.
[0128] The first processor can determine the address of the remaining storage space in the memory pool that has not been written by the first processor based on the second record, and then determine the address to request from the storage node based on the address of the remaining storage space that has not been written by the first processor. The first processor sends an address request to the storage node, and the address request includes the address requested by the first processor to the storage node. In response, the storage node determines whether the address requested by the first processor is already occupied by another computing node based on the first record:
[0129] If the storage node determines, based on the first record, that the address requested by the first processor is not occupied by another computing node, it sends an address response to the first processor, which can be used to indicate that the first processor can use the requested address. It is understood that the address requested by the first processor is the target address.
[0130] If the storage node determines, based on the first record, that the address requested by the first processor is already occupied by another computing node, it sends an address response to the first processor, indicating that the first processor cannot use the requested address. Furthermore, the storage node can determine a target address based on the first record and send the target address to the first processor, with the address response including the target address. Alternatively, after receiving the address response, the first processor determines that the requested address is already occupied by another computing node, and then determines a new address based on the first record and requests the new address from the storage node, until the first processor obtains the target address.
[0131] In addition, when the first processor determines that the address it requested has been occupied by other computing nodes, it can update the second record. In this way, the second record includes not only the record of the first processor storing data in the memory pool, but also the record of other computing nodes storing data in the memory pool.
[0132] Optionally, after determining the target address, the first processor updates the second record of the first processor based on the target address and the length of the first data; or, after determining the target address segment, the first processor updates the second record of the first processor based on the target address segment. For specific update methods, please refer to the relevant instructions in the above-mentioned storage and computing integrated architecture. Optionally, after determining the target address, the storage node updates the first record of the storage node based on the target address and the length of the first data; or, after determining the target address segment, the storage node updates the first record of the storage node based on the target address segment. For specific update methods, please refer to the relevant instructions in the above-mentioned storage and computing integrated architecture.
[0133] In addition, in the storage separation architecture 1, not only can the first processor maintain the second record, but other computing nodes can also maintain their own records of storing data in the memory pool. Furthermore, when determining the target address, the first processor can also broadcast an address request in the system based on the second record, and the address request includes the address requested by the first processor. Correspondingly, other computing nodes in the system can receive the address request and determine whether the address requested by the first processor has been occupied by themselves based on the records maintained in their respective computing nodes. If so, an address response is sent to the first processor, and the address response is used to indicate that the first processor cannot use the address it requested. When the first processor does not receive an address response, the address requested in the address request is determined as the target address. When the first processor receives an address response sent by one or more other computing nodes, it re-determines a new request address and broadcasts the address request again until the target address is determined.
[0134] Alternatively, in other examples, other computing nodes in the system may further send an address response to the first processor after determining that the address requested by the first processor is not occupied by themselves. The address response is used to indicate that the address requested by the first processor is not occupied by the initiator of the address response. Further, after the first processor determines that it has received address responses sent by all other computing nodes in the system, it determines the address requested in its address request as the target address. After the first processor determines that it has not received address responses sent by all other computing nodes in the system, it re-determines a new request address and broadcasts the address request again until the target address is determined.
[0135] In addition, the first processor may periodically synchronize the first record from the storage node, or after generating M data (or the first data), synchronize the first record from the storage node, so that the first processor can determine the target address based on the first record. After determining the target address, the first processor sends the target address (or the target address and the identifier of the first data) to the storage node, and the storage node updates the first record.
[0136] Of course, there may be other implementation methods, and this application will not give examples one by one.
[0137] In conjunction with the structural diagram of the computing node shown in Figure 5b, the first processor includes not only the first processor but also the first switch. After generating the first data, the first processor can send an address request to the storage node through the first switch, and the first processor can receive an address response from the storage node through the first switch, and determine the target address (or target address segment) based on the address response. Alternatively, after generating the first data, the first processor can send an address request to other computing nodes through the first switch, and the first processor can receive an address response from other computing nodes through the first switch, and determine the target address (or target address segment) based on the address response.
[0138] In storage separation architecture 2:
[0139] The memory pool is a storage node (such as a hard disk in a storage node), and the management node can be used to manage the memory pool, that is, a first record is maintained in the management node. For the description of the first record, please refer to the relevant description in the above-mentioned storage and computing integrated architecture. The first record can be saved in the cache of the management node, or saved in the local memory of the management node.
[0140] The first processor may request the target address from the management node, or the first processor may request the first record from the management node and determine the target address based on the first record. For specific implementation methods, please refer to the description in the above-mentioned storage separation architecture 1, and "storage node" can be replaced with "management node" for understanding.
[0141] The management node may also update the first record after determining the target address or after receiving the target address (or the target address and the identifier of the first data) from the first processor. For specific implementation methods, please refer to the description in the above-mentioned storage separation architecture 1. "Storage node" can be replaced with "management node" for understanding.
[0142] 5c shows a schematic diagram of the computing node structure. The first processor includes not only the first processor but also a first switch. After generating the first data, the first processor can send an address request to the management node via the first switch. The first processor can also receive an address response from the management node via the first switch and determine a target address (or target address segment) based on the address response.
[0143] It can be understood that the management node has strong computing power and network IO capabilities. Compared with the above-mentioned storage separation architecture 1, the storage separation architecture 2 transfers the interaction between the computing node and the storage node to the interaction between the computing node and the management node, which helps to reduce the hardware cost of the storage node and improve the access efficiency of the computing node.
[0144] After determining the target address, the first processor may store the first data in the target address of the memory pool via the cache of the first processor according to the protocol corresponding to the communication bus. In a specific implementation, the first processor first writes the first data and the target address into the cache of the first processor, and then, based on the protocol corresponding to the communication bus, the first processor writes the first data in the cache of the first processor to the target address in the memory pool. Specifically, the first processor uses the target address as the starting address and writes the first data into the memory pool, that is, writes the first data into the target address segment. Furthermore, when the protocol corresponding to the communication bus is the CXL protocol, the first processor may write the first data in the cache of the first processor to the target address in the memory pool based on store memory semantics. Optionally, after storing the first data in the memory pool via the cache of the first processor, the first processor may also write the target address to the first local memory. In one possible manner, the first processor writes metadata corresponding to the first data into the first local memory, and the metadata corresponding to the first data may include the target address, the length of the first data, the backup information of the first data, etc.
[0145] In one possible implementation, after writing the M data into the memory pool, the first processor may further merge the M data, such as performing an aggregation operation on the M data, thereby helping to reduce the storage space occupied by the M data in the memory pool.
[0146] In step 603, the second processor obtains N data from the memory pool and stores them in the cache of the second processor according to the protocol corresponding to the communication bus, wherein the M data include N data, and N is a positive integer.
[0147] In one possible implementation, the second processor may include one or more Reduce tasks. When the second processor obtains N data from the memory pool, specifically, when the second processor executes a Reduce task, the second processor obtains N data required to execute the Reduce task from the memory pool.
[0148] Furthermore, when the protocol corresponding to the communication bus is the CXL protocol, the second processor may read N data from the memory pool into the cache of the second processor based on load memory semantics.
[0149] In one possible implementation, before the second processor obtains N data from the memory pool and stores them in the cache of the second processor, the second processor first obtains the target address of each data in the N data from the first local memory through the protocol corresponding to the communication bus. The following is an example of the first data in the N data:
[0150] In one possible approach, the second processor polls the local memory of each computing node and, after obtaining the target address of the first data from the first local memory, stops accessing the local memories of subsequent computing nodes. For example, a data processing system includes 10 computing nodes. The second processor first queries the local memories of five computing nodes and fails to obtain the target address of the first data. Subsequently, the second processor queries the local memory of the sixth computing node (i.e., the first computing node) and obtains the target address of the first data. Therefore, the second processor stops querying the local memories of the remaining four computing nodes.
[0151] In another possible embodiment, the second processor accesses the first record or the second record to determine the target address of the first data. Specifically, as follows:
[0152] In the storage and computing integrated architecture, the second processor can access the first record and read the target address of the first data from the first record.
[0153] In the storage separation architecture 1, the second processor can access the first record in the storage node and determine the target address of the first data from the first record, or the second processor can access the second record (or the first record) in the first computing node and determine the target address of the first data from the second record (or the first record). Alternatively, the second processor sends a query request to the storage node, and the query request includes an identifier of the first data. Accordingly, the storage node queries the first record based on the identifier of the first data in the query request, determines the target address of the first data, and sends a query response to the second processor, and the query response includes the target address of the first data. Accordingly, the second processor receives the query response and obtains the target address of the first data from the query response.
[0154] In storage separation architecture 2, the second processor can access the first record in the management node and determine the target address of the first data from the first record, or the second processor can access the second record (or the first record) in the first computing node and determine the target address of the first data from the second record (or the first record). Alternatively, the second processor sends a query request to the management node, and the query request includes an identifier of the first data. Accordingly, the management node queries the first record based on the identifier of the first data in the query request, determines the target address of the first data, and sends a query response to the second processor, and the query response includes the target address of the first data. Accordingly, the second processor receives the query response and obtains the target address of the first data from the query response.
[0155] Step 604: The second processor processes the N data.
[0156] In one possible implementation, while executing a Reduce task, the second processor obtains N data from the memory pool and then performs computation on the N data. For example, in FIG7 , while executing Reduce task c, the second processor reads data c and data f from the memory pool and then performs computation on data c and data f.
[0157] It should also be added that in the above step 601, the first processor also generates MN data to be read by the first processor. Specifically, the first processor also runs a Reduce task, which can be used to read the MN data and process the MN data. In conjunction with the example in Figure 7, the first processor also runs Reduce task a and Reduce task b. During the process of Reduce task a, the first processor also reads data a and data d from the memory pool and performs computational processing on data a and data d. During the process of Reduce task b, the first processor also reads data b and data e from the memory pool and performs computational processing on data b and data e. The specific implementation method of the first processor reading data from the memory pool and processing the data can be found in the description of the above steps 603 and 604.
[0158] In the above technical solution, the first processor writes M data into a memory pool, and the second processor reads N data from the memory pool, wherein the storage space of the memory pool is larger than the storage space of the first local memory, and the storage space of the memory pool is larger than the storage space of the second local memory. This avoids the problem of unnecessary resource waste caused by the first processor writing M data into the first local memory, which has limited storage space, resulting in M data being swapped in and out between the first local memory and the hard disk, and / or the second processor reading N data into the second local memory, resulting in N data being swapped in and out between the second local memory and the hard disk.
[0159] Furthermore, the first processor stores the M data generated by the multiple processor cores in the first processor in the memory pool. Accordingly, the second processor can obtain the N data it needs from the memory pool at one time. Compared with the existing distributed computing process, the present application does not need to obtain N data N times from the memory space corresponding to the multiple processor cores. This helps to improve the efficiency of data acquisition. Moreover, the first processor stores M data in the memory pool via the cache of the first processor through the CXL bus or the UB bus. Compared with the first processor first storing the M data in the first local memory via the cache of the first processor and then storing the M data in the first local memory to the memory pool, the data writing process is reduced once, which helps to improve the efficiency of the first processor writing data to the memory pool; similarly, the second processor directly obtains N data from the memory pool to the cache of the second processor through the CXL bus or the UB bus. Compared with the second processor first reading N data from the memory pool to the second local memory and then reading N data in the second local memory to the cache, the data reading process is reduced once, which helps to improve the efficiency of the second processor reading data from the memory pool. Furthermore, in MapReduce, it helps to improve the processing efficiency of the Shuffle process.
[0160] The benefits of this application are described below from different perspectives:
[0161] (1) Scenario benefits
[0162] For scenarios such as big data computing (such as gene sequencing and weather forecasting), based on the memory pool, data in the node can be written directly to the memory pool through the cache, without having to be written to the local memory through the cache, then written to the hard disk through the local memory, and then written to the local memory through the hard disk. This helps avoid unnecessary swapping of data between the memory and the hard disk, helps improve the performance of the global data shuffle in MapReduce, and effectively accelerates the computing process.
[0163] For scenarios such as single-node computing (in-memory databases, big data computing, and graph computing), based on the memory pool, data in the node can be written directly to the memory pool through the cache, without having to be written to the local memory through the cache first, then written to the hard disk through the local memory, and then written to the local memory through the hard disk, which helps improve computing efficiency.
[0164] (2) Network IO income
[0165] The transmission speed of ultra-low-latency communication protocols (such as the CXL protocol or the UB protocol) is higher than that of RDMA. This application uses ultra-low-latency communication protocols and memory pools to improve the execution efficiency when reading and writing data in big data scenarios.
[0166] (3) Storage income
[0167] Since the memory pool can be installed in the computing node by plugging and unplugging, or as a separate storage node, the memory pool is scalable, which helps to achieve business expansion / contraction in seconds and rapid recovery from downtime. Furthermore, the memory pool also helps to reduce data copies between nodes. For example, in the existing solution, if there are 4 storage nodes, each storage node is 4TB, then the total storage space is 16TB, and each storage node needs to store a data copy, which takes up a total of 4TB. In this application, it can be considered that 4 storage nodes form a memory pool, the storage space of the memory pool is 16TB, and the memory pool stores a data copy, which only takes up 1TB.
[0168] (4) Cost-benefit
[0169] The decoupling of processors and memory pools helps achieve flexible allocation between processors and memory pools.
[0170] In addition, the first processor may determine that there is insufficient storage space in the memory pool to store the first data, and thus store the first data in the first local memory. When the second processor needs to read the first data, the second processor may read the first data from the first local memory into the second local memory. For specific implementation, please refer to the description of the embodiment related to FIG4.
[0171] Furthermore, the first processor may determine that there is insufficient storage space in the memory pool to store the first data in multiple ways:
[0172] In the storage-computing integrated architecture:
[0173] The first processor stores a first record, and determines based on the first record that there is insufficient storage space in the memory pool to store the first data. For example, the first processor determines based on the first record that the remaining storage space in the memory pool is less than the length of the first data.
[0174] In storage separation architecture 1:
[0175] In one possible approach, the first processor sends an address request to the storage node, where the address request includes the length of the first data. In response, the storage node determines, based on the first record, that there is insufficient storage space in the memory pool to store the first data. For example, the storage node determines, based on the first record, that the length of the remaining storage space in the memory pool is less than the length of the first data. Furthermore, the storage node sends an address response to the first processor, where the address response may indicate that there is insufficient storage space in the current memory pool to store the first data.
[0176] In another possible embodiment, when the storage node determines, based on the first record, that the remaining storage space in the memory pool is less than a preset length, the storage node sends a broadcast message indicating that there is insufficient storage space in the current memory pool to store additional data. Accordingly, the first processor determines that there is no need to request the target address from the storage node.
[0177] In storage separation architecture 2:
[0178] In one possible approach, the first processor sends an address request to the management node, where the address request includes the length of the first data. In response, the management node determines, based on the first record, that there is insufficient storage space in the memory pool to store the first data. For example, the management node determines, based on the first record, that the length of the remaining storage space in the memory pool is less than the length of the first data. Furthermore, the management node sends an address response to the first processor, where the address response may indicate that there is insufficient storage space in the memory pool to store the first data.
[0179] In another possible embodiment, when the management node determines, based on the first record, that the remaining storage space in the memory pool is less than a preset length, the management node sends a broadcast message indicating that there is insufficient storage space in the current memory pool to store additional data. Accordingly, the first processor determines that there is no need to request the target address from the storage node.
[0180] In addition, the first processor may also determine that there is insufficient storage space in the first local memory to store the first data, and therefore stores the first data on the first hard disk. When the second processor needs to read the first data, the second processor reads the first data from the first hard disk into the second local memory. The specific implementation can be described in the embodiment related to Figure 4. Further, when the first processor determines that there is insufficient storage space in the first local memory to store the first data, the first processor may maintain a record (referred to as a third record) of the data stored in the first local memory. The third record is used to indicate which address segments in the first local memory are occupied (i.e., have data stored) and which address segments are unoccupied (i.e., have no data stored). It can be understood that the unoccupied address segments are the remaining storage space in the first local memory. Optionally, the third record is also used to indicate the identifier of the data stored in the occupied address segments. The specific implementation of the third record can be described in the first record above. Further, based on the third record, the first processor determines that there is insufficient storage space in the first local memory to store the first data. For example, based on the third record, the first processor determines that the length of the remaining storage space in the first local memory is less than the length of the first data.
[0181] It is understood that after generating data, the first processor can write the data based on the following priority: memory pool, first local memory, and first hard disk. Because the storage space of the memory pool is larger than the storage space of the first local memory, the first processor and the second processor can transfer data based on the memory pool, which helps reduce unnecessary data exchange. Furthermore, because the access latency of the first local memory is lower than the access latency of the first hard disk, when the first processor determines that there is insufficient storage space in the memory pool to store the first data, the first processor will prioritize storing the first data in the first local memory, which helps reduce access latency.
[0182] Based on the same inventive concept mentioned above, the present application provides a computer-readable storage medium, which stores a computer program or instruction. When the computer program or instruction is executed by a device, it implements the method of the first computing node in the method embodiment related to Figure 6 above, or implements the method of the second computing node in the method embodiment related to Figure 6 above.
[0183] Based on the same inventive concept mentioned above, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a computing device, it implements the method of the first computing node in the method embodiment related to Figure 6 above, or implements the method of the second computing node in the method embodiment related to Figure 6 above.
[0184] Based on the same inventive concept mentioned above, the present application provides a computing device, including a processor, the processor is connected to a memory, the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the computing device implements the method of the first computing node in the method embodiment related to Figure 6 above, or implements the method of the second computing node in the method embodiment related to Figure 6 above.
[0185] In the present application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can be represented by: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of the present application, the character " / " indicates that the previous and next associated objects are in a "division" relationship.
[0186] It is understood that the various numbers used in the embodiments of this application are merely for ease of description and are not intended to limit the scope of the embodiments of this application. The order of the sequence numbers of the above-mentioned processes does not necessarily imply a specific order of execution; the order of execution of the processes should be determined by their functions and inherent logic.
[0187] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the scope of protection of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.
Claims
1. A data processing system, characterized in that: include: A first computing node including a first processor, the first processor being connected to a memory pool via a communication bus of the first computing node; a second computing node, comprising a second processor, the second processor being connected to the memory pool via a communication bus of the second computing node; The first processor is used to generate M data, and store the M data into the memory pool via the cache of the first processor according to the protocol corresponding to the communication bus; The second processor is used to obtain N data from the memory pool to the cache of the second processor according to the protocol corresponding to the communication bus, and process the N data, wherein the M data include the N data, and M and N are both positive integers.
2. The system according to claim 1, characterized in that The communication bus is a Computing Express Link CXL bus or a Unified Bus UB.
3. The system according to claim 1, characterized in that The first processor is specifically configured to generate the M data in a process of executing a Map task, and to store the M data in the memory pool in a process of executing a Shuffle task; During the process of executing the Reduce task, the second processor obtains the N data required to execute the Reduce task from the memory pool.
4. The system according to claim 1, characterized in that The memory pool is a storage node; The first computing node includes a first switch, and the first processor and the first switch are both connected to a communication bus of the first computing node; The second computing node includes a second switch, and the second processor and the second switch are both connected to a communication bus of the second computing node; The storage node is connected to the first computing node via the first switch, and the storage node is connected to the second computing node via the second switch.
5. The system according to claim 1, characterized in that The memory pool is a first extended memory in the first computing node, the first computing node further includes a first switch, and the first processor, the first extended memory and the first switch are all connected to a communication bus of the first computing node; The second computing node includes a second switch, and the second processor and the second switch are connected to a communication bus of the second computing node; The second switch is connected to the first switch, and the second processor is connected to the first extended memory through the first switch and the second switch.
6. The system according to claim 1, wherein: The memory pool includes a first extended memory in the first computing node and a second extended memory in the second computing node; The first computing node also includes a first switch, and the first processor, the first extended memory and the first switch are all connected to a communication bus of the first computing node; The second computing node also includes a second switch, and the second processor, the second extended memory and the second switch are all connected to a communication bus of the second computing node; The second switch is connected to the first switch, the second processor is connected to the first extended memory through the first switch and the second switch, and the first processor is connected to the second extended memory through the first switch and the second switch.
7. The system according to any one of claims 1 to 6, characterized in that: The first computing node further includes a local memory, the local memory is also connected to the communication bus of the first computing node, and the local memory is used to store metadata of the M data; When the second processor obtains the N data from the memory pool to the cache of the second processor, it is specifically used to: obtain metadata of the N data from the local memory of the first computing node, and obtain the N data from the memory pool according to the metadata of the N data.
8. A data processing method, characterized in that: Applicable to a data processing system comprising a first computing node and a second computing node; the first computing node comprises a first processor, and the first processor is connected to a memory pool through a communication bus of the first computing node; The second computing node includes a second processor, the second processor being connected to the memory pool via a communication bus of the second computing node; The method comprises: The first processor generates M data, and stores the M data into the memory pool via the cache of the first processor according to a protocol corresponding to the communication bus; The second processor obtains N data from the memory pool to the cache of the second processor according to the protocol corresponding to the communication bus, and processes the N data, wherein the M data include the N data, and M and N are both positive integers.
9. The method according to claim 8, characterized in that The communication bus is a Computing Express Link CXL bus or a Unified Bus UB.
10. The method according to claim 8, characterized in that The first processor generates M data, and stores the M data into the memory pool via the cache of the first processor according to a protocol corresponding to the communication bus, including: The first processor generates the M data in the process of executing the mapping task; The first processor stores the M data into the memory pool according to a protocol corresponding to the communication bus during the execution of the shuffle task; The second processor obtains N data from the memory pool to the cache of the second processor according to the protocol corresponding to the communication bus, and processes the N data, including: During the process of executing the Reduce task, the second processor obtains the N data required to execute the Reduce task from the memory pool according to the protocol corresponding to the communication bus, and processes the N data.
11. The method according to any one of claims 8 to 10, characterized in that: The first computing node further includes a local memory, the local memory is also connected to the communication bus of the first computing node, and the local memory is used to store metadata of the M data; The second processor obtains the N data from the memory pool to a cache of the second processor, comprising: The second processor obtains metadata of the N data from the local memory of the first computing node, and obtains the N data from the memory pool according to the metadata of the N data.
Citation Information
Patent Citations
Method and system for MapReduce data transmission in data center having SAN
CN102209087A
Data processing method, apparatus and system
CN108027801A
Data processing method, computing device and computer system
CN115933975A
System and method for remote direct memory access over a network switch fabric
US20050238035A1
Cited By
Storage and calculation separated storage system and method for improving network transmission efficiency of system
CN119835267A
