A method, device, medium, and program product for data transmission between computing nodes.

Abstract

This application discloses a method, device, medium, and program product for data transmission between computing nodes, relating to the field of data processing technology. The method is applied to a first computing node in a stream computing system. Both the first and second computing nodes in the stream computing system include a remote direct memory access (RDA) network interface card (NIC) and a high-speed computing link device supporting input / output sub-protocols and memory sub-protocols of a high-speed computing link protocol. The high-speed computing link device is used to store data. The method includes: obtaining a data request sent by the second computing node through the RDA; accessing the high-speed computing link device according to the data request via the high-speed computing link protocol to obtain the target data corresponding to the data request; and sending the target data to the RDA of the second computing node through the RDA. This method can improve the data transmission speed between computing nodes and save processor resources.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, device, medium, and program product for data transmission between computing nodes. Background Technology

[0002] Stream computing systems, or streaming data stream execution engines, provide distributed data storage, data communication, and fault tolerance mechanisms for distributed computing of data streams. In related technologies, stream computing systems typically include compute nodes and management nodes. Compute nodes execute data computation tasks, while management nodes handle task scheduling and resource management. Data transmission between compute nodes usually employs a TCP / IP network stack for communication. After sending data, the sending compute node needs to store the data in a user-space cache, copy it to the operating system kernel cache, then to the network device cache, and finally transmit it over the network to the receiving compute node's network device cache. The data is then copied to the receiving node's operating system kernel cache and finally to the user-space cache of the data stream processing application, thus completing the data sending and receiving process. This data transmission process involves multiple copying operations, increasing latency and reducing data transmission speed. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, device, medium, and program product for data transmission between computing nodes, which can improve the data transmission speed between computing nodes. The specific solution is as follows:

[0004] In a first aspect, this application discloses a method for data transmission between computing nodes, applied to computing nodes in a stream computing system. The computing node includes a remote direct memory access network interface card (NIC) and a high-speed computing link device supporting input / output sub-protocols and memory sub-protocols of a high-speed computing link protocol. The high-speed computing link device is used to store data. The method includes:

[0005] The data request sent by the second computing node is obtained through the remote direct memory access network card;

[0006] According to the data request, access the high-speed computing link device through the high-speed computing link protocol to obtain the target data corresponding to the data request;

[0007] The target data is sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card.

[0008] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0009] The high-speed computing link device is mapped between its physical address space and logical address space to obtain a first address mapping table; the first address mapping table contains the mapping relationship between the linear address and physical address of the high-speed computing link device.

[0010] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0011] A second address mapping table is created in the memory region corresponding to the remote direct memory access network card; the second address mapping table contains the mapping relationship between the virtual address of the remote direct memory access network card and the linear address of the high-speed computing link device.

[0012] Optionally, obtaining the data request sent by the second computing node through the remote direct memory access network card includes:

[0013] The virtual address for reading the target data corresponding to the data request is determined using the remote direct memory access network card, and the linear address corresponding to the virtual address is determined by querying the second address mapping table.

[0014] According to the data request, access the high-speed computing link device via the high-speed computing link protocol to obtain the target data corresponding to the data request, including:

[0015] The physical address corresponding to the linear address is determined by querying the first address mapping table;

[0016] Access the high-speed computing link device via the high-speed computing link protocol based on the physical address to read the target data corresponding to the data request.

[0017] Optionally, sending the target data to the remote direct memory access network interface card (RDBIC) of the second computing node via the RDBIC includes:

[0018] The remote direct memory access network card is used to determine the virtual address for writing the target data;

[0019] The target data and the write virtual address are sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card, so that the remote direct memory access network card of the second computing node can query the address mapping table stored in the second computing node according to the write virtual address, determine the physical address corresponding to the write virtual address, and write the target data to the physical address corresponding to the write virtual address.

[0020] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0021] The device endpoints of the high-speed computing link device are connected via the system bus and the root port of the high-speed computing link device.

[0022] The device information of the high-speed computing link device is obtained by scanning, and the high-speed computing link device is registered to the system memory according to the device information. Physical address space is allocated for the base address register and internal memory of the high-speed computing link device.

[0023] Optionally, the inter-node data transmission method further includes:

[0024] Get read / write operation instructions;

[0025] Access the internal memory corresponding to the high-speed computing link device, and forward the read / write operation instructions to the root port of the high-speed computing link device through the internal memory;

[0026] The root port converts the read / write operation command into a link control unit packet, and sends the link control unit packet to the endpoint of the high-speed computing link device.

[0027] The high-speed computing link controller performs address resolution on the link control unit packet and performs read / write operations on the memory of the high-speed computing link device based on the resolved physical address.

[0028] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0029] Configure a network interface card that supports the Remote Direct Memory Access Protocol (RDP) on the computing node, and create an RDP interface in the data transmission interface of the stream computing system to add the RDP network interface card to the computing node.

[0030] Optionally, obtaining the data request sent by the second computing node through the remote direct memory access network card includes:

[0031] Obtain the link establishment request sent by the second computing node, and establish a remote direct memory access channel with the second computing node according to the link establishment request;

[0032] The data request is obtained through the remote direct memory access channel, and the target data is sent through the remote direct memory access channel.

[0033] Optionally, obtaining the data request sent by the second computing node through the remote direct memory access network card includes:

[0034] The data request sent by the second computing node is obtained through the remote direct memory access network card; the data request includes free capacity.

[0035] According to the data request, access the high-speed computing link device via the high-speed computing link protocol to obtain the target data corresponding to the data request, including:

[0036] According to the data request, the high-speed computing link device is accessed through the high-speed computing link protocol, and the target sub-data with a data size equal to the free capacity is read from the target data corresponding to the data request.

[0037] Optionally, the inter-node data transmission method further includes:

[0038] Obtain a new data request sent by the second computing node; the new data request is sent by the second computing node after it has obtained new free capacity;

[0039] According to the new data request, the high-speed computing link device is accessed through the high-speed computing link protocol, and target sub-data with a data size equal to the new free capacity is read from the remaining target data corresponding to the data request.

[0040] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0041] Obtain data production tasks and execute the data generation tasks to produce new data;

[0042] The new data is serialized according to the high-speed computing link protocol to obtain the processed data.

[0043] The processed data is written to the high-speed computing link device, and storage information corresponding to the processed data is generated.

[0044] The storage information is sent to the management node, so that the management node sends the storage information to the second computing node, and the second computing node sends the data request based on the storage information.

[0045] Secondly, this application discloses an electronic device, including:

[0046] Memory, used to store computer programs;

[0047] A processor is used to execute the computer program to implement the aforementioned method for data transmission between computing nodes.

[0048] Thirdly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned method for data transmission between computing nodes.

[0049] Fourthly, this application discloses a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for data transmission between computing nodes.

[0050] This application discloses a data transmission method between computing nodes, applied to computing nodes in a stream computing system. The computing node includes a remote direct memory access (RDA) network interface card (NIC) and a high-speed computing link device supporting input / output sub-protocols and memory sub-protocols of a high-speed computing link protocol. The high-speed computing link device is used to store data. The method includes: obtaining a data request sent by a second computing node through the RDA; accessing the high-speed computing link device according to the data request via the high-speed computing link protocol to obtain the target data corresponding to the data request; and sending the target data to the RDA of the second computing node through the RDA. It is evident that by adding an RDA and a high-speed computing link device to the computing node of the stream computing system, data operations within the computing node are implemented through the high-speed computing link protocol, improving the speed of data reading and writing within the node and avoiding multiple data copying operations required when the operating system switches between different execution modes. By combining the RDA and the high-speed computing link protocol, direct data transmission within the high-speed computing link device between computing nodes can be achieved, improving transmission speed and saving processor resources. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 A flowchart of a data transmission method between computing nodes provided in this application;

[0053] Figure 2 This application provides a specific schematic diagram of data transmission within a computing node;

[0054] Figure 3 This application provides a specific schematic diagram of data transmission between computing nodes;

[0055] Figure 4 This application provides a schematic diagram of the structure of a data transmission device between computing nodes.

[0056] Figure 5 This application provides a structural diagram of an electronic device. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] In existing technologies, stream computing systems typically include compute nodes and management nodes. Compute nodes execute data computation tasks, while management nodes handle task scheduling and resource management. Data transmission between compute nodes usually employs a TCP / IP (Transmission Control Protocol / Internet Protocol) network stack for communication. After sending data, the sending compute node needs to store the data in a user-space buffer, copy it to the operating system kernel buffer, then copy it to the network device buffer, transmit it over the network to the receiving compute node's network device buffer, copy it to the receiving node's operating system kernel buffer, and finally copy it to the user-space buffer of the data stream processing application. This completes the data sending and receiving process. In other words, the data transmission process involves multiple copying operations, increasing data transmission latency, reducing data transmission speed, and requiring processor intervention. To overcome these technical problems, this application proposes a data transmission method between compute nodes that can improve the data transmission speed between compute nodes in a stream computing system and save processor resources.

[0059] This application discloses a data transmission method between computing nodes, applied to computing nodes in a stream computing system. The computing node includes a remote direct memory access network interface card (NIC) and a high-speed computing link device supporting input / output sub-protocols and memory sub-protocols of a high-speed computing link protocol. The high-speed computing link device is used to store data. See also... Figure 1 As shown, the method may include the following steps:

[0060] Step S11: Obtain the data request sent by the second computing node through the remote direct memory access network card.

[0061] In this embodiment, the first computing node obtains data requests sent by the second computing node in the stream computing system through a remote direct memory access network card. Of course, the second computing node can also receive data requests sent by the first computing node. The stream computing system may also include multiple computing nodes of the same type as the first and second computing nodes. The same type means that the computing nodes all contain their own remote direct memory access network cards and high-speed computing link devices that support input / output sub-protocols and memory sub-protocols of high-speed computing link protocols.

[0062] Before actual data transmission, a Remote Direct Memory Access (RDMA) network interface card (NIC) needs to be added to the compute nodes. Specifically, this is achieved by configuring NICs that support the RDMA protocol on the compute nodes and creating an RDMA interface in the data transmission interface of the stream computing system. In other words, this involves configuring a NIC driver that supports the RDMA protocol on each compute node and developing an interface to call the RDMA protocol in the data transmission interface of the stream computing engine.

[0063] Simultaneously, before data transmission, it is necessary to add a high-speed computing link device to the computing node that supports the input / output sub-protocol and memory sub-protocol of the ComputeExpress Link (CXL) protocol. This includes driver software for device registration, memory space allocation, read / write operations, and other functions, so that the node can support data read / write operations using the high-speed computing link protocol. The devices corresponding to the high-speed computing link protocol include three types: Type 1, Type 2, and Type 3. In this embodiment, the high-speed computing link device is specifically a Type 3 device that supports the input / output sub-protocol (CXL.io) and memory sub-protocol (CXL.mem) of the high-speed computing link protocol. It is a dedicated storage device that communicates directly with the host processor and can use the CXL protocol to achieve low-latency, high-throughput data transmission; it can also be used as a memory buffer to expand memory bandwidth and capacity.

[0064] Specifically, the system bus and the root port of the high-speed computing link device connect to the device endpoint of the high-speed computing link device. Device information of the high-speed computing link device is obtained through scanning, and the device is registered in system memory based on this information. Physical address space is allocated for the high-speed computing link device's base address register and internal memory. That is, each compute node's CPU has at least one CXL root port (RPs) configured on its system bus, and CXL device endpoints (EPs) are connected through these RPs. The compute node calls the operating system kernel driver to initiate PCIe transactions (PCIe (peripheral component interconnect express, high-speed serial computer extension bus standard) transactions) to scan the CXL device's base address register (BAR) and its internal memory (HDM). Based on the scan information, the high-speed computing link device (CXL.mem device) is registered in system memory within the node, and physical address space is allocated for the BAR and HDM in reserved memory space. The high-speed computing link device's memory is then initialized and its address space is allocated.

[0065] In some embodiments, the method further includes: obtaining read / write operation instructions; accessing the internal memory corresponding to the high-speed computing link device, and forwarding the read / write operation instructions to the root port of the high-speed computing link device through the internal memory; converting the read / write operation instructions into link control unit packets through the root port, and sending the link control unit packets to the endpoints of the high-speed computing link device; resolving the address of the link control unit packets through the high-speed computing link controller, and performing read / write operations on the memory of the high-speed computing link device according to the resolved physical address.

[0066] Understandably, further steps are needed to synchronize and map the CXL logical address space data with the physical address space of the CXL device to obtain a mapping table between the virtual and physical addresses of the high-speed computing link device. Finally, the high-speed computing link device information and address space data are written into the node's operating system configuration file so that the compute node can read the configuration information of the high-speed computing link device when it restarts. When it receives a data read / write operation request, it accesses the HDM memory of the high-speed computing link device through memory data read instructions (load) / data write instructions (store). The request is passed to the corresponding root port, which converts the request into a CXL link control unit packet (CXL flit) and sends it to the high-speed computing link device endpoint by calling the CXL protocol. Then, the CXL controller in the CXL device parses the CXL flit, converts the incoming address, and sends the converted request to the underlying memory controller to realize the read and write of the CXL device's memory resources.

[0067] In some embodiments, obtaining a data request sent by a second compute node via a remote direct memory access (RDA) network interface card (NIC) may include: obtaining a link establishment request sent by the second compute node; establishing a RDA channel with the second compute node based on the link establishment request; obtaining the data request through the RDA channel; and sending the target data through the RDA channel. That is, based on the RDAs on the two compute nodes, a RDA channel is established between the two compute nodes, and direct interaction is performed through this channel.

[0068] Step S12: Access the high-speed computing link device via the high-speed computing link protocol according to the data request, and obtain the target data corresponding to the data request.

[0069] In other words, the remote direct memory access network card obtains the request sent by the remote device, and then accesses the high-speed computing link device according to the high-speed computing link protocol to read the target data from the device.

[0070] Before obtaining data requests sent by the second computing node via the Remote Direct Memory Access (RDA) network interface card (NIC), the process includes: mapping the physical address space and logical address space of the high-speed computing link device to obtain a first address mapping table; the first address mapping table contains the mapping relationship between the linear address and physical address of the high-speed computing link device. A second address mapping table is created in the memory region corresponding to the RDA; the second address mapping table contains the mapping relationship between the virtual address of the RDA and the linear address of the high-speed computing link device. In other words, by adding a mapping table between the virtual address of the RDA and the linear address of the high-speed computing link device in the memory region corresponding to the RDA in memory, the combination of the Remote Direct Memory Access (RDA) protocol and the high-speed computing link protocol is achieved. That is, the high-speed computing link device is registered as an MR (Memory Region). The specific RDMA driver will create and fill a mapping table in memory from a virtual address (VA) to the linear address (CXL.LA) of the high-speed computing link device. The RDMA network can convert VA to CXL.LA by looking up the table when needed.

[0071] The aforementioned method of obtaining data requests sent by the second computing node via a remote direct memory access network interface card (RCINIC) includes: using the RCINIC to determine the virtual address for reading the target data corresponding to the data request; and querying a second address mapping table to determine the linear address corresponding to the virtual address. The method then accesses a high-speed computing link device via a high-speed computing link protocol to obtain the target data corresponding to the data request, including: querying a first address mapping table to determine the physical address corresponding to the linear address; and accessing the high-speed computing link device via the high-speed computing link protocol based on the physical address to read the target data corresponding to the data request.

[0072] This means that remote devices operate high-speed computing links via RDMA, primarily supporting remote memory operations and remote message operations. Remote memory operations include read and write operations. For example... Figure 2As shown, assume that compute node TM1 initiates an RDMA write operation to compute node TM2, that is, directly writes data to the memory area of ​​the right node; assume that both ends in the diagram have completed the MR registration action and created address mapping tables; the specific process of remote read and write operation is as follows: First, the application system on compute node 1 calls the driver to send a WR (Work Request), which is also known as WQE (Work Queue Element), to the remote direct memory access network card, informing the remote direct memory access network card of the local virtual address used to store the data to be sent, and the virtual address of the peer data to be written; the node maintains SQ (Send Queue) and RQ (Receive Queue). The local remote direct memory access network card queries the first address mapping table, and the high-speed computing link device queries the second address mapping table. Data is read from the memory of the high-speed computing link device through the read instruction (Load instruction), the data packet is assembled and sent out. The remote direct memory access network card (DMI) on compute node 2 receives the data packet, parses out the destination VA, and determines the actual physical address (PA) of the multifunction controller by querying its own first and second address mapping tables. Then, using a write instruction (Store instruction), it stores the data into the memory of the fast computing link device based on the physical address. Throughout the entire remote data write process, compute node 2 does not require the involvement of its CPU, neither for address translation nor for writing to memory.

[0073] In some embodiments, after creating a second address mapping table in the memory region corresponding to the remote direct memory access network interface card (MRNIC), the method further includes: creating a local key for the local computing node, adding a remote key for the qualified remote computing node, and configuring permissions for the memory region corresponding to the remote MRNIC based on the local key and the remote key; correspondingly, obtaining data requests sent by the second computing node through the remote MRNIC includes: obtaining data requests sent by the second computing node through the remote MRNIC; extracting the remote key from the data request, and accessing the high-speed computing link device through the high-speed computing link protocol according to the remote key. That is, during the MR registration process, MR permissions are set simultaneously. MR permissions mainly satisfy the read / write requirements of the local / remote nodes. By designing a local key (L_Key) and a remote key (R_Key) respectively, a key mechanism for accessing the MR is added. During MR registration, both L_Key and R_Key are created simultaneously as needed by the program. Only requests holding L_Key / R_Key keys can operate the MR; the computing node accesses the CXL memory based on whether it has obtained the MR key. Therefore, by setting the MR's operation key during the MR registration process to achieve MR permission management, data transmission speed can be improved while ensuring data integrity.

[0074] In some embodiments, before obtaining the data request sent by the second computing node via the remote direct memory access network card, the method further includes: obtaining a data production task, executing the data production task to generate new data; serializing the new data according to the high-speed computing link protocol to obtain processed data; writing the processed data into the high-speed computing link device and generating storage information corresponding to the processed data; sending the storage information to the management node, so that the management node sends the storage information to the second computing node, and the second computing node sends a data request based on the storage information. That is, after obtaining the data production task issued by the stream computing system, generating the corresponding data, sending the relevant data information to the system's management node, which then sends it to the data consumer, i.e., the aforementioned second computing node, and then the second computing node initiates a data request to the computing node that uploaded the storage information.

[0075] In some embodiments, obtaining a data request sent by a second computing node via a remote direct memory access network interface card (RCI) may include: obtaining the data request sent by the second computing node via the RCI; the data request includes free capacity; and accessing a high-speed computing link device via a high-speed computing link protocol according to the data request to obtain the target data corresponding to the data request, including: accessing the high-speed computing link device via the high-speed computing link protocol according to the data request and reading target sub-data with a data size equal to the free capacity from the target data corresponding to the data request. That is, by sending a data request containing free capacity information, only data of the free capacity size can be sent when providing feedback data, thus avoiding storage failure due to data exceeding the storage capacity, thereby achieving data backpressure transmission.

[0076] Furthermore, the system receives a new data request from the second computing node. This new data request is sent after the second computing node has allocated new free capacity. Based on the new data request, the system accesses the high-speed computing link device via the high-speed computing link protocol and reads the target sub-data with a size equal to the new free capacity from the remaining target data corresponding to the data request. In other words, after the second computing node allocates new space, it initiates another data request. The computing node then reads the target sub-data of the corresponding size from the target data that has never been sent before, based on the new free capacity. This operation is repeated until the target data is completely sent to the second computing node.

[0077] Step S13: Send the target data to the remote direct memory access network card of the second computing node via the remote direct memory access network card.

[0078] After identifying the target data, it is sent to the remote direct memory access (RDA) interface card (NIC) of the second compute node. Specifically, the RDA determines the virtual address to be written to the target data; the target data and the virtual address are then sent to the RDA of the second compute node. The RDA of the second compute node then queries its address mapping table to determine the physical address corresponding to the virtual address and writes the target data to that physical address. Similarly, the second compute node writes the target data to the memory of its high-speed computing link device by querying the address mapping table. Based on the RDA and the high-speed computing link device, the memory and cache of the system application can be expanded, quickly increasing the memory and cache capacity for upper-layer data system processing and computation, and supporting fast, high-capacity data transmission, exchange, and sharing between clusters.

[0079] As can be seen from the above, this embodiment is applied to the computing node of a stream computing system. The computing node includes a remote direct memory access network card (RDBNIC) and a high-speed computing link device that supports input / output sub-protocols and memory sub-protocols of a high-speed computing link protocol. The high-speed computing link device is used to store data. The method includes: obtaining a data request sent by a second computing node through the RDBNIC; accessing the high-speed computing link device according to the data request through the high-speed computing link protocol to obtain the target data corresponding to the data request; and sending the target data to the RDBNIC of the second computing node through the RDBNIC. It is evident that by adding a RDBNIC and a high-speed computing link device to the computing node of the stream computing system, data operations within the computing node are implemented through the high-speed computing link protocol, improving the speed of data reading and writing within the node and avoiding multiple data copying operations required when the operating system switches between different execution modes. By combining the RDBNIC and the high-speed computing link protocol, direct data transmission within the high-speed computing link device between computing nodes can be achieved, improving transmission speed and saving processor resources.

[0080] Taking Flink, a mainstream stream processing system, as an example, Flink is an open-source data stream processing engine composed of two parts: the JobManager (JM) and the TaskManager (TM). Flink uses a master-slave architecture. The JM node runs on the master node, responsible for retrieving submitted tasks from clients and managing the scheduling and resources of the entire Flink cluster. The TM node runs on the slave nodes, responsible for task execution and resource allocation and management for each task on each node. All components communicate using the Akka Framework, including task status and checkpoint triggering information. The stream processing engine involves three types of nodes: clients (responsible for submitting tasks to the cluster's JobManager via Akka connection), the JobManager node, and the TaskManager node.

[0081] For example Figure 3 The diagram illustrates the data transfer process between compute nodes in a Flink stream computing system, describing the data interaction flow between compute nodes in a specific data production-consumption job. This includes two parallel tasks: a data production task running on compute node TaskManager1 (TM1) and a data consumption task running on TaskManager2 (TM2). It should be noted that each TaskManager supports running multiple data production and data consumption tasks. Figure 3The focus is on the data transmission process between the data production task and the data consumption task nodes. Data transmission is represented by solid arrows, while messages between nodes are represented by dashed arrows.

[0082] S21: First, the data production task produces new data (ResultPartition, RP, used in Flink to represent data generated by a task); the new data is serialized according to the CXL protocol; the serialization object (RecordSerializer) selects the corresponding ChannelSelector to distribute the record based on the record processing mode. The RecordWriter (RecordWriter is a component in Flink used to write key-value pairs processed by a task to memory or disk) distributes the record. The ChannelSelector supports broadcast mode and hash mode; in broadcast mode, the record is distributed to each serialization queue; in hash partition mode, the ChannelSelector calculates the hash value on the record and distributes it to the corresponding serialization queue. The record is then written to the high-speed computing link device storage medium. The CXLmemBuffer Pool is a storage device connected via the CXL protocol, which can be organized into a very large storage pool to achieve memory pooling. Figure 3 The BufferWriter is a buffer writer, specifically CXLmemBufferWriter.

[0083] S22: The buffer writer writes the RP record sequence to memory to write the record data to the high-speed computing link device; and performs operations such as RS (ResultSubpartition, subpartition of RP) allocation, RS data space allocation, RP and RS address mapping, data writing, and returning data address to write the data record to the high-speed computing link device.

[0084] S23: RecordWriter receives parameters such as RP, RS2 identifier, storage block list, node, and remote key (R_key) returned by CXLmemBufferWriter, and uploads this data to the task new data list in the JobManager execution graph.

[0085] S24: After receiving the RP and RS2 data reported by TM1, JobManager notifies the consumer of this RP of the message that the RP and RS are ready, including the identifiers of RP and RS2, the list of storage blocks, the list of nodes, etc., and generates the data consumption task in TM2.

[0086] S25: The message sent to TM2 propagates down to the IG (InputGate) that should receive this data. The IG allocates IC2 (InputChannel) to prepare for receiving data. The IG first calls the connection management component (RCXLConnectionManager, i.e., the connection management component between the RDMA network card and the CXL device) to send a link creation request to TM1, which includes the remote key (R_key) and the available free capacity of IC2 for this task.

[0087] S26: After receiving the network request from TM2's IG, TM1 node confirms that the remote key (R_key) is correct, and then selects a combination of RS2.1 storage blocks not exceeding the capacity of IC2 from the RS2 storage block list and returns it to TM2's IG.

[0088] S27: TM2's IG calls RCXLConnectionManager to create an RCXL data transfer channel with TM1. TM1 performs data read and write operations to complete this storage block list data transfer. TM2 consumes data through RecordReader (a component in Flink used for reading and processing data), which utilizes a deserializer object (RecordDeserializer) for deserialization operations.

[0089] S28: After TM2's IG obtains new free space, it creates IC3, packages the list of untransmitted storage blocks from RS2 into a new transmission request, and repeats the S25-S27 process to complete all data transmission.

[0090] By adding remote direct memory access network cards and high-speed computing link devices to the computing nodes of the stream computing system, data operations within the computing nodes are implemented through high-speed computing link protocols, improving the speed of data reading and writing within the nodes and avoiding multiple data copying operations required when the operating system switches between different execution modes. By combining remote direct memory access protocols and high-speed computing link protocols, direct data transmission within high-speed computing link devices between computing nodes can be achieved, which can improve transmission speed and save processor resources.

[0091] Accordingly, this application also discloses a data transmission device between computing nodes, applied to computing nodes in a stream computing system. The computing node includes a remote direct memory access network interface card (NIC) and a high-speed computing link device supporting input / output sub-protocols and memory sub-protocols of a high-speed computing link protocol. The high-speed computing link device is used for data storage. See also... Figure 4 As shown, the device includes:

[0092] Data request acquisition module 11 is used to acquire data requests sent by the second computing node through the remote direct memory access network card;

[0093] Target data determination module 12 is used to access the high-speed computing link device through the high-speed computing link protocol according to the data request, and obtain the target data corresponding to the data request;

[0094] The data transmission module 13 is used to send the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card.

[0095] As can be seen from the above, the data transmission method between computing nodes disclosed in this application is applied to computing nodes in a stream computing system. The computing node includes a remote direct memory access network card (RDBNIC) and a high-speed computing link device supporting input / output sub-protocols and memory sub-protocols of a high-speed computing link protocol. The high-speed computing link device is used to store data. The method includes: obtaining a data request sent by a second computing node through the RDBNIC; accessing the high-speed computing link device according to the data request through the high-speed computing link protocol to obtain the target data corresponding to the data request; and sending the target data to the RDBNIC of the second computing node through the RDBNIC. It is evident that by adding a RDBNIC and a high-speed computing link device to the computing nodes of the stream computing system, data operations within the computing node are implemented through the high-speed computing link protocol, improving the speed of data reading and writing within the node and avoiding multiple data copying operations required when the operating system switches between different execution modes. By combining the RDBNIC and the high-speed computing link protocol, direct data transmission within the high-speed computing link device between computing nodes can be achieved, improving transmission speed and saving processor resources.

[0096] In some specific embodiments, the inter-node data transmission device may specifically include:

[0097] The first mapping unit is used to map the physical address space and logical address space of the high-speed computing link device to obtain a first address mapping table; the first address mapping table contains the mapping relationship between the linear address and physical address of the high-speed computing link device.

[0098] In some specific embodiments, the inter-node data transmission device may specifically include:

[0099] The second mapping unit is used to create a second address mapping table in the memory region corresponding to the remote direct memory access network card; the second address mapping table contains the mapping relationship between the virtual address of the remote direct memory access network card and the linear address of the high-speed computing link device.

[0100] In some specific embodiments, the data request acquisition module 11 may specifically include:

[0101] A linear address determination unit is used to determine the virtual address for reading the target data corresponding to the data request using the remote direct memory access network card, and to determine the linear address corresponding to the virtual address for reading by querying the second address mapping table;

[0102] The target data determination module 12 may specifically include:

[0103] A physical address determination unit is used to determine the physical address corresponding to the linear address by querying the first address mapping table;

[0104] The target data reading unit is used to access the high-speed computing link device through the high-speed computing link protocol according to the physical address, so as to read the target data corresponding to the data request.

[0105] In some specific embodiments, the data transmission module 13 may specifically include:

[0106] A writing unit is used to determine the virtual address for writing the target data using the remote direct memory access network card;

[0107] The sending unit is configured to send the target data and the write virtual address to the remote direct memory access network card of the second computing node through the remote direct memory access network card, so that the remote direct memory access network card of the second computing node can query the address mapping table stored in the second computing node according to the write virtual address, determine the physical address corresponding to the write virtual address, and write the target data to the physical address corresponding to the write virtual address.

[0108] In some specific embodiments, the inter-node data transmission device may specifically include:

[0109] The connection unit is used to connect to the device endpoint of the high-speed computing link device via the system bus and the root port of the high-speed computing link device before obtaining the data request sent by the second computing node through the remote direct memory access network card;

[0110] The scanning unit is used to obtain the device information of the high-speed computing link device by scanning, register the high-speed computing link device to the system memory according to the device information, and allocate physical address space for the base address register and internal memory of the high-speed computing link device.

[0111] In some specific embodiments, the data transmission device between computing nodes may further include:

[0112] The instruction fetching unit is used to fetch read / write operation instructions;

[0113] The instruction forwarding unit is used to access the internal memory corresponding to the high-speed computing link device and forward the read / write operation instructions to the root port of the high-speed computing link device through the internal memory.

[0114] The link control unit packet sending unit is used to convert the read / write operation command into a link control unit packet through the root port, and send the link control unit packet to the endpoint of the high-speed computing link device;

[0115] The address resolution unit is used to resolve the address of the link control unit packet through the high-speed computing link controller, and to perform read / write operations on the memory of the high-speed computing link device according to the resolved physical address.

[0116] In some specific embodiments, the inter-node data transmission device may specifically include:

[0117] The network interface card (NIC) configuration unit is used to configure a NIC supporting the Remote Direct Memory Access (RDA) protocol on the computing node before obtaining a data request sent by the second computing node through the RDA network interface card, and to create an RDA interface in the data transmission interface of the stream computing system, so as to add the RDA network interface card to the computing node.

[0118] In some specific embodiments, the data request acquisition module 11 may specifically include:

[0119] The channel establishment unit is used to obtain the link establishment request sent by the second computing node and establish a remote direct memory access channel with the second computing node according to the link establishment request.

[0120] A data sending unit is configured to acquire the data request through the remote direct memory access channel and send the target data through the remote direct memory access channel.

[0121] In some specific embodiments, the data request acquisition module 11 may specifically include:

[0122] The request acquisition unit is used to acquire a data request sent by the second computing node through the remote direct memory access network card; the data request includes free capacity.

[0123] The target data determination module 12 may specifically include:

[0124] The device access unit is used to access the high-speed computing link device through the high-speed computing link protocol according to the data request, and read the target sub-data with a data size equal to the free capacity from the target data corresponding to the data request.

[0125] In some specific embodiments, the data transmission device between computing nodes may further include:

[0126] The new data request acquisition unit is used to acquire new data requests sent by the second computing node; the new data request is sent by the second computing node after it has obtained new free capacity;

[0127] The target sub-data reading unit is used to access the high-speed computing link device through the high-speed computing link protocol according to the new data request, and read the target sub-data with a data size equal to the new free capacity from the remaining target data corresponding to the data request.

[0128] In some specific embodiments, the inter-node data transmission device may specifically include:

[0129] The data production task acquisition unit is used to acquire a data production task and execute the data production task to generate new data before acquiring the data request sent by the second computing node through the remote direct memory access network card;

[0130] A serialization unit is used to serialize the new data according to a high-speed computing link protocol to obtain processed data;

[0131] The writing unit is used to write the processed data into the high-speed computing link device and generate storage information corresponding to the processed data.

[0132] A storage information sending unit is used to send the storage information to a management node, so that the management node sends the storage information to a second computing node, and the second computing node sends the data request based on the storage information.

[0133] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 5 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.

[0134] Figure 5This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data transmission method between computing nodes disclosed in any of the foregoing embodiments.

[0135] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0136] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including target data, etc. The storage method can be temporary storage or permanent storage.

[0137] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the inter-node data transmission method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0138] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the data transmission method steps between computing nodes disclosed in any of the foregoing embodiments.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0140] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0141] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] The present invention has provided a detailed description of a data transmission method, device, medium, and program product between computing nodes. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for data transmission between computing nodes, characterized in that, A first computing node is applied to a stream computing system, the stream computing system including at least the first computing node and a second computing node, both the first computing node and the second computing node including a remote direct memory access network card and a high-speed computing link device that supports input / output sub-protocols and memory sub-protocols of high-speed computing link protocols, the high-speed computing link device being used to store data; The inter-node data transmission method includes: The data request sent by the second computing node is obtained through the remote direct memory access network card; According to the data request, access the high-speed computing link device through the high-speed computing link protocol to obtain the target data corresponding to the data request; The target data is sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card; Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the process further includes: Create a local key for the local computing node, add a remote key for the eligible remote computing node, and configure permissions for the memory region corresponding to the remote direct memory access network card based on the local key and the remote key. Accordingly, obtaining data requests sent by the second computing node through the remote direct memory access network card includes: obtaining data requests sent by the second computing node through the remote direct memory access network card; extracting the remote key from the data request; and accessing the high-speed computing link device through the high-speed computing link protocol according to the remote key. Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the process further includes: The high-speed computing link device is mapped between its physical address space and logical address space to obtain a first address mapping table; the first address mapping table contains the mapping relationship between the linear address and physical address of the high-speed computing link device. Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the process further includes: The driver of the remote direct memory access network card (RDB) creates a second address mapping table in the memory region corresponding to the RDB; the second address mapping table contains the mapping relationship between the virtual address of the RDB and the linear address of the high-speed computing link device. The process of obtaining data requests sent by the second computing node through the remote direct memory access network card includes: The data request sent by the second computing node is obtained through the remote direct memory access network card; the data request includes free capacity. According to the data request, access the high-speed computing link device via the high-speed computing link protocol to obtain the target data corresponding to the data request, including: According to the data request, the high-speed computing link device is accessed through the high-speed computing link protocol, and the target sub-data with a data size equal to the free capacity is read from the target data corresponding to the data request.

2. The data transmission method between computing nodes according to claim 1, characterized in that, Obtaining the data request sent by the second computing node through the remote direct memory access network card includes: The virtual address for reading the target data corresponding to the data request is determined using the remote direct memory access network card, and the linear address corresponding to the virtual address is determined by querying the second address mapping table. According to the data request, access the high-speed computing link device via the high-speed computing link protocol to obtain the target data corresponding to the data request, including: The physical address corresponding to the linear address is determined by querying the first address mapping table; Access the high-speed computing link device via the high-speed computing link protocol based on the physical address to read the target data corresponding to the data request.

3. The data transmission method between computing nodes according to claim 1, characterized in that, Sending the target data to the remote direct memory access network interface card (RDBIC) of the second computing node via the RDBIC includes: The remote direct memory access network card is used to determine the virtual address for writing the target data; The target data and the write virtual address are sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card, so that the remote direct memory access network card of the second computing node can query the address mapping table stored in the second computing node according to the write virtual address, determine the physical address corresponding to the write virtual address, and write the target data to the physical address corresponding to the write virtual address.

4. The data transmission method between computing nodes according to claim 1, characterized in that, Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: The device endpoints of the high-speed computing link device are connected via the system bus and the root port of the high-speed computing link device. The device information of the high-speed computing link device is obtained by scanning, and the high-speed computing link device is registered to the system memory according to the device information. Physical address space is allocated for the base address register and internal memory of the high-speed computing link device.

5. The data transmission method between computing nodes according to claim 4, characterized in that, Also includes: Get read / write operation instructions; Access the internal memory corresponding to the high-speed computing link device, and forward the read / write operation instructions to the root port of the high-speed computing link device through the internal memory; The root port converts the read / write operation command into a link control unit packet, and sends the link control unit packet to the endpoint of the high-speed computing link device. The high-speed computing link controller performs address resolution on the link control unit packet and performs read / write operations on the memory of the high-speed computing link device based on the resolved physical address.

6. The data transmission method between computing nodes according to claim 1, characterized in that, Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: Configure a network interface card that supports the Remote Direct Memory Access Protocol (RDP) on the computing node, and create an RDP interface in the data transmission interface of the stream computing system to add the RDP network interface card to the computing node.

7. The data transmission method between computing nodes according to claim 1, characterized in that, Obtaining the data request sent by the second computing node through the remote direct memory access network card includes: Obtain the link establishment request sent by the second computing node, and establish a remote direct memory access channel with the second computing node according to the link establishment request; The data request is obtained through the remote direct memory access channel, and the target data is sent through the remote direct memory access channel.

8. The data transmission method between computing nodes according to claim 1, characterized in that, Also includes: Obtain the new data request sent by the second computing node; The new data request is sent by the second computing node after it has obtained new free capacity; According to the new data request, the high-speed computing link device is accessed through the high-speed computing link protocol, and target sub-data with a data size equal to the new free capacity is read from the remaining target data corresponding to the data request.

9. The data transmission method between computing nodes according to claim 1, characterized in that, Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: Obtain data production tasks and execute the data production tasks to generate new data; The new data is serialized according to the high-speed computing link protocol to obtain the processed data. The processed data is written to the high-speed computing link device, and storage information corresponding to the processed data is generated. The storage information is sent to the management node, so that the management node sends the storage information to the second computing node, and the second computing node sends the data request based on the storage information.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data transmission method between computing nodes as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the inter-node data transmission method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data transmission method between computing nodes as described in any one of claims 1 to 9.

Images