Data transmission method and device between computing nodes, medium and program product

Abstract

The invention discloses a data transmission method and device between computing nodes, a medium and a program product, and relates to the technical field of data processing. The method is applied to a first computing node of the stream computing system, each of the first computing node and a second computing node of the stream computing system comprises a remote direct memory access network card and a high-speed computing link device supporting an input / output sub-protocol of a high-speed computing link protocol and a memory sub-protocol, and the high-speed computing link device is used for storing data; the method comprises the following steps: acquiring a data request sent by a second computing node through a remote direct memory access network card; accessing a high-speed computing link device through a high-speed computing link protocol according to the data request, and obtaining target data corresponding to the data request; and sending the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card. The data transmission speed between computing nodes can be improved, and processor resources are saved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, medium and program product for data transmission between computing nodes. Background Art

[0002] The stream computing system is a streaming data flow execution engine, which provides functions such as data distribution storage, data communication and fault tolerance mechanism for distributed computing of data flow. In the related technology, the stream computing system usually includes computing nodes and management nodes. The computing node users perform data computing tasks, and the management computing is used for task scheduling and resource management; the data transmission between computing nodes usually adopts the TCP / IP network stack for communication mechanism; after the sending computing node sends the data, it needs to store the data in the user state cache pool, copy it to the operating system kernel state cache pool, and then copy it to the network device cache, transmit it to the receiving computing node network device cache through the network, and then copy it to the receiving node operating system kernel state cache pool, and then copy it to the user state cache pool of the data flow processing application, so as to complete the data sending and receiving process. That is, the data transmission process will involve multiple copy operations, which increases the delay of data transmission and reduces the data transmission speed. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method, device, medium and program product for data transmission between computing nodes, which can improve the data transmission speed between computing nodes. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a method for data transmission between computing nodes, which is applied to computing nodes of a stream computing system, wherein the computing nodes include a remote direct memory access network card and a high-speed computing link device supporting an input / output sub-protocol and a memory sub-protocol of a high-speed computing link protocol, wherein the high-speed computing link device is used to store data; the method includes:

[0005] Acquire a data request sent by the second computing node through the remote direct memory access network card;

[0006] Accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain target data corresponding to the data request;

[0007] The target data is sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card.

[0008] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0009] The physical address space and the logical address space of the high-speed computing link device are mapped to obtain a first address mapping table; the first address mapping table contains a mapping relationship between the linear address and the physical address of the high-speed computing link device.

[0010] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0011] A second address mapping table is created in the memory area corresponding to the remote direct memory access network card; the second address mapping table contains a mapping relationship between the virtual address of the remote direct memory access network card and the linear address of the high-speed computing link device.

[0012] Optionally, obtaining, through the remote direct memory access network card, a data request sent by the second computing node includes:

[0013] Determine a read virtual address of target data corresponding to the data request by using the remote direct memory access network card, and determine a linear address corresponding to the read virtual address by querying the second address mapping table;

[0014] Accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain target data corresponding to the data request includes:

[0015] Determine the physical address corresponding to the linear address by querying the first address mapping table;

[0016] The high-speed computing link device is accessed through a high-speed computing link protocol according to the physical address to read target data corresponding to the data request.

[0017] Optionally, sending the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card includes:

[0018] Determining a write virtual address of the target data by using the remote direct memory access network card;

[0019] The target data and the write virtual address are sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card, so that the remote direct memory access network card of the second computing node queries the address mapping table stored in the second computing node according to the write virtual address, determines the physical address corresponding to the write virtual address, and writes the target data into the physical address corresponding to the write virtual address.

[0020] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0021] connecting a device endpoint of the high-speed computing link device via a system bus and a root port of the high-speed computing link device;

[0022] The device information of the high-speed computing link device is acquired by scanning, and the high-speed computing link device is registered to the system memory according to the device information, and a physical address space is allocated to the base address register and the internal memory of the high-speed computing link device.

[0023] Optionally, the method for transmitting data between computing nodes further includes:

[0024] Get read / write operation instructions;

[0025] Accessing an internal memory corresponding to the high-speed computing link device, and forwarding the read / write operation instruction to a root port of the high-speed computing link device through the internal memory;

[0026] converting the read / write operation instruction into a link control unit packet through the root port, and sending the link control unit packet to an endpoint of the high-speed computing link device;

[0027] The link control unit packet is addressed by a high-speed computing link controller, and a read / write operation is performed on the memory of the high-speed computing link device according to the physical address obtained by the address resolution.

[0028] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0029] A network card supporting remote direct memory access protocol is configured on the computing node, and an interface of the remote direct memory access protocol is created in the data transmission interface of the stream computing system to implement adding the remote direct memory access network card on the computing node.

[0030] Optionally, obtaining, through the remote direct memory access network card, a data request sent by the second computing node includes:

[0031] Obtaining a link establishment request sent by the second computing node, and establishing a remote direct memory access channel with the second computing node according to the link establishment request;

[0032] The data request is obtained through the remote direct memory access channel, and the target data is sent through the remote direct memory access channel.

[0033] Optionally, obtaining, through the remote direct memory access network card, a data request sent by the second computing node includes:

[0034] Acquire, through the remote direct memory access network card, a data request sent by the second computing node; the data request includes free capacity;

[0035] Accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain target data corresponding to the data request includes:

[0036] The high-speed computing link device is accessed through a high-speed computing link protocol according to the data request, and target sub-data having a data size equal to the free capacity is read from the target data corresponding to the data request.

[0037] Optionally, the method for transmitting data between computing nodes further includes:

[0038] Obtaining a new data request sent by the second computing node; the new data request is sent by the second computing node after applying for new free capacity;

[0039] The high-speed computing link device is accessed through a high-speed computing link protocol according to the new data request, and target sub-data having a data size equal to the new free capacity is read from the remaining target data corresponding to the data request.

[0040] Optionally, before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes:

[0041] Obtaining a data generation task, and executing the data generation task to generate new data;

[0042] Serializing the new data according to the high-speed computing link protocol to obtain processed data;

[0043] Writing the processed data into the high-speed computing link device and generating storage information corresponding to the processed data;

[0044] The storage information is sent to a management node so that the management node sends the storage information to a second computing node, and the second computing node sends the data request according to the storage information.

[0045] In a second aspect, the present application discloses an electronic device, comprising:

[0046] Memory, used to store computer programs;

[0047] The processor is used to execute the computer program to implement the aforementioned method for transmitting data between computing nodes.

[0048] In a third aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned method for transmitting data between computing nodes.

[0049] In a fourth aspect, the present application discloses a computer program product, including a computer program, which implements the aforementioned method for transmitting data between computing nodes when executed by a processor.

[0050] The present application discloses a method for data transmission between computing nodes, which is applied to computing nodes of a stream computing system, wherein the computing nodes include a remote direct memory access network card and a high-speed computing link device that supports the input and output sub-protocol and memory sub-protocol of a high-speed computing link protocol, and the high-speed computing link device is used to store data; the method includes: obtaining a data request sent by a second computing node through a remote direct memory access network card; accessing the high-speed computing link device through a high-speed computing link protocol according to the data request to obtain target data corresponding to the data request; sending the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card. It can be seen that by adding a remote direct memory access network card and a high-speed computing link device to the computing nodes of the stream computing system, data operations within the computing nodes are implemented through the high-speed computing link protocol, which improves the speed of data reading and writing within the nodes, and avoids the operating system switching between different execution modes requiring multiple data copy operations; by combining the remote direct memory access protocol and the high-speed computing link protocol, the direct transmission of data within the high-speed computing link device between computing nodes is realized, which can improve the transmission speed and save processor resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A flow chart of a method for data transmission between computing nodes provided in this application;

[0053] Figure 2 A specific diagram of data transmission within a computing node provided in this application;

[0054] Figure 3 A specific schematic diagram of data transmission between computing nodes provided in this application;

[0055] Figure 4 A schematic diagram of the structure of a data transmission device between computing nodes provided in this application;

[0056] Figure 5 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In the prior art, a stream computing system usually includes computing nodes and management nodes. The computing node users perform data computing tasks, and the management computing is used for task scheduling and resource management. The data transmission between computing nodes usually adopts the TCP / IP (Transmission Control Protocol / Internet Protocol) network stack for communication. After the sending computing node sends the data, it needs to store the data in the user state buffer pool, copy it to the operating system kernel state buffer pool, and then copy it to the network device buffer, transmit it to the receiving computing node network device buffer through the network, and then copy it to the receiving node operating system kernel state buffer pool, and then copy it to the user state buffer pool of the data stream processing application, so as to complete the data sending and receiving process. That is, the data transmission process will involve multiple copy operations, which increases the delay of data transmission, reduces the data transmission speed, and requires the processor to participate in the processing. In order to overcome the above technical problems, the present application proposes a data transmission method between computing nodes, which can improve the data transmission speed between computing nodes in the stream computing system and save processor resources.

[0059] The present application embodiment discloses a method for data transmission between computing nodes, which is applied to computing nodes of a stream computing system. The computing nodes include a remote direct memory access network card and a high-speed computing link device that supports the input and output sub-protocol and the memory sub-protocol of the high-speed computing link protocol. The high-speed computing link device is used to store data; see Figure 1 As shown, the method may include the following steps:

[0060] Step S11: Acquire a data request sent by the second computing node through a remote direct memory access network card.

[0061] In this embodiment, the first computing node obtains the data request sent by the second computing node in the stream computing system through the remote direct memory access network card; of course, the second computing node can also receive the data request sent by the first computing node, and the stream computing system can also include multiple computing nodes of the same type as the first computing node and the second node. The same type means that the computing nodes all include their own remote direct memory access network cards and high-speed computing link devices that support the input and output sub-protocols and memory sub-protocols of the high-speed computing link protocol.

[0062] Before actual data transmission, you need to add a remote direct memory access network card to the computing node. Specifically, you can add a remote direct memory access network card to the computing node by configuring a network card that supports the remote direct memory access protocol on the computing node and creating an interface for the remote direct memory access protocol in the data transmission interface of the stream computing system. That is, configure a network card driver that supports the Remote Direct Memory Access (RDMA) protocol on each computing node, and develop an interface for calling the RDMA protocol in the data transmission interface of the stream computing engine.

[0063] At the same time, before data transmission, it is necessary to add a high-speed computing link device that supports the input and output sub-protocol and memory sub-protocol of the high-speed computing link protocol (ComputeExpress Link, CXL) to the computing node, that is, the function driver software such as device registration, memory space allocation, read and write operations, so that the data reading and writing of the high-speed computing link protocol can be supported in the node. The devices corresponding to the high-speed computing link protocol include Type 1, Type 2 and Type 3. In this embodiment, the high-speed computing link device is specifically a Type 3 device that supports the input and output sub-protocol (CXL.io) and memory sub-protocol (CXL.mem) of the high-speed computing link protocol. It is a dedicated storage device that communicates directly with the host processor and can use the CXL protocol to achieve low-latency, high-throughput data transmission; it can also be used as a memory buffer to expand memory bandwidth and memory capacity.

[0064] Specifically, the device endpoint of the high-speed computing link device is connected through the system bus and the root port of the high-speed computing link device; the device information of the high-speed computing link device is obtained by scanning, and the high-speed computing link device is registered in the system memory according to the device information, and the physical address space is allocated for the base address register and internal memory of the high-speed computing link device. That is, at least one CXL root port (RPs, RootPorts) is configured on the system bus of each computing node CPU, and the CXL device endpoint (EP, End Point) is connected through RPs; the computing node calls the operating system kernel driver to start PCIe transactions (PCIe (peripheral component interconnect express, high-speed serial computer expansion bus standard) transactions) to scan the base address register (BAR, Base address register) of the CXL device and its internal memory (HDM, Host-managed Device Memory); according to the scan information, the high-speed computing link device (CXL.mem device) is registered in the node to the system memory, and the physical address space is allocated for BAR and HDM in the reserved memory space, and the memory of the high-speed computing link device is initialized and the address space is allocated.

[0065] In some embodiments, the above method also includes: obtaining read / write operation instructions; accessing the internal memory corresponding to the high-speed computing link device, and forwarding the read / write operation instructions to the root port of the high-speed computing link device through the internal memory; converting the read / write operation instructions into a link control unit packet through the root port, and sending the link control unit packet to the endpoint of the high-speed computing link device; performing address resolution on the link control unit packet through the high-speed computing link controller, and performing read / write operations on the memory of the high-speed computing link device according to the physical address obtained by resolution.

[0066] It can be understood that, further, it is necessary to synchronously map the CXL logical address space data with the physical address space of the CXL device to obtain a mapping table of the virtual address and physical address of the high-speed computing link device; finally, the high-speed computing link device information and address space data are written into the node operating system configuration file so that the computing node can read the configuration information of the high-speed computing link device when it is restarted, and when receiving a data read and write operation request, the HDM memory of the high-speed computing link device is accessed through a memory data read instruction (load) / data write memory instruction (store), and the request is passed to the corresponding root port, which converts the request into a CXL link control unit packet (CXL flit) and calls the CXL protocol to send it to the high-speed computing link device endpoint; then the CXL controller in the CXL device parses the CXL flit, converts the incoming address, and sends the converted request to the underlying memory controller to realize the reading and writing of the CXL device memory resources.

[0067] In some embodiments, obtaining a data request sent by the second computing node through a remote direct memory access network card may include: obtaining a link establishment request sent by the second computing node, establishing a remote direct memory access channel with the second computing node according to the link establishment request; obtaining a data request through the remote direct memory access channel, and sending target data through the remote direct memory access channel. That is, based on the direct memory access network cards of the two computing nodes, a remote direct memory access channel is established between the two computing nodes, and the two computing nodes interact directly through the channel.

[0068] Step S12: Access the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain target data corresponding to the data request.

[0069] That is, the remote direct memory access network card obtains the request sent by the remote device, and then accesses the high-speed computing link device according to the high-speed computing link protocol to read the target data from the device.

[0070] Before obtaining the data request sent by the second computing node through the remote direct memory access network card, it also includes: mapping the physical address space and the logical address space of the high-speed computing link device to obtain a first address mapping table; the first address mapping table contains the mapping relationship between the linear address of the high-speed computing link device and the physical address. Create a second address mapping table in the memory area corresponding to the remote direct memory access network card; the second address mapping table contains the mapping relationship between the virtual address of the remote direct memory access network card and the linear address of the high-speed computing link device. That is, by adding the mapping relationship table between the virtual address of the remote direct memory access network card and the linear address of the high-speed computing link device in the memory area corresponding to the remote direct memory access network card in the memory, the remote direct memory access protocol and the high-speed computing link protocol are combined. That is, the high-speed computing link device is registered in MR (Memory Region), and the specific RDMA driver will create and fill in a mapping table from a virtual address (VA) to a linear address (CXL.LA) of the high-speed computing link device in the memory, and the RDMA network can convert VA into CXL.LA by looking up the table when needed.

[0071] The above-mentioned method of obtaining the data request sent by the second computing node through the remote direct memory access network card includes: using the remote direct memory access network card to determine the read virtual address of the target data corresponding to the data request, and determining the linear address corresponding to the read virtual address by querying the second address mapping table; accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain the target data corresponding to the data request, including: determining the physical address corresponding to the linear address by querying the first address mapping table; accessing the high-speed computing link device through the high-speed computing link protocol according to the physical address to read the target data corresponding to the data request.

[0072] That is, the remote device mainly supports remote memory operations and remote message operations through RDMA operation high-speed computing link devices. Remote memory operations include read and write operations. For example Figure 2As shown in the figure, it is assumed that computing node TM1 initiates an RDMA write operation to computing node TM2, that is, directly writes data to the memory area of ​​the right node; it is assumed that both ends in the figure have completed the action of registering MR, and also created an address mapping table; the specific process of remote read and write operation is as follows: first, the application system on computing node 1 will call the driver to send a WR (Work Request), that is, WQE (Work Queue Element) to the remote direct memory access network card, informing the remote direct memory access network card of the local virtual address for storing the data to be sent, and the virtual address of the peer data to be written; the node maintains SQ (Send Queue) and RQ (Receive Queue). The local remote direct memory access network card queries the first address mapping table, and the high-speed computing link device queries the second address mapping table, reads data from the memory of the high-speed computing link device through the read instruction (Load instruction), assembles the data packet and sends it out. The remote direct memory access network card at the computing node 2 receives the data packet and parses the destination VA from it. The computing node 2 determines the actual physical address PA (Physical Address) of the multi-function by querying its corresponding first address mapping table and second address mapping table, and stores the data in the memory of the fast computing link device according to the physical address through a write instruction (‌Store instruction). For the entire data remote writing process, for computing node 2, whether it is address conversion or writing to memory, its CPU does not need to be involved at all.

[0073] In some embodiments, after creating the second address mapping table in the memory area corresponding to the remote direct memory access network card, it also includes: creating a local key for the local computing node, adding a remote key for the qualified remote computing node, and configuring permissions for the memory area corresponding to the remote direct memory access network card based on the local key and the remote key; accordingly, obtaining the data request sent by the second computing node through the remote direct memory access network card, including: obtaining the data request sent by the second computing node through the remote direct memory access network card; extracting the remote key from the data request, and accessing the high-speed computing link device through the high-speed computing link protocol according to the remote key. That is, in the process of registering MR, the permissions of MR are set at the same time, and the MR permissions mainly meet the read / write requirements of the local / remote nodes. By designing the local key Local Key (L_Key) and the remote key Remote Key (R_Key) respectively, that is, adding a key mechanism for accessing MR, MR creates L_Key and R_Key at the same time according to the program requirements when registering, and only the request holding the L_Key / R_Key key can operate MR; the computing node accesses the CXL memory according to whether the key of MR is obtained. Therefore, by setting the MR operation key during the MR registration process to implement MR authority management, the data transmission speed can be improved while ensuring data integrity.

[0074] In some embodiments, before obtaining the data request sent by the second computing node through the remote direct memory access network card, it also includes: obtaining a data production task, executing the data production task to generate new data; serializing the new data according to the high-speed computing link protocol to obtain processed data; writing the processed data into the high-speed computing link device, and generating storage information corresponding to the processed data; sending the storage information to the management node so that the management node sends the storage information to the second computing node, and the second computing node sends a data request according to the storage information. That is, after obtaining the data production task issued by the stream computing system, generating the corresponding data, the relevant information of the data is sent to the management node of the system, and the management node sends it to the consumer of the data, that is, the above-mentioned second computing node, and then the second computing node initiates a data request to the computing node that uploaded the storage information.

[0075] In some embodiments, obtaining a data request sent by a second computing node through a remote direct memory access network card may include: obtaining a data request sent by a second computing node through a remote direct memory access network card; the data request includes free capacity; accessing a high-speed computing link device through a high-speed computing link protocol according to the data request to obtain target data corresponding to the data request, including: accessing a high-speed computing link device through a high-speed computing link protocol according to the data request, and reading target sub-data whose data size is the free capacity from the target data corresponding to the data request. That is, by sending a data request including free capacity information, only data of the free capacity size can be sent when feeding back data, so as to avoid storage failure caused by data exceeding the storage capacity, thereby realizing data back pressure transmission.

[0076] Further, a new data request sent by the second computing node is obtained; the new data request is sent after the second computing node applies for new free capacity; the high-speed computing link device is accessed through the high-speed computing link protocol according to the new data request, and the target sub-data with a data size of the new free capacity is read from the remaining target data corresponding to the data request. That is, after the second computing node applies for new space, it initiates a data request again, and the computing node reads the target sub-data of the corresponding size from the target data that has never been sent according to the new free capacity; thus repeating this operation until the target data is completely sent to the second computing node.

[0077] Step S13: Send the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card.

[0078] After determining the target data, the target data is sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card. Specifically, the write virtual address of the target data is determined by using the remote direct memory access network card; the target data and the write virtual address are sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card, so that the remote direct memory access network card of the second computing node queries the address mapping table stored in the second computing node according to the write virtual address, determines the physical address corresponding to the write virtual address, and writes the target data to the physical address corresponding to the write virtual address. Similarly, the second computing node writes the target data to the memory of the high-speed computing link device of the second computing node by querying the address mapping table. Based on the direct memory access network card and the high-speed computing link device, the memory and cache of the system application can be expanded, and the memory and cache capacity of the upper-level data system processing and calculation can be quickly realized, supporting the realization of fast, large-capacity data transmission, exchange and sharing between clusters.

[0079] As can be seen from the above, this embodiment is applied to the computing node of the stream computing system, and the computing node includes a remote direct memory access network card and a high-speed computing link device that supports the input and output sub-protocol and memory sub-protocol of the high-speed computing link protocol, and the high-speed computing link device is used to store data; the method includes: obtaining a data request sent by the second computing node through the remote direct memory access network card; accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain the target data corresponding to the data request; sending the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card. It can be seen that by adding a remote direct memory access network card and a high-speed computing link device to the computing node of the stream computing system, the data operation in the computing node is realized through the high-speed computing link protocol, the speed of reading and writing data in the node is improved, and the operating system is prevented from switching between different execution modes. Multiple data copy operations are required; by combining the remote direct memory access protocol and the high-speed computing link protocol, the direct transmission of data in the high-speed computing link device between computing nodes is realized, which can improve the transmission speed and save processor resources.

[0080] Take the mainstream stream computing system engine Flink as an example. Flink is an open source data stream computing engine, which consists of two parts, namely the job management node (JobManager, JM) and the task computing node (TaskManager, TM). The Flink architecture is a master-slave architecture design. The JM node runs on the management node, that is, the master node, and is responsible for obtaining submitted tasks from the client, responsible for the scheduling of tasks in the entire Flink cluster, and responsible for resource management. The TM node runs on the slave node, that is, the computing node, and is responsible for specific task execution and resource application and management of the corresponding tasks on each node. All components need to communicate with each other through the Akka Framework, including information such as task status and checkpoint triggering. There are three types of nodes involved in the operation of the stream computing engine: the client, which is responsible for submitting tasks to the cluster's JobManager through Akka connection; the task management node JobManager; and the task computing node TaskManager.

[0081] For example Figure 3 The figure shows a flow chart of the data transmission method between computing nodes based on the Flink stream computing system, which describes the data interaction process between computing nodes in a specific data production-consumption type job. It includes two parallel tasks, running the data production task on the computing node TaskManager1 (TM1) and the data consumption task on TaskManager2 (TM2); it should be noted that each TaskManager supports running multiple data production tasks and data consumption tasks. Figure 3The focus is on the data transmission process between the two nodes of the data production task and the data consumption task. Data transmission is represented by solid arrows, and messages between nodes are represented by dotted arrows.

[0082] S21: First, the data production task produces a new data (ResultPartition, RP, ‌in Flink, used to represent the data generated by a task); the new data is serialized according to the CXL protocol; the serialized object (RecordSerializer) selects the corresponding channel selector (ChannelSelector) to distribute the record according to the record processing mode, RecordWriter (‌RecordWriter‌ is a component in Flink used to write key-value pairs after task processing to memory or disk). ChannelSelector record processing mode supports broadcast mode and hash mode; in broadcast mode, records will be distributed to each serialization queue; in hash partition mode, ChannelSelector will calculate the hash value on the record and distribute it to the corresponding serialization queue. And write the record to the storage medium of the high-speed computing link device. CXLmemBuffer Pool is a storage device connected via the CXL protocol, which can be organized into a very large storage pool to realize memory pooling; Figure 3 BufferWriter (buffer writer), that is, CXLmemBufferWriter.

[0083] S22: The buffer writer writes the RP record sequence to the memory to write the record data into the high-speed computing link device; and implements RS (ResultSubpartition, RP subpartition) allocation, RS data space application, RP and RS address mapping, data writing, return data address and other operations to write the data record to the high-speed computing link device.

[0084] S23: RecordWriter receives the RP, RS2 identifier, storage block list, node, remote key (R_key) and other parameters returned by CXLmemBufferWriter, and uploads these data to the new data list generated by the task in the JobManager execution graph.

[0085] S24: After receiving the RP and RS2 data reported by TM1, JobManager notifies the consumer of this RP of the message that RP and RS have been prepared, including the identification of RP and RS2, storage block list, node list and other information, and generates a data consumption task in TM2.

[0086] S25: The message sent to TM2 is propagated downward to the IG (InputGate, input encapsulation component) that should receive this data. The IG allocates IC2 (InputChannel, input channel) to prepare to receive the data. The IG first calls the connection management component (RCXLConnectionManager, i.e., the connection management component between the RDMA network card and the CXL device) to send a create link request to TM1, including the remote key (R_key) and the available free capacity of IC2 for this task.

[0087] S26: After receiving the network request from TM2's IG, TM1 node confirms that the remote key (R_key) is correct, and then selects the RS2.1 storage list whose combination is not greater than the IC2 capacity from the RS2 storage block list and returns it to TM2's IG.

[0088] S27: TM2's IG calls RCXLConnectionManager to create an RCXL data transmission channel with TM1. TM1 performs data reading and writing to complete the data transmission of this storage block list. TM2 consumes data through RecordReader (a component used to read and process data in Flink), in which the deserialization object (RecordDeserializer) is used for deserialization operations.

[0089] S28: After applying for new free space, TM2's IG creates IC3, packages the list of storage blocks that have not been transmitted from RS2 into a new transmission request, and repeats the S25-S27 process to complete all data transmission.

[0090] By adding remote direct memory access network cards and high-speed computing link devices to the computing nodes of the stream computing system, data operations within the computing nodes are implemented through the high-speed computing link protocol, which improves the speed of data reading and writing within the nodes and avoids the need for multiple data copy operations when the operating system switches between different execution modes; by combining the remote direct memory access protocol and the high-speed computing link protocol, direct transmission of data within the high-speed computing link device between computing nodes is achieved, which can improve transmission speed and save processor resources.

[0091] Correspondingly, the embodiment of the present application also discloses a data transmission device between computing nodes, which is applied to the computing nodes of the stream computing system, wherein the computing nodes include a remote direct memory access network card and a high-speed computing link device supporting the input and output sub-protocol and the memory sub-protocol of the high-speed computing link protocol, and the high-speed computing link device is used to store data; see Figure 4 As shown, the device comprises:

[0092] A data request acquisition module 11, configured to acquire a data request sent by the second computing node through the remote direct memory access network card;

[0093] A target data determination module 12, configured to access the high-speed computing link device through a high-speed computing link protocol according to the data request, and obtain target data corresponding to the data request;

[0094] The data transmission module 13 is used to send the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card.

[0095] As can be seen from the above, the data transmission method between computing nodes disclosed in the present application is applied to the computing nodes of the stream computing system, and the computing nodes include a remote direct memory access network card and a high-speed computing link device that supports the input and output sub-protocol and memory sub-protocol of the high-speed computing link protocol, and the high-speed computing link device is used to store data; the method includes: obtaining a data request sent by the second computing node through the remote direct memory access network card; accessing the high-speed computing link device through the high-speed computing link protocol according to the data request, and obtaining the target data corresponding to the data request; sending the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card. It can be seen that by adding a remote direct memory access network card and a high-speed computing link device to the computing nodes of the stream computing system, the data operation in the computing node is realized through the high-speed computing link protocol, the speed of reading and writing data in the node is improved, and the switching of the operating system in different execution modes requires multiple data copy operations; by combining the remote direct memory access protocol and the high-speed computing link protocol, the direct transmission of data in the high-speed computing link device between computing nodes is realized, which can improve the transmission speed and save processor resources.

[0096] In some specific embodiments, the data transmission device between computing nodes may specifically include:

[0097] The first mapping unit is used to map the physical address space and the logical address space of the high-speed computing link device to obtain a first address mapping table; the first address mapping table contains the mapping relationship between the linear address and the physical address of the high-speed computing link device.

[0098] In some specific embodiments, the data transmission device between computing nodes may specifically include:

[0099] The second mapping unit is used to create a second address mapping table in the memory area corresponding to the remote direct memory access network card; the second address mapping table contains the mapping relationship between the virtual address of the remote direct memory access network card and the linear address of the high-speed computing link device.

[0100] In some specific embodiments, the data request acquisition module 11 may specifically include:

[0101] A linear address determination unit, configured to determine a read virtual address of target data corresponding to the data request by using the remote direct memory access network card, and determine a linear address corresponding to the read virtual address by querying the second address mapping table;

[0102] The target data determination module 12 may specifically include:

[0103] A physical address determining unit, configured to determine a physical address corresponding to the linear address by querying the first address mapping table;

[0104] The target data reading unit is used to access the high-speed computing link device through the high-speed computing link protocol according to the physical address to read the target data corresponding to the data request.

[0105] In some specific embodiments, the data transmission module 13 may specifically include:

[0106] A writing unit, used for determining a writing virtual address of the target data by using the remote direct memory access network card;

[0107] A sending unit is used to send the target data and the write virtual address to the remote direct memory access network card of the second computing node through the remote direct memory access network card, so that the remote direct memory access network card of the second computing node queries the address mapping table stored in the second computing node according to the write virtual address, determines the physical address corresponding to the write virtual address, and writes the target data into the physical address corresponding to the write virtual address.

[0108] In some specific embodiments, the data transmission device between computing nodes may specifically include:

[0109] a connection unit, configured to connect a device endpoint of the high-speed computing link device through a system bus and a root port of the high-speed computing link device before acquiring the data request sent by the second computing node through the remote direct memory access network card;

[0110] A scanning unit is used to obtain device information of the high-speed computing link device through scanning, register the high-speed computing link device to the system memory according to the device information, and allocate physical address space for the base address register and internal memory of the high-speed computing link device.

[0111] In some specific embodiments, the data transmission device between computing nodes may further include:

[0112] An instruction acquisition unit, used for acquiring read / write operation instructions;

[0113] An instruction forwarding unit, used to access the internal memory corresponding to the high-speed computing link device, and forward the read / write operation instruction to the root port of the high-speed computing link device through the internal memory;

[0114] a link control unit packet sending unit, configured to convert the read / write operation instruction into a link control unit packet through the root port, and send the link control unit packet to an endpoint of the high-speed computing link device;

[0115] The address resolution unit is used to perform address resolution on the link control unit packet through the high-speed computing link controller, and perform read / write operations on the memory of the high-speed computing link device according to the resolved physical address.

[0116] In some specific embodiments, the data transmission device between computing nodes may specifically include:

[0117] A network card configuration unit is used to configure a network card that supports a remote direct memory access protocol on the computing node before obtaining a data request sent by the second computing node through the remote direct memory access network card, and to create an interface of the remote direct memory access protocol in the data transmission interface of the stream computing system, so as to implement adding the remote direct memory access network card on the computing node.

[0118] In some specific embodiments, the data request acquisition module 11 may specifically include:

[0119] a channel establishing unit, configured to obtain a link establishment request sent by the second computing node, and establish a remote direct memory access channel with the second computing node according to the link establishment request;

[0120] A data sending unit is used to obtain the data request through the remote direct memory access channel and send the target data through the remote direct memory access channel.

[0121] In some specific embodiments, the data request acquisition module 11 may specifically include:

[0122] A request acquisition unit, configured to acquire a data request sent by the second computing node through the remote direct memory access network card; the data request includes free capacity;

[0123] The target data determination module 12 may specifically include:

[0124] The device access unit is used to access the high-speed computing link device through the high-speed computing link protocol according to the data request, and read the target sub-data with a data size equal to the free capacity from the target data corresponding to the data request.

[0125] In some specific embodiments, the data transmission device between computing nodes may further include:

[0126] A new data request acquisition unit, used to acquire a new data request sent by the second computing node; the new data request is sent by the second computing node after applying for new free capacity;

[0127] The target sub-data reading unit is used to access the high-speed computing link device through the high-speed computing link protocol according to the new data request, and read the target sub-data with a data size equal to the new free capacity from the remaining target data corresponding to the data request.

[0128] In some specific embodiments, the data transmission device between computing nodes may specifically include:

[0129] A data production task acquisition unit, used to acquire a data production task and execute the data production task to generate new data before acquiring the data request sent by the second computing node through the remote direct memory access network card;

[0130] A serialization unit, used for serializing the new data according to the high-speed computing link protocol to obtain processed data;

[0131] A writing unit, used to write the processed data into the high-speed computing link device and generate storage information corresponding to the processed data;

[0132] The storage information sending unit is used to send the storage information to the management node, so that the management node sends the storage information to the second computing node, and the second computing node sends the data request according to the storage information.

[0133] Furthermore, the present application also discloses an electronic device, see Figure 5 As shown, the contents in the drawings should not be considered as any limitation on the scope of use of the present application.

[0134] Figure 5The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the method for transmitting data between computing nodes disclosed in any of the aforementioned embodiments.

[0135] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0136] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including target data, etc. The storage method can be temporary storage or permanent storage.

[0137] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data transmission method between computing nodes executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0138] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the method for transmitting data between computing nodes disclosed in any of the aforementioned embodiments are implemented.

[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0140] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0141] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0142] The above is a detailed introduction to a method, device, medium and program product for data transmission between computing nodes provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for data transmission between computing nodes, characterized in that: A first computing node applied to a stream computing system, the stream computing system comprising at least the first computing node and a second computing node, the first computing node and the second computing node both comprising a remote direct memory access network card and a high-speed computing link device supporting an input / output sub-protocol and a memory sub-protocol of a high-speed computing link protocol, the high-speed computing link device being used to store data; The method for data transmission between computing nodes comprises: Acquire a data request sent by the second computing node through the remote direct memory access network card; Accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain target data corresponding to the data request; The target data is sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card.

2. The method for data transmission between computing nodes according to claim 1, characterized in that: Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: The physical address space and the logical address space of the high-speed computing link device are mapped to obtain a first address mapping table; the first address mapping table contains a mapping relationship between the linear address and the physical address of the high-speed computing link device.

3. The method for data transmission between computing nodes according to claim 2, characterized in that: Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: A second address mapping table is created in the memory area corresponding to the remote direct memory access network card; the second address mapping table contains a mapping relationship between the virtual address of the remote direct memory access network card and the linear address of the high-speed computing link device.

4. The method for data transmission between computing nodes according to claim 3, characterized in that: Acquiring the data request sent by the second computing node through the remote direct memory access network card includes: Determine a read virtual address of target data corresponding to the data request by using the remote direct memory access network card, and determine a linear address corresponding to the read virtual address by querying the second address mapping table; Accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain target data corresponding to the data request includes: Determine the physical address corresponding to the linear address by querying the first address mapping table; The high-speed computing link device is accessed through a high-speed computing link protocol according to the physical address to read target data corresponding to the data request.

5. The method for data transmission between computing nodes according to claim 3, characterized in that: Sending the target data to the remote direct memory access network card of the second computing node through the remote direct memory access network card includes: Determining a write virtual address of the target data by using the remote direct memory access network card; The target data and the write virtual address are sent to the remote direct memory access network card of the second computing node through the remote direct memory access network card, so that the remote direct memory access network card of the second computing node queries the address mapping table stored in the second computing node according to the write virtual address, determines the physical address corresponding to the write virtual address, and writes the target data into the physical address corresponding to the write virtual address.

6. The method for data transmission between computing nodes according to claim 1, characterized in that: Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: connecting a device endpoint of the high-speed computing link device via a system bus and a root port of the high-speed computing link device; The device information of the high-speed computing link device is acquired by scanning, and the high-speed computing link device is registered to the system memory according to the device information, and a physical address space is allocated to the base address register and the internal memory of the high-speed computing link device.

7. The method for data transmission between computing nodes according to claim 6, characterized in that: Also includes: Get read / write operation instructions; Accessing an internal memory corresponding to the high-speed computing link device, and forwarding the read / write operation instruction to a root port of the high-speed computing link device through the internal memory; converting the read / write operation instruction into a link control unit packet through the root port, and sending the link control unit packet to an endpoint of the high-speed computing link device; The link control unit packet is addressed by a high-speed computing link controller, and a read / write operation is performed on the memory of the high-speed computing link device according to the physical address obtained by the address resolution.

8. The method for data transmission between computing nodes according to claim 1, characterized in that: Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: A network card supporting remote direct memory access protocol is configured on the computing node, and an interface of the remote direct memory access protocol is created in the data transmission interface of the stream computing system to implement adding the remote direct memory access network card on the computing node.

9. The method for data transmission between computing nodes according to claim 1, characterized in that: Acquiring the data request sent by the second computing node through the remote direct memory access network card includes: Obtaining a link establishment request sent by the second computing node, and establishing a remote direct memory access channel with the second computing node according to the link establishment request; The data request is obtained through the remote direct memory access channel, and the target data is sent through the remote direct memory access channel.

10. The method for data transmission between computing nodes according to claim 1, characterized in that: Acquiring the data request sent by the second computing node through the remote direct memory access network card includes: Acquire, through the remote direct memory access network card, a data request sent by the second computing node; the data request includes free capacity; Accessing the high-speed computing link device through the high-speed computing link protocol according to the data request to obtain target data corresponding to the data request includes: The high-speed computing link device is accessed through a high-speed computing link protocol according to the data request, and target sub-data having a data size equal to the free capacity is read from the target data corresponding to the data request.

11. The method for data transmission between computing nodes according to claim 10, characterized in that: Also includes: Obtaining a new data request sent by the second computing node; The new data request is sent after the second computing node applies for new free capacity; The high-speed computing link device is accessed through a high-speed computing link protocol according to the new data request, and target sub-data having a data size equal to the new free capacity is read from the remaining target data corresponding to the data request.

12. The method for data transmission between computing nodes according to claim 1, characterized in that: Before obtaining the data request sent by the second computing node through the remote direct memory access network card, the method further includes: Obtaining a data production task, and executing the data production task to generate new data; Serializing the new data according to the high-speed computing link protocol to obtain processed data; Writing the processed data into the high-speed computing link device and generating storage information corresponding to the processed data; The storage information is sent to a management node so that the management node sends the storage information to a second computing node, and the second computing node sends the data request according to the storage information.

13. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method for transmitting data between computing nodes as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by the processor, the method for data transmission between computing nodes as described in any one of claims 1 to 12 is implemented.

15. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the method for transmitting data between computing nodes as described in any one of claims 1 to 12.

Images