Data transmission system, method and apparatus, communication device and storage medium

The direct data transmission between the target channel adapter module is remotely controlled through the host channel adapter module, which solves the problems of data transmission delay and low network utilization in the prior art, realizes efficient data transmission and reduces the latency of the FPGA cloud platform.

WO2024227389A1PCT designated stage expired Publication Date: 2024-11-07INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/084532
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-03-28
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

In the prior art, there is a problem that the data transmission process is large and the network utilization rate is low during the host channel adapter module.

Method used

The host channel adapter module remotely controls data transmission between the target channel adapter modules, establishes a queue connection, and realizes direct remote and direct data access and transmission between the target channel adapter, avoiding forwarding through the host channel adapter module.

Benefits of technology

It reduces data transmission delay, improves network utilization, and is easy to implement FPGA, significantly reducing the latency of distributed computing on the FPGA cloud platform.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024084532_07112024_PF_FP_ABST
    Figure CN2024084532_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide a data transmission system, method and apparatus, a communication device and a storage medium, applied to the technical field of communications. The system comprises a host channel adapter module, a first target channel adapter module and a second target channel adapter module. The host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module; the first target channel adapter module is communicatively connected to the second target channel adapter module; the host channel adapter module is used for sending a remote direct data access operation packet to the first target channel adapter module, and remotely controlling, according to the remote direct data access operation packet, the first target channel adapter module to transmit data to the second target channel adapter module. According to the embodiments of the present application, data transmission between target channel adapter modules is remotely controlled by means of a host channel adapter module, so that direct data transmission between the target channel adapters is realized without adding a device.
Need to check novelty before this filing date? Find Prior Art

Description

Data transmission system, method, device, communication equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 4, 2023, with application number 202310488688.1, and application name “Data transmission system, method, device, communication equipment and storage medium”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of communication technology, and in particular to a data transmission system, method, apparatus, communication equipment and storage medium. Background Art

[0004] With the rapid development of technologies such as cloud computing, artificial intelligence, and big data, the amount of data that needs to be processed has increased exponentially, and the requirements for network latency and bandwidth for data transmission have become increasingly higher. Therefore, Remote Direct Memory Access (RDMA) has gradually become the mainstream data transmission protocol in data centers.

[0005] Data transmission based on the Remote Direct Data Access protocol requires establishing a Queue Pair (QP) connection between nodes before transmission. Each node can have multiple Queue Pair module resources, and when establishing a connection, the node must notify the other node of the QP module number it is using. Remote Direct Data Access divides computing nodes into two types: Host Channel Adapters (HCAs) and Target Channel Adapters (TCAs). HCAs are servers with operating systems and can initiate Remote Direct Data Access operations (sending data to or reading data from a remote end). TCAs, on the other hand, are network hard drives, accelerator cards, and other devices without a host and can only respond to Remote Direct Data Access operations.

[0006] In the prior art, data transmission between two target channel adapter modules is performed by reading data from one target channel adapter module through the host channel adapter module and then sending it to the other target channel adapter module, which will result in a large delay in the data transmission process and low network utilization.

[0007] Summary of the Invention

[0008] The purpose of the embodiments of the present application is to provide a data transmission system, method, apparatus, communication device, and storage medium to address the technical issues of large delays and low network utilization during data transmission via a host channel adapter module in the prior art. The specific technical solutions are as follows:

[0009] In a first aspect of the present application, a data transmission system is first provided, the data transmission system comprising: a host channel adapter module, a first target channel adapter module and a second target channel adapter module;

[0010] The host channel adapter module is communicatively connected to a first target channel adapter module and a second target channel adapter module;

[0011] The first target channel adapter module is communicatively coupled to the second target channel adapter module;

[0012] The host channel adapter module is used to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

[0013] In some embodiments of the present application, the communication connection between the first target channel adapter module and the second target channel adapter module includes: the host channel adapter module sends a first remote direct data access data packet and a second remote direct data access data packet to the first target channel adapter module and the second target channel adapter module, respectively, so that the first target channel adapter module and the second target channel adapter module establish a connection based on the first remote direct data access data packet and the second remote direct data access data packet.

[0014] In some embodiments of the present application, the first target channel adapter module includes a remote work queue event engine module, a remote completion queue event engine module, a first queue pair module, and a second queue pair module;

[0015] The remote work queue event engine module is used to send the work queue event data recorded by the internal register of the first queue pair module into a work queue event to the second queue pair module, and trigger the remote direct data access operation;

[0016] The remote completion queue event engine module is used to combine the completion queue event information returned by the second queue pair module and the physical address corresponding to the host channel adapter module recorded in the internal register of the first queue pair module to form a work queue event and send it to the first queue pair module, so that the first queue pair module returns the completion queue event information to the host channel adapter module.

[0017] In some embodiments of the present application, the first target channel adapter module further includes a first parsing module;

[0018] The first parsing module is used for receiving a remote direct data access data packet sent by the host channel adapter module, parsing the remote direct data access data packet, and converting the parsed remote direct data access data packet into a data bus write operation.

[0019] In some embodiments of the present application, the second target channel adapter module includes a third queue pair module and a fourth queue pair module;

[0020] The third queue pair module is communicatively coupled to a sixth queue pair module in the host channel adapter module;

[0021] The fourth queue pair module is communicatively coupled to the second queue pair module in the first target channel adapter module.

[0022] In some embodiments of the present application, the second target channel adapter module further includes a second parsing module;

[0023] The second parsing module is used for receiving the remote direct data access data packet sent by the host channel adapter module, parsing the remote direct data access data packet, and identifying the first target channel adapter module according to the parsed remote direct data access data packet.

[0024] In some embodiments of the present application, the host channel adapter module includes a fifth queue pair module and a sixth queue pair module;

[0025] The fifth queue pair module is communicatively coupled to the third queue pair module in the second target channel adapter module;

[0026] The sixth queue pair module is communicatively coupled to the first queue pair module in the second target channel adapter module.

[0027] In some embodiments of the present application, the host channel adapter module is further configured to send a first remote direct data access data packet to a first parsing module in a first target channel adapter module, the first remote direct data access data packet including a physical address corresponding to a second target channel adapter module, and a second remote direct data access data packet to a second parsing module in a second target channel adapter module, the second remote direct data access data packet including a physical address corresponding to the first target channel adapter module, so that the first target channel adapter module and the second target channel adapter module establish a connection based on the remote direct data access data packet.

[0028] In some embodiments of the present application, a remote completion queue event register and a remote work queue event register are pre-set in the host memory corresponding to the host channel adapter module;

[0029] The remote completion queue event register is used to store the completion queue event information returned by the target channel adapter;

[0030] The remote work queue event register is used to store remote direct data access operation information according to a preset remote write register format.

[0031] In some embodiments of the present application, the remote direct data access operation information includes work queue event information and a physical address of a remote completion queue event register;

[0032] The work queue event corresponds to an internal register of a remote work queue event engine module in the first target channel adapter module;

[0033] The physical address of the remote completion queue event register corresponds to an internal register of the remote work queue event engine module in the first target channel adapter module.

[0034] In another aspect of the present application, a data transmission method is provided, which is applied to a first target channel adapter module. The data transmission method includes:

[0035] receiving a remote direct data access operation data packet sent by the host channel adapter module when detecting that a connection is established between the first target channel adapter module and the second target channel adapter module, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module;

[0036] The target storage data corresponding to the first target channel adapter module is sent to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

[0037] In some embodiments of the present application, a remote direct data access operation data packet includes remote work queue event register data and a physical address of a remote completion queue event register, and sending target storage data corresponding to a first target channel adapter module to a second target channel adapter module according to the remote direct data access operation data packet so that the second target channel adapter module obtains the target storage data includes:

[0038] Parsing a remote direct data access operation data packet, writing a first work queue event in the remote work queue event register data into an internal register of the remote work queue event engine module according to the parsed remote direct data access operation data packet, and writing a physical address of the remote completion queue event register into an internal register of the remote completion queue event engine module;

[0039] The remote work queue event engine module sends the first work queue event to the second queue pair module;

[0040] The second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the first work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module so that the second target channel adapter module obtains the target remote direct data access operation data packet.

[0041] In some embodiments of the present application, a work queue event includes a work queue event identifier, a storage physical address of a first target channel adapter module, a destination storage address of a second target channel adapter module, and a data byte length. The second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, including:

[0042] The second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the storage physical address of the first target channel adapter module, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module according to the destination storage address of the second target channel adapter module.

[0043] In some embodiments of the present application, after the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the first work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, the method includes:

[0044] A data transmission completion signal returned by the second target channel adapter module is received, and according to the data transmission completion signal, it is determined that the second target channel adapter module receives the target remote direct data access operation data packet.

[0045] In some embodiments of the present application, receiving a data transmission completion signal returned by the second target channel adapter module, and determining, according to the data transmission completion signal, that the second target channel adapter module receives the target remote direct data access operation data packet includes:

[0046] The fourth queue pair module in the second target channel adapter module receives a confirmation character data packet returned based on the remote direct data access protocol, and determines that the second target channel adapter module receives the target remote direct data access operation data packet according to the confirmation character data packet.

[0047] In some embodiments of the present application, after receiving an acknowledgment character data packet returned by the fourth queue pair module in the second target channel adapter module based on the remote direct data access protocol and determining, based on the acknowledgment character data packet, that the second target channel adapter module receives a target remote direct data access operation data packet, the method includes:

[0048] Generate completion queue events based on confirmation character data packets;

[0049] A second work queue event is generated according to the physical address of the completion queue event and the remote completion queue event register, and the second work queue event is sent to the host channel adapter module so that the host channel adapter module stores the completion queue event information in the remote completion queue event register according to the second work queue event.

[0050] In some embodiments of the present application, before the step of receiving the remote direct data access data packet sent by the host channel adapter module, the method includes:

[0051] receiving a first remote direct data access data packet sent by a host channel adapter module, wherein the first remote direct data access data packet includes a physical address corresponding to a second target channel adapter module;

[0052] The second target channel adapter module is identified according to a physical address corresponding to the second target channel adapter module.

[0053] In some embodiments of the present application, receiving a remote direct data access data packet sent by a host channel adapter module includes:

[0054] When it is detected that the preset configuration in the host channel adapter module is completed and the trigger register corresponding to the fifth queue pair module in the host channel adapter module is in a trigger state, a remote direct data access data packet sent by the host channel adapter module is received.

[0055] In another aspect of the present application, a data transmission method is provided, which is applied to the second target channel adapter module. The first target channel adapter module and the second target channel adapter module are communicatively connected. The data transmission method includes:

[0056] Target storage data sent by the first target channel adapter module is received, wherein the target storage data is transmitted by the first target channel adapter module under remote control of the host channel adapter module according to a remote direct data access operation data packet.

[0057] In another aspect of the present application, a data transmission method is provided, which is applied to a host channel adapter module, wherein the host channel adapter module is communicatively connected to a first target channel adapter module and a second target channel adapter module. The data transmission method includes:

[0058] A remote direct data access operation data packet is sent to the first target channel adapter module, so that the first target channel adapter module performs data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

[0059] In another aspect of the present application, a data transmission device is provided, which is applied to the first target channel adapter module. The data transmission device includes:

[0060] a first receiving module, configured to receive a remote direct data access operation data packet sent by the host channel adapter module when detecting that a connection is established between the first target channel adapter module and the second target channel adapter module, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission from the first target channel adapter module to the second target channel adapter module;

[0061] The first sending module is used to send the target storage data corresponding to the first target channel adapter module to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

[0062] In another aspect of the present application, a data transmission device is provided, which is applied to the second target channel adapter module. The first target channel adapter module and the second target channel adapter module are communicatively connected. The data transmission device includes:

[0063] The second receiving module is configured to receive target storage data sent by the first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module under remote control of the host channel adapter module according to a remote direct data access operation data packet.

[0064] In another aspect of the present application, a data transmission device is provided, which is applied to a host channel adapter module. The host channel adapter module is communicatively connected to a first target channel adapter module and a second target channel adapter module. The data transmission device includes:

[0065] The second sending module is configured to send a remote direct data access operation data packet to the first target channel adapter module, so that the first target channel adapter module performs data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

[0066] In another aspect of the present application, a communication device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0067] Memory for storing computer programs;

[0068] The processor is configured to implement any of the above-mentioned data transmission methods when executing a program stored in the memory.

[0069] In another aspect of the present application, a computer non-volatile readable storage medium is provided, in which instructions are stored. When the computer non-volatile readable storage medium is run on a computer, the computer executes any of the above-mentioned data transmission methods.

[0070] In another aspect of the implementation of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned data transmission methods.

[0071] The data transmission system provided by the embodiment of the present application includes: a host channel adapter module, a first target channel adapter module and a second target channel adapter module; the host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module; the first target channel adapter module and the second target channel adapter module are communicatively connected; the host channel adapter module is used to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission on the second target channel adapter module based on the remote direct data access operation data packet. That is, the embodiment of the present application remotely controls the data transmission between the target channel adapter modules through the host channel adapter module, thereby enabling direct data transmission between target channel adapter devices without adding additional devices, realizing direct remote direct data access transmission of data between target channel adapters, improving network utilization, reducing data transmission delay, and being easy to implement in FPGA, which can significantly reduce the delay of distributed computing on the FPGA cloud platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0073] FIG1 shows a schematic diagram of a data transmission system provided by an embodiment of the present application;

[0074] FIG2 is a schematic diagram of a data transmission system in a related art provided by an embodiment of the present application;

[0075] FIG3 shows a schematic diagram of another data transmission system provided in an embodiment of the present application;

[0076] FIG4 shows a schematic diagram of a target channel adapter module provided in an embodiment of the present application;

[0077] FIG5 shows a flowchart of a data transmission method according to an embodiment of the present application;

[0078] FIG6 shows a flowchart of another data transmission method provided by an embodiment of the present application;

[0079] FIG7 shows a flowchart of another data transmission method provided by an embodiment of the present application;

[0080] FIG8 shows a flowchart of another data transmission method provided by an embodiment of the present application;

[0081] FIG9 shows a block diagram of a data transmission device provided in an embodiment of the present application;

[0082] FIG10 shows a block diagram of another data transmission device provided in an embodiment of the present application;

[0083] FIG11 shows a block diagram of another data transmission device provided in an embodiment of the present application;

[0084] FIG12 shows a structural block diagram of a communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0086] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0087] It should be noted that data transmission based on the remote direct data access protocol requires the establishment of a queue pair connection between nodes before transmission. Each node can have multiple queue pair module resources. When establishing a connection, the other party needs to be notified of the queue pair module number used by this node. Remote direct data access divides computing nodes into two types: host channel adapter modules and target channel adapters. The host channel adapter module is a server with an operating system, etc., which can initiate remote direct data access operations; while the target channel adapter is a network hard disk, accelerator card, etc. without a host and can only respond to remote direct data access operations. Referring to Figure 2, Figure 2 shows a schematic diagram of a data transmission system in a related technology provided by an embodiment of the present application. In the related technology, data transmission between two target channel adapter modules is performed by the host channel adapter module reading data from one target channel adapter module and then sending it to the other target channel adapter module. Specifically, the host channel adapter module obtains data by connecting to queue pair 1 in the first target channel adapter module through queue pair 1, and the host channel adapter module connects to queue pair 1 in the second target channel adapter module through queue pair 2, thereby transmitting data in the first target channel adapter module to the second target channel adapter module. As a result, during the data transmission process, due to the need to transfer through the host channel adapter module, the host may experience a large delay in allocating or using resources such as the hard disk or accelerator card in the target channel adapter.

[0088] Referring to Figures 1 and 3, Figure 1 shows a schematic diagram of a data transmission system provided in an embodiment of the present application, and Figure 3 shows a schematic diagram of another data transmission system provided in an embodiment of the present application. An embodiment of the present application provides a data transmission system, which includes a host channel adapter module, a first target channel adapter module, and a second target channel adapter module.

[0089] It should be noted that, in some embodiments of the present application, the host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module, and the first target channel adapter module and the second target channel adapter module are communicatively connected.

[0090] Specifically, the first target channel adapter module and the second target channel adapter module communicate via a queue pair. It should be noted that in the Remote Direct Data Access protocol, a WQ is a queue that stores work requests, and a WQE is an element within a WQ queue. A WQE can be considered a "task specification." This work request is sent from software to hardware. This specification contains the task the software wants the hardware to perform and detailed information about the task. For example, if a task is to "send 10 bytes of data located at address 0x12345678 to the remote node," upon receiving the task, the hardware will use DMA to retrieve the data from memory, assemble the data packet, and then send it.

[0091] Any communication process must have both sending and receiving ends, so a queue pair is a combination of a sending work queue and a receiving work queue. These two queues can be called SQ (Send Queue) and RQ (Receive Queue) respectively. Therefore, in an embodiment of the present application, when the host channel adapter module and the target channel adapter module communicate or transmit data, there will be multiple corresponding queue pair modules. For example, the first queue pair module in Figure 1 is one of the queue pair modules in the first target channel adapter module, which can establish a connection with the fifth queue pair module in the host channel adapter module. When the host channel adapter module needs to perform a remote direct data access operation, it can send a remote direct data access data packet to the first queue pair module through the fifth queue pair module.

[0092] Therefore, in some embodiments of the present application, in order to implement direct data transmission between target channel adapters, it is necessary to first establish a connection between the first target channel adapter module and the second target channel adapter module.

[0093] In some embodiments of the present application, the communication connection between the first target channel adapter module and the second target channel adapter module includes: the host channel adapter module sends a first remote direct data access data packet and a second remote direct data access data packet to the first target channel adapter module and the second target channel adapter module, respectively, so that the first target channel adapter module and the second target channel adapter module establish a connection based on the first remote direct data access data packet and the second remote direct data access data packet.

[0094] In some embodiments of the present application, the host channel adapter module is further configured to send a first remote direct data access data packet to a first parsing module in a first target channel adapter module, the first remote direct data access data packet including a physical address corresponding to a second target channel adapter module, and a second remote direct data access data packet to a second parsing module in a second target channel adapter module, the second remote direct data access data packet including a physical address corresponding to the first target channel adapter module, so that the first target channel adapter module and the second target channel adapter module establish a connection based on the remote direct data access data packet.

[0095] It should be noted that, in some embodiments of the present application, the parsing module can be expressed as a reg_operation module. Therefore, the first parsing module can be expressed as a first reg_operation module, and the second parsing module can be expressed as a second reg_operation module.

[0096] It should be noted that the host channel adapter module can perform remote transmission control based on the remote direct data access protocol. Therefore, the host channel adapter module can send remote direct data access data packets to multiple target channel adapter modules (e.g., a first target channel adapter module and a second target channel adapter module). In some embodiments of the present application, the data packets can include the physical addresses corresponding to each target channel adapter module or the identification information corresponding to each target channel adapter module. For example, the host channel adapter module can send a first remote direct data access data packet to a first parsing module in the first target channel adapter module. The first remote direct data access data packet can include the physical address corresponding to the second target channel adapter module or the identification information of the second target channel adapter module. In this way, the parsing module in the first target channel adapter module can obtain the identification information of the second target channel adapter module after parsing the data packet. Similarly, the second parsing module in the second target channel adapter module can receive a second remote direct data access data packet. The second remote direct data access data packet includes the physical address corresponding to the first target channel adapter module or the identification information of the first target channel adapter module. Therefore, the host channel adapter module can establish a queue pair connection between the first target channel adapter module and the second target channel adapter module.

[0097] In some embodiments of the present application, the host channel adapter module can remotely write registers to configure idle queue pair module registers within a target channel adapter, thereby establishing a queue pair connection between a first target channel adapter module and a second target channel adapter module. After the two target channel adapters establish the queue pair connection, the host channel adapter module can initiate a remote direct data access operation to control the writing of data from the first target channel adapter module to a preset address in the second target channel adapter module.

[0098] In some embodiments of the present application, the host channel adapter module is used to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

[0099] The remote direct data access operation data packet may include a payload and a header. The payload of the data packet may be the write address and write data of the internal register of the target channel adapter module, and the data format may be predefined as a preset remote write register format. The destination storage address field of the base transport layer header (BTH) of the remote direct data access data packet may be the address of the parsing module reg_operation within the target channel adapter module. After receiving the data packet, the reg_operation module of the target channel adapter module extracts the register write address and write data from the payload, converts them into a data bus write operation, and then performs a write operation on all registers connected to the data bus within the target channel adapter.

[0100] In some embodiments of the present application, the first target channel adapter module includes a rwqe_engine module, a rcqe_engine module, a first queue pair module and a second queue pair module; the rwqe_engine module is used to compose the work queue event WQE data recorded by the internal register of the first queue pair module into a WQE and send it to the second queue pair module, and trigger a remote direct data access operation; the rcqe_engine module is used to compose the completion queue event CQE information returned by the second queue pair module and the physical address corresponding to the host channel adapter module recorded by the internal register of the first queue pair module into a WQE and send it to the first queue pair module, so that the first queue pair module returns the CQE information to the host channel adapter module.

[0101] In some embodiments of the present application, the first target channel adapter module further includes a first parsing module; the first parsing module is used to receive a remote direct data access data packet sent by the host channel adapter module, parse the remote direct data access data packet, and convert the parsed remote direct data access data packet into a data bus write operation.

[0102] In some embodiments of the present application, two modules are added to the first target channel adapter module, namely the rwqe_engine module and the rcqe_engine module. The rwqe_engine module can form a WQE with the remote WQE data recorded in the internal register and send it to the second queue pair module, triggering a remote direct data access operation. The rcqe_engine module can form a WQE with the CQE information returned by the second queue pair module and the physical address of the remote CQE stored and returned by the host channel adapter module recorded in the internal register, and send it to the queue pair module, so that the CQE information generated by the first target channel adapter module is fed back to the host channel adapter module in the form of a remote direct data access data packet.

[0103] In some embodiments of the present application, a first parsing module is further added inside the first target channel adapter module. The first parsing module is a parsing module that is used to receive remote direct data access data packets sent by the host channel adapter module, parse the remote direct data access data packets, and convert the parsed remote direct data access data packets into data bus write operations.

[0104] In some embodiments of the present application, the host channel adapter module includes a fifth queue pair module and a sixth queue pair module; the fifth queue pair module is communicatively connected to the third queue pair module in the second target channel adapter module; and the sixth queue pair module is communicatively connected to the first queue pair module in the second target channel adapter module. The host channel adapter module is configured to send a remote direct data access operation data packet to the first target channel adapter module and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module based on the remote direct data access operation data packet.

[0105] It should be noted that, in some embodiments of the present application, the host channel adapter module is mainly used to remotely control the first target channel adapter module, so that the first target channel adapter module transmits stored data to the second target channel adapter module.

[0106] In some embodiments of the present application, a remote completion queue event register and a remote work queue event register are pre-set in the host memory corresponding to the host channel adapter module; the remote completion queue event register is used to store the CQE information returned by the target channel adapter; and the remote work queue event register is used to store remote direct data access operation information according to a preset remote write register format.

[0107] Remote direct data access operation information includes WQE information and the physical address of the remote completion queue event register; WQE corresponds to the internal register of the rwqe_engine module in the first target channel adapter module; the physical address of the remote completion queue event register corresponds to the internal register of the rcqe_engine module in the first target channel adapter module.

[0108] It should be noted that in some embodiments of the present application, the host channel adapter module is connected to a host, which can be a server. Before the target channel adapter module performs data transmission, the host memory of the host channel adapter module pre-sets a remote completion queue event register and a remote work queue event register. The remote completion queue event register is used to store the CQE information returned by the target channel adapter; the remote work queue event register is used to store the information required to initiate a remote direct data access operation in a remote write register format. This information includes the WQE information for remote direct data access and the physical address of the remote completion queue event register, which correspond to the internal registers of the rwqe_engine module and the internal registers of the rcqe_engine module in the target channel adapter, respectively.

[0109] In some embodiments of the present application, the second target channel adapter module includes a third queue pair module and a fourth queue pair module; the third queue pair module is communicatively connected to the sixth queue pair module in the host channel adapter module; and the fourth queue pair module is communicatively connected to the second queue pair module in the first target channel adapter module. The second target channel adapter module also includes a second parsing module; the second parsing module is configured to receive remote direct data access data packets sent by the host channel adapter module, parse the remote direct data access data packets, and identify the first target channel adapter module based on the parsed remote direct data access data packets.

[0110] It should be noted that, in some embodiments of the present application, there may be multiple second target channel adapter modules, and the second target channel adapter module is mainly used to receive data stored in the first target channel adapter module.

[0111] Therefore, the second target channel adapter module includes a fourth queue pair module that needs to establish a queue pair connection with the first target channel adapter module, a third queue pair module that is connected to the host channel adapter module, and a second parsing module for receiving remote direct data access data packets sent by the host channel adapter module, parsing the remote direct data access data packets, and identifying the first target channel adapter module based on the parsed remote direct data access data packets.

[0112] The data transmission system provided by the embodiment of the present application includes: a host channel adapter module, a first target channel adapter module and a second target channel adapter module; the host channel adapter module is communicatively connected to the first target channel adapter module and the second target channel adapter module; the first target channel adapter module and the second target channel adapter module are communicatively connected; the host channel adapter module is used to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module according to the remote direct data access operation data packet. That is, the embodiment of the present application remotely controls the data transmission between the target channel adapter modules through the host channel adapter module, thereby realizing that data can be directly transmitted between the target channel adapter devices without passing through the host channel adapter module device without adding any devices, thereby greatly reducing the data transmission delay and improving the network bandwidth utilization.

[0113] Referring to Figure 4, Figure 4 shows a schematic diagram of a target channel adapter module provided by an embodiment of the present application. In some embodiments of the present application, any server can serve as a host, and the host is connected to it via a PCI-E slot as a host channel adapter module. Two accelerator cards that are not connected to the host and are independently powered are the first target channel adapter module and the second target channel adapter module. The structure of the accelerator card is shown in Figure 4. The main chip can be an FPGA, and the external devices include sdram, PCI-E slots and 100G optical modules. Through the method and device of the present application, it is possible to realize that the host controls the data in the sdram of the first target channel adapter module through the host channel adapter module, and directly moves it to the sdram of the second target channel adapter module through Ethernet without forwarding it through the host channel adapter module.

[0114] Therefore, the embodiment of the present application can realize direct remote data access and transmission between target channel adapters without adding servers, improve network utilization, reduce data transmission delay, and is easy to implement in FPGA, which can significantly reduce the delay of distributed computing in the FPGA cloud platform.

[0115] 5 , a flowchart of a data transmission method according to an embodiment of the present application is shown. The method may include:

[0116] Step 101: upon detecting that a connection is established between a first target channel adapter module and a second target channel adapter module, receiving a remote direct data access operation data packet sent by a host channel adapter module, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module.

[0117] Step 102 : Sending target storage data corresponding to the first target channel adapter module to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

[0118] In some embodiments of the present application, step 102 includes: parsing the remote direct data access operation data packet, writing the first work queue event in the remote work queue event register data into the internal register of the remote work queue event engine module according to the parsed remote direct data access operation data packet, and writing the physical address of the remote completion queue event register into the internal register of the remote completion queue event engine module; the remote work queue event engine module sends the first work queue event to the second queue pair module; the second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the first work queue event, and generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, so that the second target channel adapter module obtains the target remote direct data access operation data packet.

[0119] It should be noted that in the above steps 101-102, when it is detected that the first target channel adapter module and the second target channel adapter module have established a connection, a remote direct data access operation data packet sent by the host channel adapter module is received, wherein the remote direct data access operation data packet includes the physical address of the remote work queue event register data and the remote completion queue event register;

[0120] It should be noted that, in some embodiments of the present application, the first target channel adapter module includes an rwqe_engine module, an rcqe_engine module, a first queue pair module, a second queue pair module, and a first parsing module; the rwqe_engine module is used to form a WQE with the work queue event WQE data recorded in the internal register of the first queue pair module and send it to the second queue pair module, and trigger a remote direct data access operation; the rcqe_engine module is used to form a WQE with the completion queue event CQE information returned by the second queue pair module and the physical address corresponding to the host channel adapter module recorded in the internal register of the first queue pair module, and send it to the first queue pair module, so that the first queue pair module returns the CQE information to the host channel adapter module. The first parsing module is used to receive the remote direct data access data packet sent by the host channel adapter module, parse the remote direct data access data packet, and convert the parsed remote direct data access data packet into a data bus write operation.

[0121] After establishing a queue pair connection with the second target channel adapter module, the first target channel adapter module can perform data transmission. At this time, the first target channel adapter module can receive a remote direct data access operation data packet sent by the host channel adapter module. The remote direct data access operation data packet includes the remote work queue event register data and the physical address of the remote completion queue event register.

[0122] It should be noted that before data transmission, the host memory corresponding to the host connected to the host channel adapter module pre-sets the remote completion queue event register and the remote work queue event register. The remote completion queue event register is used to store the CQE information returned by the target channel adapter, while the remote work queue event register is used to store the information required to initiate remote direct data access operations in the remote write register format. This information includes the remote direct data access WQE information and the physical address of the remote completion queue event register, which correspond to the internal registers of the rwqe_engine module and the rcqe_engine module in the target channel adapter, respectively.

[0123] Therefore, the remote direct data access operation data packet sent by the host channel adapter module to the first target channel adapter module includes the physical address of the remote work queue event register data and the remote completion queue event register.

[0124] In some embodiments of the present application, receiving a remote direct data access data packet sent by a host channel adapter module includes: upon detecting that a preset configuration in the host channel adapter module is completed and a trigger register corresponding to a fifth queue pair module in the host channel adapter module is in a triggered state, receiving the remote direct data access data packet sent by the host channel adapter module.

[0125] Specifically, the local storage address of the WQE in the host memory is set to the physical address of the remote work queue event register, the destination address is set to the reg_operation module address of the first target channel adapter module, and the data length is the size of the remote work queue event register. When the host performs a trigger operation on the trigger register of the fifth queue pair module of the host channel adapter module, the fifth queue pair module can automatically obtain the WQE in the host memory, automatically obtain the remote work queue event register data based on the WQE, and compose a remote direct data access operation data packet to send to the first target channel adapter module.

[0126] It should be noted that in some embodiments of the present application, after receiving a remote direct data access operation data packet, the data packet will be parsed. The parsed remote direct data access operation data packet will write the first WQE in the remote work queue event register data into the internal register of the rwqe_engine module, and write the physical address of the remote completion queue event register into the internal register of the rcqe_engine module.

[0127] Specifically, the reg_operation module of the first target channel adapter module receives the data packet and parses the payload, and writes the WQE information of the remote direct data access in the remote work queue event register data into the internal register of the rwqe_engine module according to the register address; and writes the physical address of the remote completion queue event register into the internal register of the rcqe_engine module.

[0128] It should be noted that in some embodiments of the present application, the rwqe_engine module sends the WQE of the remote direct data access to the WQE port of the queue pair module according to the register configuration, triggering the remote direct data access to write data.

[0129] In some embodiments of the present application, the WQE includes a WQE identifier, a storage physical address of a first target channel adapter module, a destination storage address of a second target channel adapter module, and a data byte length. The second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the WQE, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module. The steps include: the second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the storage physical address of the first target channel adapter module, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module according to the destination storage address of the second target channel adapter module.

[0130] It should be noted that in some embodiments of the present application, the WQE includes information such as a WQE identifier (WQE_id), the local (first target channel adapter module) physical address where the data is stored (local_addr), the destination storage address (remote_addr) where the data is to be sent to the remote (second target channel adapter module), and the data byte length. Based on the WQE information, the second queue pair module automatically obtains the data stored locally (the first target channel adapter module), composes a target remote direct data access operation data packet, and sends it to the second target channel adapter module.

[0131] In some embodiments of the present application, after the second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the first WQE, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, the method includes: receiving a data transmission completion signal returned by the second target channel adapter module, and determining, according to the data transmission completion signal, that the second target channel adapter module receives the target remote direct data access operation data packet.

[0132] In some embodiments of the present application, receiving a data transmission completion signal returned by the second target channel adapter module, and determining that the second target channel adapter module receives the target remote direct data access operation data packet based on the data transmission completion signal includes: receiving an acknowledgment character (ACK) data packet returned by the fourth queue pair module in the second target channel adapter module based on the remote direct data access protocol, and determining that the second target channel adapter module receives the target remote direct data access operation data packet based on the ACK data packet.

[0133] In some embodiments of the present application, after receiving an ACK data packet returned by the fourth queue pair module in the second target channel adapter module based on the remote direct data access protocol, and determining, based on the ACK data packet, that the second target channel adapter module receives the target remote direct data access operation data packet, the method includes: generating a CQE based on the ACK data packet; generating a second WQE based on the CQE and the physical address of the remote completion queue event register, and sending the second WQE to the host channel adapter module, so that the host channel adapter module stores the CQE information in the remote completion queue event register based on the second WQE.

[0134] It should be noted that in some embodiments of the present application, the fourth queue pair module of the second target channel adapter module stores data into the preset second target channel adapter module storage space according to the remote_addr (destination storage address of the data sent to the second target channel adapter module) information contained in the remote direct data access data packet, and returns an ACK data packet to the first target channel adapter module in accordance with the remote direct data access protocol. After the second queue pair module of the first target channel adapter module receives the ACK data packet, it determines that the data transmission is completed and generates CQE information to the rcqe_engine module. The rcqe_engine module combines the CQE information with the physical address of the remote completion queue event register recorded in the register to form a WQE, and sends it to the first queue pair module connected to the host channel adapter module. The first queue pair module combines the CQE information into a remote direct data access data packet according to the WQE and sends it to the host channel adapter module.

[0135] In an embodiment of the present application, when detecting that a connection is established between a first target channel adapter module and a second target channel adapter module, a remote direct data access operation data packet sent by a host channel adapter module is received, wherein the remote direct data access operation data packet includes remote work queue event register data and a physical address of a remote completion queue event register; the remote direct data access operation data packet is parsed, and according to the parsed remote direct data access operation data packet, a first WQE in the remote work queue event register data is written into an internal register of the rwqe_engine module, and the physical address of the remote completion queue event register is written into an internal register of the rcqe_engine module; the rwqe_engine module sends the first WQE to the second queue pair module; the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the first WQE, generates a target remote direct data access operation data packet according to the target storage data, and sends the remote direct data access operation data packet to the second target channel adapter module, so that the second target channel adapter module obtains the remote direct data access operation data packet. That is, in the embodiment of the present application, when a connection is established between the first target channel adapter module and the second target channel adapter module, the first target channel adapter module directly transmits the data stored in the first target channel adapter module to the second target channel adapter module through remote control of the host channel adapter module. Without adding servers, direct remote data access and transmission of data between target channel adapters is achieved, network utilization is improved, data transmission delay is reduced, and it is easy to implement in FPGA, which can significantly reduce the delay of distributed computing on the FPGA cloud platform.

[0136] 6 , a flowchart of another data transmission method provided by an embodiment of the present application is shown. The method may include:

[0137] Step 001: receiving a first remote direct data access data packet sent by a host channel adapter module, wherein the first remote direct data access data packet includes a physical address corresponding to a second target channel adapter module;

[0138] Step 002: Identify the second target channel adapter module according to the physical address corresponding to the second target channel adapter module;

[0139] It should be noted that in the above steps 001-002, before data transmission between the target channel adapters is performed, it is necessary to establish a queue pair connection between the first target channel adapter module and the second target channel adapter module. Therefore, at this time, the first target channel adapter module will receive the first remote direct data access data packet sent by the host channel adapter module. The first remote direct data access data packet includes the physical address corresponding to the second target channel adapter module, and the second target channel adapter module is identified according to the physical address corresponding to the second target channel adapter module.

[0140] Likewise, the second target channel adapter module also receives the second remote direct data access data packet, thereby identifying the first target channel adapter module.

[0141] Step 101: upon detecting that a connection is established between a first target channel adapter module and a second target channel adapter module, receiving a remote direct data access operation data packet sent by a host channel adapter module, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission from the first target channel adapter module to the second target channel adapter module;

[0142] Step 102 : Sending target storage data corresponding to the first target channel adapter module to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

[0143] It should be noted that, in some embodiments of the present application, the above steps 101-102 refer to the previous discussion and will not be repeated here.

[0144] In an embodiment of the present application, when a connection is established between the first target channel adapter module and the second target channel adapter module, the first target channel adapter module directly transmits data stored in the first target channel adapter module to the second target channel adapter module through remote control of the host channel adapter module. This achieves direct remote data access and transmission between target channel adapters without adding servers, improves network utilization, reduces data transmission latency, and is easy to implement in FPGA, which can significantly reduce the latency of distributed computing on the FPGA cloud platform.

[0145] 7 , a flowchart of another data transmission method provided by an embodiment of the present application is shown. The method may include:

[0146] Step 201 : receiving target storage data sent by a first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module under remote control of the host channel adapter module according to a remote direct data access operation data packet.

[0147] It should be noted that, in some embodiments of the present application, applied to the second target channel adapter module, the first target channel adapter module and the second target channel adapter module are communicatively connected.

[0148] In an embodiment of the present application, when a connection is established between the first target channel adapter module and the second target channel adapter module, the first target channel adapter module directly transmits data stored in the first target channel adapter module to the second target channel adapter module through remote control of the host channel adapter module. This achieves direct remote data access and transmission between target channel adapters without adding servers, improves network utilization, reduces data transmission latency, and is easy to implement in FPGA, which can significantly reduce the latency of distributed computing on the FPGA cloud platform.

[0149] 8 , a flowchart of another data transmission method provided by an embodiment of the present application is shown. The method may include:

[0150] Step 301: Send a remote direct data access operation data packet to a first target channel adapter module, so that the first target channel adapter module transmits data to a second target channel adapter module according to the remote direct data access operation data packet.

[0151] It should be noted that, in some embodiments of the present application, the host channel adapter module is applied, and the host channel adapter module is communicatively connected to a first target channel adapter module and a second target channel adapter module.

[0152] In an embodiment of the present application, when a connection is established between the first target channel adapter module and the second target channel adapter module, the first target channel adapter module directly transmits data stored in the first target channel adapter module to the second target channel adapter module through remote control of the host channel adapter module. This achieves direct remote data access and transmission between target channel adapters without adding servers, improves network utilization, reduces data transmission latency, and is easy to implement in FPGA, which can significantly reduce the latency of distributed computing on the FPGA cloud platform.

[0153] Referring to FIG. 9 , FIG. 9 shows a data transmission device provided in an embodiment of the present application. The device may include:

[0154] The first receiving module 401 is configured to receive a remote direct data access operation data packet sent by the host channel adapter module when detecting that a connection is established between the first target channel adapter module and the second target channel adapter module, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission from the first target channel adapter module to the second target channel adapter module;

[0155] The first sending module 402 is configured to send target storage data corresponding to the first target channel adapter module to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

[0156] In an embodiment of the present application, when a connection is established between the first target channel adapter module and the second target channel adapter module, the first target channel adapter module directly transmits data stored in the first target channel adapter module to the second target channel adapter module through remote control of the host channel adapter module. This achieves direct remote data access and transmission between target channel adapters without adding servers, improves network utilization, reduces data transmission latency, and is easy to implement in FPGA, which can significantly reduce the latency of distributed computing on the FPGA cloud platform.

[0157] Referring to FIG. 10 , FIG. 10 shows another data transmission device provided in an embodiment of the present application. The device may include:

[0158] The second receiving module 501 is configured to receive target storage data sent by the first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module under remote control of the host channel adapter module according to a remote direct data access operation data packet.

[0159] In an embodiment of the present application, when a connection is established between the first target channel adapter module and the second target channel adapter module, the first target channel adapter module directly transmits data stored in the first target channel adapter module to the second target channel adapter module through remote control of the host channel adapter module. This achieves direct remote data access and transmission between target channel adapters without adding servers, improves network utilization, reduces data transmission latency, and is easy to implement in FPGA, which can significantly reduce the latency of distributed computing on the FPGA cloud platform.

[0160] Referring to FIG. 11 , FIG. 11 shows another data transmission device provided in an embodiment of the present application. The device may include:

[0161] The second sending module 601 is configured to send a remote direct data access operation data packet to the first target channel adapter module, so that the first target channel adapter module performs data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

[0162] In an embodiment of the present application, when a connection is established between the first target channel adapter module and the second target channel adapter module, the first target channel adapter module directly transmits data stored in the first target channel adapter module to the second target channel adapter module through remote control of the host channel adapter module. This achieves direct remote data access and transmission between target channel adapters without adding servers, improves network utilization, reduces data transmission latency, and is easy to implement in FPGA, which can significantly reduce the latency of distributed computing on the FPGA cloud platform.

[0163] The embodiment of the present application further provides a communication device, as shown in FIG12 , including a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other via the communication bus 904.

[0164] Memory 903, used for storing computer programs;

[0165] When the processor 901 is used to execute the program stored in the memory 903, it can implement the following steps:

[0166] receiving a remote direct data access operation data packet sent by the host channel adapter module when detecting that a connection is established between the first target channel adapter module and the second target channel adapter module, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module;

[0167] The target storage data corresponding to the first target channel adapter module is sent to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

[0168] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0169] The communication interface is used for communication between the above terminal and other devices.

[0170] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0171] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0172] In another embodiment provided by the present application, a computer non-volatile readable storage medium is also provided. The computer non-volatile readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the data transmissions in the above embodiments.

[0173] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the data transmissions in the above embodiments.

[0174] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a non-volatile computer readable storage medium, or transmitted from a non-volatile computer readable storage medium to another non-volatile computer readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server or a third database by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, server or third database. The non-volatile computer readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a third database that includes one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0175] It should be noted that, in this document, relational terms such as first and first are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0176] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0177] The above are only some embodiments of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A data transmission system, It is characterized in that The data transmission system comprises: a host channel adapter module, a first target channel adapter module and a second target channel adapter module; The host channel adapter module is communicatively connected with the first target channel adapter module and the second target channel adapter module; The first target channel adapter module is communicatively connected with the second target channel adapter module; The host channel adapter module is used to send a remote direct data access operation data packet to the first target channel adapter module, and remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

2. The data transmission system according to claim 1, It is characterized in that The communication connection between the first target channel adapter module and the second target channel adapter module includes: the host channel adapter module sends a first remote direct data access data packet and a second remote direct data access data packet to the first target channel adapter module and the second target channel adapter module respectively, so that the first target channel adapter module and the second target channel adapter module establish a connection according to the first remote direct data access data packet and the second remote direct data access data packet.

3. The data transmission system according to claim 1, It is characterized in that The first target channel adapter module includes a remote work queue event engine module, a remote completion queue event engine module, a first queue pair module, and a second queue pair module; The remote work queue event engine module is used to send the work queue event data recorded in the internal register of the first queue pair module to the second queue pair module, and trigger the remote direct data access operation; The remote completion queue event engine module is used to combine the completion queue event information returned by the second queue pair module and the physical address corresponding to the host channel adapter module recorded in the internal register of the first queue pair module to form a work queue event and send it to the first queue pair module, so that the first queue pair module returns the completion queue event information to the host channel adapter module.

4. The data transmission system according to claim 1, It is characterized in that The first target channel adapter module also includes a first parsing module; The first parsing module is used for receiving a remote direct data access operation data packet sent by the host channel adapter module, parsing the remote direct data access operation data packet, and converting the parsed remote direct data access operation data packet into a data bus write operation.

5. The data transmission system according to claim 1, It is characterized in that The second target channel adapter module includes a third queue pair module and a fourth queue pair module; The third queue pair module is communicatively connected with a sixth queue pair module in the host channel adapter module; The fourth queue pair module is communicatively connected to the second queue pair module in the first target channel adapter module.

6. The data transmission system according to claim 1, It is characterized in that The second target channel adapter module also includes a second parsing module; The second parsing module is used to receive a second remote direct data access data packet sent by the host channel adapter module, and parse the second remote direct data access data packet, and The data access packet identifies the first target channel adapter module.

7. The data transmission system according to claim 1, It is characterized in that The host channel adapter module includes a fifth queue pair module and a sixth queue pair module; The fifth queue pair module is communicatively connected with the third queue pair module in the second target channel adapter module; The sixth queue pair module is communicatively connected to the first queue pair module in the second target channel adapter module.

8. The data transmission system according to claim 1, It is characterized in that The host channel adapter module is further used to send a first remote direct data access data packet to a first parsing module in the first target channel adapter module, the first remote direct data access data packet includes a physical address corresponding to the second target channel adapter module, and to send a second remote direct data access data packet to a second parsing module in the second target channel adapter module, the second remote direct data access data packet includes a physical address corresponding to the first target channel adapter module, so that the first target channel adapter module and the second target channel adapter module establish a connection according to the remote direct data access data packet.

9. The data transmission system according to claim 1, It is characterized in that A remote completion queue event register and a remote work queue event register are pre-set in the host memory corresponding to the host channel adapter module; The remote completion queue event register is used to store the completion queue event information returned by the target channel adapter; The remote work queue event register is used to store remote direct data access operation information according to a preset remote write register format.

10. The data transmission system according to claim 6, It is characterized in that The remote direct data access operation information includes work queue event information and a physical address of a remote completion queue event register; The work queue event corresponds to an internal register of a remote work queue event engine module in the first target channel adapter module; The physical address of the remote completion queue event register corresponds to an internal register of the remote work queue event engine module in the first target channel adapter module.

11. A data transmission method, It is characterized in that Applied to the first target channel adapter module, the data transmission method comprises: In the case of detecting that the first target channel adapter module and the second target channel adapter module are connected, receiving a remote direct data access operation data packet sent by the host channel adapter module, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module; The target storage data corresponding to the first target channel adapter module is sent to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

12. The data transmission method according to claim 11, It is characterized in that The remote direct data access operation data packet includes remote work queue event register data and a physical address of a remote completion queue event register, and sending the target storage data corresponding to the first target channel adapter module to the second target channel adapter module according to the remote direct data access operation data packet so that the second target channel adapter module obtains the target storage data includes: The remote direct data access operation data packet is parsed, and the remote direct data access operation data packet is parsed according to the parsed remote direct data access operation data packet. Writing the first work queue event in the remote work queue event register data into the internal register of the remote work queue event engine module as a data packet, and writing the physical address of the remote completion queue event register into the internal register of the remote completion queue event engine module; The remote work queue event engine module sends the first work queue event to the second queue pair module; The second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the first work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, so that the second target channel adapter module obtains the target remote direct data access operation data packet.

13. The data transmission method according to claim 12, It is characterized in that The work queue event includes a work queue event identifier, a storage physical address of a first target channel adapter module, a destination storage address of a second target channel adapter module, and a data byte length; the second queue pair module obtains target storage data corresponding to the first target channel adapter module according to the work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, including: The second queue pair module obtains the target storage data corresponding to the first target channel adapter module according to the storage physical address of the first target channel adapter module, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module according to the destination storage address of the second target channel adapter module.

14. The data transmission method according to claim 12, It is characterized in that After the second queue pair module acquires the target storage data corresponding to the first target channel adapter module according to the first work queue event, generates a target remote direct data access operation data packet according to the target storage data, and sends the target remote direct data access operation data packet to the second target channel adapter module, the method includes: A data transmission completion signal returned by the second target channel adapter module is received, and according to the data transmission completion signal, it is determined that the second target channel adapter module receives the target remote direct data access operation data packet.

15. The data transmission method according to claim 14, It is characterized in that The step of receiving a data transmission completion signal returned by the second target channel adapter module and determining, according to the data transmission completion signal, that the second target channel adapter module receives the target remote direct data access operation data packet comprises: A confirmation character data packet returned by the fourth queue pair module in the second target channel adapter module based on the remote direct data access protocol is received, and according to the confirmation character data packet, it is determined that the second target channel adapter module receives the target remote direct data access operation data packet.

16. The data transmission method according to claim 15, It is characterized in that After the step of receiving the confirmation character data packet returned by the fourth queue pair module in the second target channel adapter module based on the remote direct data access protocol and determining that the second target channel adapter module receives the target remote direct data access operation data packet according to the confirmation character data packet, the method includes: Generate a completion queue event according to the confirmation character data packet; A second work queue event is generated according to the completion queue event and the physical address of the remote completion queue event register, and the second work queue event is sent to the host channel adapter module so that the host channel adapter module stores the completion queue event information in the remote completion queue event register according to the second work queue event.

17. The data transmission method according to claim 11, It is characterized in that Before the step of receiving the remote direct data access data packet sent by the host channel adapter module, the method includes: receiving a first remote direct data access data packet sent by the host channel adapter module, wherein the first remote direct data access data packet includes a physical address corresponding to the second target channel adapter module; The second target channel adapter module is identified according to a physical address corresponding to the second target channel adapter module.

18. The data transmission method according to claim 11, It is characterized in that The receiving remote direct data access data packet sent by the host channel adapter module comprises: When it is detected that the preset configuration in the host channel adapter module is completed and the trigger register corresponding to the fifth queue pair module in the host channel adapter module is in a triggered state, a remote direct data access data packet sent by the host channel adapter module is received.

19. A data transmission method, It is characterized in that Applied to the second target channel adapter module, the first target channel adapter module is communicatively connected with the second target channel adapter module, and the data transmission method includes: Receive target storage data sent by the first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module under remote control of the host channel adapter module according to a remote direct data access operation data packet.

20. A data transmission method, It is characterized in that Applied to a host channel adapter module, the host channel adapter module is communicatively connected with a first target channel adapter module and a second target channel adapter module, the data transmission method comprises: A remote direct data access operation data packet is sent to the first target channel adapter module, so that the first target channel adapter module performs data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

21. A data transmission device, It is characterized in that Applied to the first target channel adapter module, the data transmission device comprises: A first receiving module is used to receive a remote direct data access operation data packet sent by a host channel adapter module when it is detected that the first target channel adapter module and the second target channel adapter module are connected, wherein the remote direct data access operation data packet is used to remotely control the first target channel adapter module to perform data transmission to the second target channel adapter module; The first sending module is used to send the target storage data corresponding to the first target channel adapter module to the second target channel adapter module according to the remote direct data access operation data packet, so that the second target channel adapter module obtains the target storage data.

22. A data transmission device, It is characterized in that Applied to the second target channel adapter module, the first target channel adapter module is communicatively connected with the second target channel adapter module, and the data transmission device includes: The second receiving module is used to receive the target storage data sent by the first target channel adapter module, wherein the target storage data is transmitted by the first target channel adapter module under remote control of the host channel adapter module according to the remote direct data access operation data packet.

23. A data transmission device, It is characterized in that Applied to a host channel adapter module, the host channel adapter module is communicatively connected with a first target channel adapter module and a second target channel adapter module, the data transmission device comprises: The second sending module is used to send a remote direct data access operation data packet to the first target channel adapter module, so that the first target channel adapter module performs data transmission to the second target channel adapter module according to the remote direct data access operation data packet.

24. A communication device, It is characterized in that include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the data transmission method according to any one of claims 11 to 17, or the data transmission method according to claim 19, or the data transmission method according to claim 20.

25. A non-volatile readable storage medium for storing a program, It is characterized in that When the program is executed by a processor, it implements the data transmission method according to any one of claims 11 to 17, or the data transmission method according to claim 19, or the data transmission method according to claim 20.

Citation Information

Patent Citations

  • PCIe P2P Method and system for transmitting data between storage devices over peer-to-peerP2P connections of PCI-express

    CN109542814A

  • Data processing method and storage system based on NVMe-oF protocol

    CN113014662A

  • Data transmission system, method and device, communication equipment and storage medium

    CN116226027A

  • Cluster wide rebuild reduction against storage node failures

    US20230013798A1