An RDMA read operation method, requesting end and response end

Abstract

This application relates to an RDMA read operation method, a requesting end, and a responding end. The method includes: before sending an RDMA read request, the requesting end first creates CRRL data locally based on the SQ WQE and stores it in SRAM memory. It also stores the relevant information of the SQ WQE in a cache and records the correspondence between tags and memory cell addresses through a TCAM memory. Then, the requesting end sends an RDMA read request to the responding end based on the SQ WQE. When receiving read response data from the responding end, the requesting end again uses the tag to query the memory cell address in the TCAM memory to find the corresponding CRRL data. Finally, it uses the CRRL data to correctly process the received read response data. This application can reduce the cache capacity and complexity of the RDMA subsystem, reduce unnecessary PCIe read / write operations, save PCIe bandwidth and power consumption, and reduce host-side memory.

Description

Technical Field

[0001] This application relates to the field of RDMA (Remote Direct Memory Access) technology, specifically to an RDMA read operation method, a requesting end, a responding end, and an RDMA system. Background Technology

[0002] like Figure 1 The diagram shows an RDMA system, which includes a requester and a responder. The processing flow of an RDMA SQ WQE is as follows:

[0003] The requesting end's sending processor (RequesterTXP) receives the sending queue work queue unit (SQWQE) issued by the driver, obtains relevant information for receiving read response data (including but not limited to work queue element index, starting data packet sequence number, and receive read data address) based on the RDMA read work queue element, and generates an RDMA read request and a CRRL (client-side RDMA read list) data structure. The CRRL data structure contains relevant information for receiving read response data. The CRRL data structure is stored in the requesting end's cache. If the cache is full, it needs to be stored in the host DDR and the RDMA read request is sent to the responding end.

[0004] The Responder RXP receives the RDMA read request, obtains relevant information for generating read response data based on the RDMA read request, and generates an SRRL (Server-Side RDMA Read List) data structure. The SRRL data structure contains relevant information for generating read response data. The SRRL data structure is stored in the Responder's cache. If the cache is full, it needs to be stored in the host DDR. The Responder RXP also generates a Receive Queue Request (RQ doorbell) to the Responder TXP, triggering the Responder TXP to generate read response data.

[0005] The sending processor at the response end responds to the receiving queue request, reads the SRRL data structure to obtain relevant information for generating read response data, and generates read response data based on the relevant information for generating read response data;

[0006] The requester's receiver processor (RequesterRXP) receives the read response data, reads the CRRL data structure to obtain relevant information about receiving the read response data, processes the read response data according to the relevant information, and generates an SQ CQE (Send Queue Complete Queue Unit) to the host driver; thus, one SQWQE process is completed.

[0007] exist Figure 1 In this scheme, each queue pair (QP) on the requesting side needs a CRRL, and each queue pair (QP) on the responding side needs a SRRL. The RDMA subsystem's cache needs to allocate resources for these CRRLs and SRRLs. CRRLs and SRRLs, like other RDMA resources, are stored in the RDMA subsystem's cache, which increases the cache capacity. Furthermore, the presence of CRRLs and SRRLs in the cache along with other RDMA resources increases the cache's complexity. If the resources in the cache are insufficient, the cache needs to write some infrequently used resources to the host's DDR via the PCIe bus, and then read them back from the host's DDR via the PCIe bus when needed. This consumes valuable PCIe bandwidth and increases power consumption. Summary of the Invention

[0008] The purpose of this application is to propose an RDMA read operation method, a requesting end, a response end, and an RDMA system to reduce the cache capacity and complexity of the RDMA subsystem, reduce unnecessary PCIe read and write operations, and save valuable PCIe bandwidth and power consumption.

[0009] To achieve the above objectives, according to a first aspect of this application, an RDMA read operation method is provided, the method comprising:

[0010] The receiving processor at the requesting end receives the SQ WQE sent by the host, caches the SQ WQE in the cache, obtains the CRRL tag and CRRL data based on the SQ WQE, and sends a CRRL creation command to the storage control module at the requesting end; the CRRL creation command includes the CRRL tag and CRRL data.

[0011] The requesting storage control module stores the CRRL data into a free storage cell in the SRAM memory according to the CRRL creation command, and stores the address of the storage cell and the CRRL tag into the TCAM memory;

[0012] The sending processor on the requesting end generates a read request based on the SQ WQE and sends the read request to the responding end, so that the responding end returns read response data based on the read request;

[0013] The receiving processor on the requesting end receives the read response data returned by the responding end and sends a CRRL query command to the storage control module on the requesting end; the CRRL query command includes the CRRL tag;

[0014] The storage control module of the requesting end responds to the CRRL query command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, reads the CRRL data from the SRAM memory according to the address of the storage unit, and returns the CRRL data to the receiving processor of the requesting end.

[0015] The receiving processor on the requesting end processes the read response data returned by the responding end according to the CRRL data, generates an SQ CQE according to the SQ WQE stored in the cache, and sends the SQ CQE to the host.

[0016] The RDMA read operation method of the first aspect of this application extracts the CRRL data from the cache on the requesting end and stores it separately in SRAM memory. It uses TCAM memory to store the CRRL tag and memory cell address corresponding to the CRRL data. By changing the data storage method, the dependence on host memory (DDR) and requesting end cache is reduced. The types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the requesting end cache are reduced, unnecessary PCIe read and write operations are reduced, and valuable PCIe bandwidth and power consumption are saved.

[0017] Preferably, the CRRL tag includes a queue pair functional identifier, a queue pair index, and a CRRL index.

[0018] Preferably, the method further includes:

[0019] When the receiving processor of the requesting end finishes processing the last read response data corresponding to the read request, it sends a CRRL deletion command to the storage control module of the requesting end; the CRRL deletion command includes the CRRL tag;

[0020] The storage control module on the requesting end responds to the CRRL deletion command, queries the TCAM memory based on the CRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the CRRL tag from the TCAM memory.

[0021] Preferably, the method further includes:

[0022] The requesting end's storage control module responds to the CRRL creation command and returns a full storage signal to the requesting end's receiving processor when there are no free storage cells in the SRAM memory.

[0023] The receiving processor at the requesting end pauses sending the read request to the responding end based on the signal that the storage unit is full.

[0024] According to a second aspect of this application, another RDMA read operation method is provided, the method comprising:

[0025] The receiving processor at the response end receives the read request sent by the requesting end, obtains the SRRL tag and SRRL data according to the read request, and sends an SRRL creation command to the storage control module at the response end; the SRRL creation command includes the SRRL tag and SRRL data;

[0026] The storage control module at the response end responds to the SRRL creation command by storing the SRRL data in an empty storage cell in the SRAM memory, and storing the address of the storage cell and the SRRL tag in the TCAM memory;

[0027] The receiving processor at the response end sends a doorbell signal to the sending processor at the response end to request the sending of read response data;

[0028] The transmitting processor at the responding end responds to the doorbell by sending an SRRL query command to the storage control module at the responding end; the SRRL query command includes the SRRL tag;

[0029] The storage control module at the response end responds to the SRRL query command, queries the TCAM memory based on the SRRL tag to obtain the address of the storage unit, reads the SRRL data from the SRAM memory based on the address of the storage unit, and returns the SRRL data to the sending processor at the response end.

[0030] The sending processor at the response end obtains the read response data based on the SRRL data and sends the read response data to the requesting end.

[0031] The RDMA read operation method provided in the second aspect of this application extracts the SRRL data from the requesting cache and stores it separately in SRAM memory. At the same time, the SRRL tag and memory cell address corresponding to the SRRL data are stored in TCAM memory. By changing the data storage method, the dependence on host memory (DDR) and response cache is reduced. The types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the response cache are reduced, unnecessary PCIe read and write operations are reduced, and valuable PCIe bandwidth and power consumption are saved.

[0032] Preferably, the SRRL tag includes a function identifier for the queue pair, a queue pair index, and an SRRL index.

[0033] Preferably, storing the SRRL data in a free storage cell of the SRAM memory, and storing the address of the storage cell and the SRRL tag in the TCAM memory, includes:

[0034] The SRRL index is represented in binary;

[0035] When the least significant bit of the SRRL index is 0, the SRRL data is stored in the first part of the storage space of the free storage cell in the SRAM memory, and the address of the storage cell and the SRRL tag are stored in the TCAM memory.

[0036] When the least significant bit of the SRRL index is 1, the address of the storage cell is obtained by querying the TCAM memory according to the SRRL tag, and the SRRL data is stored in the second part of the storage space of the storage cell corresponding to the address of the storage cell in the SRAM memory.

[0037] Preferably, the method further includes:

[0038] When the sending processor at the response end completes the sending of the last read response data corresponding to the read request, and the least significant bit of the SRRL index of the last read response data is 1, it sends an SRRL deletion command to the storage control module at the response end; the SRRL deletion command includes the SRRL tag;

[0039] The storage control module at the response end responds to the SRRL deletion command, queries the TCAM memory based on the SRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the SRRL tag from the TCAM memory.

[0040] Preferably, the method further includes:

[0041] The storage control module at the response end responds to the SRRL creation command. When there are no free storage cells in the SRAM memory, it returns a storage cell full signal to the sending processor at the response end.

[0042] The sending processor at the responding end sends a negative response signal to the requesting end based on the signal that the storage unit is full, so as to notify the requesting end to wait for a preset time before sending the read request.

[0043] According to a third aspect of this application, a request terminal for implementing RDMA read operations is provided, including a transmit processor, a storage control module, a receive processor, a cache, an SRAM memory, and a TCAM memory;

[0044] The receiving processor on the requesting end is used to receive the SQ WQE sent by the host, cache the SQ WQE in the cache, obtain the CRRL tag and CRRL data according to the SQ WQE, and send a CRRL creation command to the storage control module on the requesting end; the CRRL creation command includes the CRRL tag and CRRL data.

[0045] The storage control module on the requesting end is used to store the CRRL data into a free storage cell in the SRAM memory according to the CRRL creation command, and store the address of the storage cell and the CRRL tag into the TCAM memory;

[0046] The sending processor on the requesting end is used to generate a read request based on the SQ WQE and send the read request to the responding end so that the responding end returns read response data based on the read request;

[0047] The receiving processor on the requesting end is also used to receive read response data returned by the responding end and send a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag;

[0048] The storage control module of the requesting end is also used to respond to the CRRL query command, query the TCAM memory according to the CRRL tag to obtain the address of the storage unit, read the CRRL data from the SRAM memory according to the address of the storage unit, and return the CRRL data to the receiving processor of the requesting end;

[0049] The receiving processor on the requesting end is also used to process the read response data returned by the responding end according to the CRRL data, generate an SQ CQE according to the SQ WQE stored in the Cache, and send the SQ CQE to the host.

[0050] Preferably, the CRRL tag includes a queue pair functional identifier, a queue pair index, and a CRRL index.

[0051] According to a fourth aspect of this application, a response terminal for implementing RDMA read operations is provided, including a transmitting processor, a storage control module, a TCAM memory, an SRAM memory, and a receiving processor;

[0052] The receiving processor at the response end is used to receive the read request sent by the requesting end, obtain the SRRL tag and SRRL data according to the read request, and send an SRRL creation command to the storage control module of the response end; the SRRL creation command includes the SRRL tag and SRRL data;

[0053] The storage control module at the response end is used to respond to the SRRL creation command, store the SRRL data in an empty storage cell in the SRAM memory, and store the address of the storage cell and the SRRL tag in the TCAM memory;

[0054] The receiving processor at the response end is used to send a doorbell signal to the sending processor at the response end to request the sending of read response data;

[0055] The sending processor at the response end is used to respond to the doorbell and send an SRRL query command to the storage control module at the response end; the SRRL query command includes the SRRL tag;

[0056] The storage control module at the response end is used to respond to the SRRL query command, query the TCAM memory according to the SRRL tag to obtain the address of the storage unit, read the SRRL data from the SRAM memory according to the address of the storage unit, and return the SRRL data to the sending processor at the response end;

[0057] The sending processor at the response end is used to obtain read response data based on the SRRL data and send the read response data to the requesting end.

[0058] Preferably, the SRRL tag includes a queue pair functional identifier, a queue pair index, and an SRRL index.

[0059] Preferably, the SRRL index is represented in binary;

[0060] The storage control module at the response end is used to store the SRRL data into the first part of the storage space of the free storage cell in the SRAM memory when the least significant bit of the SRRL index is 0, and to store the address of the storage cell and the SRRL tag into the TCAM memory; when the least significant bit of the SRRL index is 1, it queries the TCAM memory according to the SRRL tag to obtain the address of the storage cell, and stores the SRRL data into the second part of the storage space of the storage cell in the SRAM memory corresponding to the address of the storage cell.

[0061] According to a fifth aspect of this application, an RDMA system is provided, including a requesting end as described in the third aspect and a responding end as described in the fourth aspect.

[0062] Other features and advantages of this application will be set forth in the following description. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings required in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of an RDMA read operation method disclosed in the background art.

[0065] Figure 2 This is a flowchart of an RDMA read operation method according to Embodiment 1 of this application.

[0066] Figure 3 This is an example diagram of an RDMA system according to Embodiment 1 of this application.

[0067] Figure 4 This is a flowchart of an RDMA read operation method according to Embodiment 2 of this application. Detailed Implementation

[0068] The detailed description of the accompanying drawings is intended to illustrate the present preferred embodiments of this application and is not intended to represent only the forms in which this application can be implemented. It should be understood that the same or equivalent functions may be performed by different embodiments intended to be included within the scope of this application.

[0069] See Figure 2 Embodiment 1 of this application provides an RDMA read operation method, the method comprising:

[0070] Step S11: The receiving processor of the requesting end receives the SQ WQE sent by the host, caches the SQ WQE in the Cache, obtains the CRRL tag and CRRL data according to the SQ WQE, and sends a CRRL creation command to the storage control module of the requesting end; the CRRL creation command includes the CRRL tag and CRRL data.

[0071] Specifically, the requester's (TXP) send processor receives the SQWQE (Send Queue Work Completion Unit) from the host driver, corresponding to... Figure 3 Step ①: The sending processor on the requesting end can parse the SQ WQE to obtain the CRRL tag and CRRL data. The CRRL tag can be understood as the identity of the CRRL data, used to distinguish different CRRL data. In this embodiment, the requesting end is equipped with a dedicated storage module for storing CRRL data. This storage module is independent of the requesting end's cache. The SQ WQE is cached in the requesting end's cache for later use.

[0072] Step S12: The storage control module of the requesting end stores the CRRL data into an empty storage cell in the SRAM memory according to the CRRL creation command, and stores the address of the storage cell and the CRRL tag into the TCAM memory.

[0073] Specifically, this embodiment sets up an SRAM memory and a TCAM memory (Ternary Content Addressable Memory). Using TCAM, the memory cell address can be quickly queried. After receiving the CRRL creation command, the storage control module on the requesting end queries the current status of each memory cell in the SRAM memory, selects a memory cell that is currently free, stores the CRRL data in the free memory cell, and stores the address of this memory cell and the corresponding CRRL tag together in the TCAM memory. This is used to quickly find the address of the memory cell where the CRRL data is located based on the CRRL tag later.

[0074] Step S13: The sending processor of the requesting end generates a read request according to the SQ WQE and sends the read request to the responding end, so that the responding end returns read response data according to the read request;

[0075] Specifically, the requesting end's sending processor generates a standard RDMA read request based on SQ WQE (SQ WQE) and sends it to the responding end for processing. Figure 3In steps ② and ③, after receiving the RDMA read request, the responder executes the method steps of Embodiment 2 disclosed below and returns at least one read response data corresponding to the read request.

[0076] Step S14: The receiving processor of the requesting end receives the read response data returned by the responding end and sends a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag;

[0077] Specifically, the receiving processor (RXP) on the requesting end receives the read response data from the responding end, corresponding to... Figure 3 In step ⑦, the receiving processor on the requesting end needs to process the read response data based on the CRRL data (e.g., where the read response data should be stored). The receiving processor on the requesting end sends a CRRL query command to the storage control module on the requesting end. The CRRL query command contains the CRRL tag corresponding to the CRRL data and retrieves the CRRL data returned by the storage control module on the requesting end. Figure 3 Step ⑧ in the process; Figure 3 The storage module includes a storage control module, SRAM memory, and TCAM memory.

[0078] Step S15: The storage control module of the requesting end responds to the CRRL query command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, reads the CRRL data from the SRAM memory according to the address of the storage unit, and returns the CRRL data to the receiving processor of the requesting end.

[0079] Specifically, the requesting end's storage control module uses the CRRL tag to search in the TCAM memory to obtain the storage cell address of the SRAM memory where the CRRL data is located. Based on the storage cell address, it reads the previously stored CRRL data from the SRAM memory and returns the read CRRL data to the requesting end's receiving processor.

[0080] Step S16: The receiving processor of the requesting end processes the read response data returned by the responding end according to the CRRL data, generates SQ CQE according to the SQ WQE stored in the Cache, and sends the SQ CQE to the host.

[0081] Specifically, after the receiving processor on the requesting end receives the CRRL data returned from the storage control module, it correctly processes the received read response data based on the CRRL data (e.g., copying the data to the specified memory address), and finally generates an SQ CQE (Send Queue Work Complete Unit), notifying the host driver that the SQ WQE has been completed. Figure 3 Step 9 in the process.

[0082] The RDMA read operation method in this embodiment extracts the CRRL data from the requesting cache and stores it separately in SRAM memory. It uses TCAM memory to store the CRRL tag and memory cell address corresponding to the CRRL data. By changing the data storage method, the dependence on host memory (DDR) and requesting cache is reduced. The types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the requesting cache are reduced, unnecessary PCIe read and write operations are reduced, and valuable PCIe bandwidth and power consumption are saved.

[0083] In some embodiments, the CRRL tag includes a function identifier (function_id), a queue pair index (qpn, queue pair number), a data type (type), and a CRRL index (entry_idx), wherein the data type of the CRRL tag is CRRL.

[0084] Specifically, `qpn` is used to uniquely identify a queue pair (QP). A QP can support multiple RDMA operations (read, write, etc.); `function_id` is used to distinguish which virtual machine a QP belongs to; `type` is used to distinguish between CRRL data and SRRL data. In RDMA communication, the RDMA device simultaneously acts as both a receiver and a responder when communicating with other devices. That is, when acting as a responder, it involves the processing of SRRL data, such as... Figure 3 As shown, both the receiving and responding ends' storage modules contain CRRL and SRRL data. Therefore, in this embodiment, the storage module distinguishes CRRL and SRRL data according to type and performs corresponding storage operations. Each CRRL data has a sequence number (entry_idx), and a CRRL data can be uniquely identified based on function_id, qpn, type, and entry_idx.

[0085] like Figure 3As shown, assuming a CRRL data width of 512 bits and an SRRL data width of 256 bits, a storage unit of 512 bits can hold one CRRL data entry or two SRRL data entries. Two adjacent SRRL data entries from the same function and the same QP can be placed in the same storage unit. Based on the packet rate of a typical scenario, the required number of CRRL and SRRL entries can be calculated. For example, if the RTT (Round-Trip Time) is 10µs and the RDMA read request packet rate (pps) is 100M, then the required CRRL entries are 10µs * 100M = 1000. With some design margin, the required number of CRRL entries can be set at 1200. Similarly, the required number of SRRL entries can be calculated. It should be noted that while two SRRL data entries occupy one storage unit, not every storage unit can store two valid SRRL data entries; therefore, a larger design margin is required.

[0086] In some embodiments, the method of Embodiment 1 further includes:

[0087] Step S17: When the receiving processor of the requesting end finishes processing the last read response data corresponding to the read request, it sends a CRRL deletion command to the storage control module of the requesting end; the CRRL deletion command includes the CRRL tag;

[0088] Specifically, when the receiving processor (Requester RXP) of the requesting end completes the processing of the last read response data corresponding to the read request, the entire RDMA read operation is completed and the CRRL data is no longer needed to associate or process subsequent data packets or statuses. At this time, the receiving processor of the requesting end generates a CRRL deletion command and sends the deletion command containing the CRRL tag to the storage control module of the requesting end.

[0089] In step S18, the storage control module of the requesting end responds to the CRRL deletion command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the CRRL tag from the TCAM memory.

[0090] Specifically, the storage control module on the requesting end, based on the CRRL deletion command, uses the CRRL tag (containing functionid, qpn, type=CRRL, entry_idx) to query the TCAM memory (TCAM) to find the address in the SRAM memory where the CRRL data is stored. Based on the queried address, the corresponding CRRL data item is deleted (cleared or marked as invalid) from the SRAM memory, which releases the SRAM memory address, which can be reused by subsequent RDMA operations. At the same time, the storage control module also deletes the entry in the TCAM (TCAM memory) corresponding to the CRRL tag (i.e., the address of the storage cell and the CRRL tag itself), which releases the storage space of the TCAM, allowing the TCAM lookup space to be restored, improving the availability and query efficiency of the TCAM, and preventing the TCAM from being affected by too many entries or running out of space.

[0091] In some embodiments, although the required number of CRRL and SRRL entries is calculated based on typical scenarios and design margins are increased, there may still be a situation where there are insufficient CRRL and SRRL entry resources. To address this, the method in Embodiment 1 further includes:

[0092] Step S19: The storage control module of the requesting end responds to the CRRL creation command. When there are no free storage cells in the SRAM memory, it returns a storage cell full signal to the receiving processor of the requesting end.

[0093] Specifically, when the system load is high or SRAM storage resources are very limited, there may be a situation where there are no available free storage cells. In this case, the storage control module cannot complete the creation of CRRL data (because it cannot allocate storage space), so it will return a specific signal to the receiving processor of the requesting end - "Storage cell full signal".

[0094] In step S110, the receiving processor of the requesting end pauses sending the read request to the responding end based on the signal that the storage unit is full.

[0095] Specifically, if the receiving processor on the requesting end receives a "storage unit full signal" from the storage control module, it indicates that the system is currently unable to handle more RDMA read requests because the storage unit resources used to store CRRL data have been exhausted. The receiving processor on the requesting end will suspend sending subsequent read requests to the responding end, and new RDMA read WQEs issued by the driver will not be processed (or processing will be blocked) until CRRL resources in the system are released (such as the aforementioned CRRL deletion mechanism).

[0096] See Figure 4 Embodiment 2 of this application provides an RDMA read operation method, the method comprising:

[0097] Step S21: The receiving processor of the responding end receives the read request sent by the requesting end, obtains the SRRL tag and SRRL data according to the read request, and sends an SRRL creation command to the storage control module of the responding end; the SRRL creation command includes the SRRL tag and SRRL data.

[0098] Specifically, the responder's receiver processor (Responder RXP) receives the RDMA read request sent by the requester, corresponding to... Figure 3 Step ③ in the process; the receiving processor at the responding end obtains the SRRL tag and SRRL data according to the received read request, and then... Figure 3 In step ④, the receiving processor at the response end sends an SRRL creation command to the storage control module at the response end. The SRRL creation command contains the SRRL tag and SRRL data. Figure 3 The storage module includes a storage control module, SRAM memory, and TCAM memory.

[0099] Step S22: The storage control module at the response end responds to the SRRL creation command, stores the SRRL data in an empty storage cell in the SRAM memory, and stores the address of the storage cell and the SRRL tag in the TCAM memory;

[0100] Specifically, after receiving the SRRL creation command, the storage control module at the response end queries the current status of each storage cell in the SRAM memory, selects a storage cell that is currently free, stores the SRRL data in the free storage cell, and stores the address of this storage cell and the corresponding SRRL tag together in the TCAM memory, which is used to quickly find the address of the storage cell where the SRRL data is located based on the SRRL tag later.

[0101] Step S23: The receiving processor at the responding end sends a doorbell signal to the sending processor at the responding end to request the sending of read response data;

[0102] Specifically, after creating the SRRL, the receiver processor at the responding end sends a Receive Queue Request (RQ DB, Receive queue doorbell) to the send processor (Responder TXP) at the responding end. Figure 3 Step ⑤ in the process is used to notify the sending processor of the responding end that there is new read response data to be sent.

[0103] Step S24: The transmitting processor at the responding end responds to the doorbell by sending an SRRL query command to the storage control module at the responding end; the SRRL query command includes the SRRL tag;

[0104] Specifically, the receiver processor (RXP) at the response end receives data from the RQ DB, according to the corresponding... Figure 3 In step ⑦, the receiving processor at the response end needs to generate and send read response data based on the SRRL data. The receiving processor at the response end sends an SRRL query command to the storage control module at the response end. The SRRL query command contains the SRRL tag corresponding to the SRRL data, and retrieves the SRRL data returned by the storage control module at the response end. Figure 3 Step 6 in the process.

[0105] Step S25: The storage control module of the response end responds to the SRRL query command, queries the TCAM memory according to the SRRL tag to obtain the address of the storage unit, reads the SRRL data from the SRAM memory according to the address of the storage unit, and returns the SRRL data to the sending processor of the response end.

[0106] Specifically, this embodiment sets up an SRAM memory and a TCAM memory (Ternary Content Addressable Memory). The TCAM can be used to quickly query the memory cell address. The storage control module of the response end uses the SRRL tag to search in the TCAM memory to obtain the memory cell address of the SRAM memory where the SRRL data is located. Based on the memory cell address, it reads the previously stored SRRL data from the SRAM memory and returns the read SRRL data to the receiving processor of the response end.

[0107] Step S26: The sending processor at the response end obtains the read response data based on the SRRL data and sends the read response data to the requesting end;

[0108] Specifically, the receiving processor at the responding end prepares the read response data based on the information in the SRRL data, and returns at least one read response data to the requesting end, thus completing the RDMA read operation at the responding end.

[0109] The RDMA read operation method in this embodiment extracts the SRRL data from the requesting cache and stores it separately in SRAM memory. At the same time, the SRRL tag and memory cell address corresponding to the SRRL data are stored in TCAM memory. By changing the data storage method, the dependence on host memory (DDR) and response cache is reduced. The types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the response cache are reduced, unnecessary PCIe read and write operations are reduced, and valuable PCIe bandwidth and power consumption are saved.

[0110] In some embodiments, the SRRL tag includes a functional identifier for the queue pair, a queue pair index, and an SRRL index.

[0111] Specifically, `qpn` is used to uniquely identify a queue pair (QP). A QP can support multiple RDMA operations (read, write, etc.); `function_id` is used to distinguish the virtual machine to which the QP belongs; `type` is used to distinguish between CRRL data and SRRL data. In RDMA communication, the RDMA device simultaneously acts as both a receiver and a responder when communicating with other RDMA devices. That is, when acting as a responder, it involves the processing of SRRL data, such as... Figure 3 As shown, both the receiving and responding ends' storage modules contain CRRL and SRRL data. Therefore, in this embodiment, the storage control module distinguishes CRRL and SRRL data according to type and performs corresponding storage operations. Each SRRL data has a sequence number (entry_idx), and an SRRL data can be uniquely identified based on function_id, qpn, type, and entry_idx.

[0112] In some embodiments, storing the SRRL data in a free storage cell of the SRAM memory, and storing the address of the storage cell and the SRRL tag in the TCAM memory, includes:

[0113] The SRRL index is represented in binary;

[0114] When the least significant bit of the SRRL index is 0, the SRRL data is stored in the first part of the storage space of the free storage cell in the SRAM memory, and the address of the storage cell and the SRRL tag are stored in the TCAM memory.

[0115] When the least significant bit of the SRRL index is 1, the address of the storage cell is obtained by querying the TCAM memory according to the SRRL tag, and the SRRL data is stored in the second part of the storage space of the storage cell corresponding to the address of the storage cell in the SRAM memory.

[0116] Specifically, with a CRRL data width of 512 bits and an SRRL data width of 256 bits, a storage unit is 512 bits in size and can hold one CRRL data or two SRRL data. Two adjacent SRRL data of the same function and the same QP are placed in the same storage unit. In this embodiment, the SRRL index is represented in binary so that the location of the SRRL data in the storage unit can be determined according to the least significant bit (0 / 1) of the sequence number.

[0117] In some embodiments, the method further comprises:

[0118] Step S27: When the sending processor of the response end completes the sending of the last read response data corresponding to the read request, and the least significant bit of the SRRL index of the last read response data is 1, an SRRL deletion command is sent to the storage control module of the response end; the SRRL deletion command includes the SRRL tag;

[0119] Specifically, when the Responder RXP finishes processing the last read response data corresponding to the read request, and the least significant bit of the SRRL index of the last read response data is 1, the entire RDMA read operation corresponding to a storage unit has been completed, and the SRRL data is no longer needed to associate or process subsequent data packets or statuses. At this time, the Responder RXP generates an SRRL deletion command and sends the deletion command containing the SRRL tag to the storage control module of the Responder.

[0120] In step S28, the storage control module of the responding end responds to the SRRL deletion command, queries the TCAM memory according to the SRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the SRRL tag from the TCAM memory.

[0121] Specifically, the storage control module (storage module) at the responding end queries the TCAM memory (TCAM) using the SRRL tag (containing functionid, qpn, type=SRRL, entry idx) according to the SRRL deletion command to find the storage cell address in the SRAM memory where the SRRL data is stored. Based on the found address, the corresponding SRRL data item is deleted (cleared or marked as invalid) from the SRAM memory, which releases the SRAM storage cell resources / address, which can be reused by subsequent RDMA operations. At the same time, the storage control module also deletes the entry in the TCAM (TCAM memory) corresponding to the SRRL tag (i.e., the address of the storage cell and the SRRL tag itself), which releases the storage space of the TCAM, allowing the TCAM lookup space to be restored, improving the availability and query efficiency of the TCAM, and preventing the TCAM from being affected by too many entries or exhausting its space.

[0122] In some embodiments, the method further comprises:

[0123] Step S29: The storage control module of the response end responds to the SRRL creation command. When there are no free storage cells in the SRAM memory, it returns a storage cell full signal to the sending processor of the response end.

[0124] Specifically, when the storage control module of the responder receives the SRRL creation command, it attempts to find a free storage cell in the SRAM memory to store the new SRRL data. If no free cell is found (resource exhaustion), the storage control module of the responder will generate a storage cell full signal. This signal is returned to the receiver processor of the responder and sent to the transmitter processor (Responder TXP) of the responder along with the doorbell.

[0125] In step S210, the sending processor of the responding end sends a negative response signal to the requesting end based on the signal that the storage unit is full, so as to notify the requesting end to wait for a preset time before sending the read request.

[0126] Specifically, after receiving a "memory full" signal, the sending processor on the responding end knows it cannot process new read requests or prepare the corresponding read response data. It needs to notify the requesting end of this situation to prevent data loss or system corruption caused by the requesting end continuing to send read requests. The sending processor on the responding end sends a negative acknowledgment signal (RNR NAK) to the requesting end, instructing it to suspend sending read requests. Upon receiving the RNR NAK signal, the requesting end will pause sending new RDMA read requests to the responding end and wait for a preset time period. After this time, the requesting end can attempt to resend the read request.

[0127] Embodiment 3 of this application proposes a requester for executing the RDMA read operation method steps described in Embodiment 1, such as... Figure 3 As shown, the requester includes a send processor (Requester TXP), a storage control module, an SRAM memory, a TCAM memory, and a send processor (Requester RXP).

[0128] The receiving processor on the requesting end is used to receive the SQ WQE sent by the host, cache the SQ WQE in the cache, obtain the CRRL tag and CRRL data according to the SQ WQE, and send a CRRL creation command to the storage control module on the requesting end; the CRRL creation command includes the CRRL tag and CRRL data.

[0129] The storage control module on the requesting end is used to store the CRRL data into a free storage cell in the SRAM memory according to the CRRL creation command, and store the address of the storage cell and the CRRL tag into the TCAM memory;

[0130] The sending processor on the requesting end is used to generate a read request based on the SQ WQE and send the read request to the responding end so that the responding end returns read response data based on the read request;

[0131] The receiving processor on the requesting end is also used to receive read response data returned by the responding end and send a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag;

[0132] The storage control module of the requesting end is also used to respond to the CRRL query command, query the TCAM memory according to the CRRL tag to obtain the address of the storage unit, read the CRRL data from the SRAM memory according to the address of the storage unit, and return the CRRL data to the receiving processor of the requesting end;

[0133] The receiving processor on the requesting end is also used to process the read response data returned by the responding end according to the CRRL data, generate an SQ CQE according to the SQ WQE stored in the Cache, and send the SQ CQE to the host.

[0134] In some embodiments, the CRRL tag includes a functional identifier for the queue pair, a queue pair index, and a CRRL index.

[0135] Embodiment 4 of this application proposes a responder for executing the RDMA read operation method steps described in Embodiment 2, such as... Figure 3 As shown, the responder includes a transmitter processor (Responder TXP), a storage control module, a TCAM memory, an SRAM memory, and a transmitter processor (Responder RXP).

[0136] The receiving processor at the response end is used to receive the read request sent by the requesting end, obtain the SRRL tag and SRRL data according to the read request, and send an SRRL creation command to the storage control module of the response end; the SRRL creation command includes the SRRL tag and SRRL data;

[0137] The storage control module at the response end is used to respond to the SRRL creation command, store the SRRL data in an empty storage cell in the SRAM memory, and store the address of the storage cell and the SRRL tag in the TCAM memory;

[0138] The receiving processor at the response end is used to send a doorbell signal to the sending processor at the response end to request the sending of read response data;

[0139] The sending processor at the response end is used to respond to the doorbell and send an SRRL query command to the storage control module at the response end; the SRRL query command includes the SRRL tag;

[0140] The storage control module at the response end is used to respond to the SRRL query command, query the TCAM memory according to the SRRL tag to obtain the address of the storage unit, read the SRRL data from the SRAM memory according to the address of the storage unit, and return the SRRL data to the sending processor at the response end;

[0141] The sending processor at the response end is used to obtain read response data based on the SRRL data and send the read response data to the requesting end.

[0142] In some embodiments, the SRRL tag includes a functional identifier for the queue pair, a queue pair index, and an SRRL index.

[0143] In some embodiments, the SRRL index is represented in binary;

[0144] The storage control module at the response end is used to store the SRRL data into the first part of the storage space of the free storage cell in the SRAM memory when the least significant bit of the SRRL index is 0, and to store the address of the storage cell and the SRRL tag into the TCAM memory; when the least significant bit of the SRRL index is 1, it queries the TCAM memory according to the SRRL tag to obtain the address of the storage cell, and stores the SRRL data into the second part of the storage space of the storage cell in the SRAM memory corresponding to the address of the storage cell.

[0145] Embodiment 5 of this application proposes an RDMA system, such as Figure 3 As shown, it includes the request terminal described in Embodiment 3 above, and the response terminal described in Embodiment 4 above.

[0146] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many updates and modifications will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An RDMA read operation method, characterized in that, The method includes: The receiving processor at the requesting end receives the SQ WQE sent by the host, caches the SQ WQE in the cache, obtains the CRRL tag and CRRL data based on the SQ WQE, and sends a CRRL creation command to the storage control module at the requesting end; the CRRL creation command includes the CRRL tag and CRRL data. The requesting storage control module stores the CRRL data into a free storage cell in the SRAM memory according to the CRRL creation command, and stores the address of the storage cell and the CRRL tag into the TCAM memory; The sending processor on the requesting end generates a read request based on the SQ WQE and sends the read request to the responding end, so that the responding end returns read response data based on the read request; The receiving processor on the requesting end receives the read response data returned by the responding end and sends a CRRL query command to the storage control module on the requesting end; the CRRL query command includes the CRRL tag; The storage control module of the requesting end responds to the CRRL query command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, reads the CRRL data from the SRAM memory according to the address of the storage unit, and returns the CRRL data to the receiving processor of the requesting end. The receiving processor on the requesting end processes the read response data returned by the responding end according to the CRRL data, generates an SQ CQE according to the SQ WQE stored in the cache, and sends the SQ CQE to the host.

2. The method according to claim 1, characterized in that, The CRRL tag includes a queue pair function identifier, a queue pair index, and a CRRL index.

3. The method according to claim 2, characterized in that, The method further includes: When the receiving processor of the requesting end completes the processing of the last read response data corresponding to the read request, it sends a CRRL deletion command to the storage control module of the requesting end; the CRRL deletion command includes the CRRL tag; The storage control module on the requesting end responds to the CRRL deletion command, queries the TCAM memory based on the CRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the CRRL tag from the TCAM memory.

4. The method according to claim 1, characterized in that, The method further includes: The requesting end's storage control module responds to the CRRL creation command and returns a full storage signal to the requesting end's receiving processor when there are no free storage cells in the SRAM memory. The receiving processor at the requesting end pauses sending the read request to the responding end based on the signal that the storage unit is full.

5. An RDMA read operation method, characterized in that, The method includes: The receiving processor at the response end receives the read request sent by the requesting end, obtains the SRRL tag and SRRL data according to the read request, and sends an SRRL creation command to the storage control module at the response end; the SRRL creation command includes the SRRL tag and SRRL data; The storage control module at the response end responds to the SRRL creation command by storing the SRRL data in an empty storage cell in the SRAM memory, and storing the address of the storage cell and the SRRL tag in the TCAM memory; The receiving processor at the response end sends a doorbell signal to the sending processor at the response end to request the sending of read response data; The transmitting processor at the responding end responds to the doorbell by sending an SRRL query command to the storage control module at the responding end; the SRRL query command includes the SRRL tag; The storage control module at the response end responds to the SRRL query command, queries the TCAM memory based on the SRRL tag to obtain the address of the storage unit, reads the SRRL data from the SRAM memory based on the address of the storage unit, and returns the SRRL data to the sending processor at the response end. The sending processor at the response end obtains the read response data based on the SRRL data and sends the read response data to the requesting end.

6. The method according to claim 5, characterized in that, The SRRL tag includes a function identifier for the queue pair, a queue pair index, and an SRRL index.

7. The method according to claim 6, characterized in that, The step of storing the SRRL data in a free storage cell of the SRAM memory, and storing the address of the storage cell and the SRRL tag in the TCAM memory, includes: The SRRL index is represented in binary; When the least significant bit of the SRRL index is 0, the SRRL data is stored in the first part of the storage space of the free storage cell in the SRAM memory, and the address of the storage cell and the SRRL tag are stored in the TCAM memory. When the least significant bit of the SRRL index is 1, the address of the storage cell is obtained by querying the TCAM memory according to the SRRL tag, and the SRRL data is stored in the second part of the storage space of the storage cell corresponding to the address of the storage cell in the SRAM memory.

8. The method according to claim 5, characterized in that, The method further includes: When the sending processor at the response end completes the sending of the last read response data corresponding to the read request, and the least significant bit of the SRRL index of the last read response data is 1, it sends an SRRL deletion command to the storage control module at the response end; the SRRL deletion command includes the SRRL tag; The storage control module at the response end responds to the SRRL deletion command, queries the TCAM memory based on the SRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the SRRL tag from the TCAM memory.

9. The method according to claim 5, characterized in that, The method further includes: The storage control module at the response end responds to the SRRL creation command. When there are no free storage cells in the SRAM memory, it returns a storage cell full signal to the sending processor at the response end. The sending processor at the responding end sends a negative response signal to the requesting end based on the signal that the storage unit is full, so as to notify the requesting end to wait for a preset time before sending the read request.

10. A request terminal for implementing RDMA read operations, characterized in that, This includes a transmit processor, a storage control module, a receive processor, a cache, an SRAM memory, and a TCAM memory; The receiving processor on the requesting end is used to receive the SQ WQE sent by the host, cache the SQ WQE in the cache, obtain the CRRL tag and CRRL data according to the SQ WQE, and send a CRRL creation command to the storage control module on the requesting end; the CRRL creation command includes the CRRL tag and CRRL data. The storage control module on the requesting end is used to store the CRRL data into a free storage cell in the SRAM memory according to the CRRL creation command, and store the address of the storage cell and the CRRL tag into the TCAM memory; The sending processor on the requesting end is used to generate a read request based on the SQ WQE and send the read request to the responding end so that the responding end returns read response data based on the read request; The receiving processor on the requesting end is also used to receive read response data returned by the responding end and send a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag; The storage control module of the requesting end is also used to respond to the CRRL query command, query the TCAM memory according to the CRRL tag to obtain the address of the storage unit, read the CRRL data from the SRAM memory according to the address of the storage unit, and return the CRRL data to the receiving processor of the requesting end; The receiving processor on the requesting end is also used to process the read response data returned by the responding end according to the CRRL data, generate an SQ CQE according to the SQ WQE stored in the Cache, and send the SQ CQE to the host.

11. The requesting terminal according to claim 10, characterized in that, The CRRL tag includes a queue pair function identifier, a queue pair index, and a CRRL index.

12. A response terminal for implementing RDMA read operations, characterized in that, This includes a transmitting processor, a storage control module, a TCAM memory, an SRAM memory, and a receiving processor; The receiving processor at the response end is used to receive the read request sent by the requesting end, obtain the SRRL tag and SRRL data according to the read request, and send an SRRL creation command to the storage control module of the response end; the SRRL creation command includes the SRRL tag and SRRL data; The storage control module at the response end is used to respond to the SRRL creation command, store the SRRL data in an empty storage cell in the SRAM memory, and store the address of the storage cell and the SRRL tag in the TCAM memory; The receiving processor at the response end is used to send a doorbell signal to the sending processor at the response end to request the sending of read response data; The sending processor at the response end is used to respond to the doorbell and send an SRRL query command to the storage control module at the response end; the SRRL query command includes the SRRL tag; The storage control module at the response end is used to respond to the SRRL query command, query the TCAM memory according to the SRRL tag to obtain the address of the storage unit, read the SRRL data from the SRAM memory according to the address of the storage unit, and return the SRRL data to the sending processor at the response end; The sending processor at the response end is used to obtain read response data based on the SRRL data and send the read response data to the requesting end.

13. The response terminal according to claim 12, characterized in that, The SRRL tag includes a function identifier for the queue pair, a queue pair index, and an SRRL index.

14. The response terminal according to claim 13, characterized in that, The SRRL index is represented in binary; The storage control module at the response end is used to store the SRRL data into the first part of the storage space of the free storage cell in the SRAM memory when the least significant bit of the SRRL index is 0, and to store the address of the storage cell and the SRRL tag into the TCAM memory; when the least significant bit of the SRRL index is 1, it queries the TCAM memory according to the SRRL tag to obtain the address of the storage cell, and stores the SRRL data into the second part of the storage space of the storage cell in the SRAM memory corresponding to the address of the storage cell.

15. An RDMA system, characterized in that, It includes the requesting end as described in claim 10 or 11, and the response end as described in any one of claims 12 to 14.

Images