RDMA read operation method, request end and response end

Abstract

The invention relates to an RDMA read operation method, a request end and a response end, and the method comprises the steps that before the request end sends an RDMA read request, CRRL data is created locally according to SQ WQE, the CRRL data is stored in an SRAM, related information of the SQ WQE is stored in a Cache, the corresponding relation between a tag and a storage unit address is recorded through a TCAM, and the corresponding relation between the tag and the storage unit address is stored in the SRAM; and then the request end sends an RDMA read request to the response end according to the SQ WQE. And when the read response data of the response end is received, the request end uses the tag again to query the address of the storage unit in the TCAM memory, finds the corresponding CRRL data, and finally correctly processes the received read response data by using the CRRL data. According to the method and the device, the Cache capacity and complexity of the RDMA subsystem can be reduced, unnecessary PCIE read-write operation is reduced, and PCIE bandwidth and power consumption as well as a memory of a host side are saved.

Description

Technical Field

[0001] The present application relates to the technical field of RDMA (Remote Direct Memory Access), and in particular to an RDMA read operation method, a request end, a response end, and an RDMA system. Background Art

[0002] like Figure 1 The figure shows a schematic diagram of an RDMA system. The RDMA system includes a requester and a responder. The processing flow of an RDMA SQ WQE is as follows: The send processor (RequesterTXP) on the requesting end receives the send queue work queue element (SQWQE) issued by the driver, obtains relevant information for receiving read response data (including but not limited to the work queue element index, the starting data packet sequence number, the address of the received read data, etc.) according to the RDMA read work queue element, and generates an RDMA read request (RDMAreadrequest) and a CRRL (client-side RDMA read linked list) data structure. The CRRL data structure contains relevant information for receiving read response data. The CRRL data structure is stored in the cache of the requesting end. If the cache is full, it needs to be stored in the host DDR, and the RDMA read request is sent to the responding end. The responder RXP receives the RDMA read request, obtains relevant information for generating read response data (readresponse) based on the RDMA read request, and generates an SRRL (server-side RDMA read linked list) data structure. The SRRL data structure contains relevant information for generating read response data. The SRRL data structure is stored in the responder's cache. If the cache is full, it needs to be stored in the host DDR. The responder RXP also generates a receive queue request (RQ doorbell) to the responder's transmit processor (TXP), triggering the responder's transmit processor to generate read response data. The sending processor of the responding end responds to the receiving queue request, reads the SRRL data structure to obtain relevant information for generating read response data, and generates read response data according to the relevant information for generating read response data; The receiving processor (RequesterRXP) on the requesting end receives the read response data (readresponse), reads the CRRL data structure to obtain relevant information for receiving the read response data, processes the read response data according to the relevant information for receiving the read response data, and generates an SQ CQE (send queue completion queue element) to the host driver; at this point, an SQWQE process is completed.

[0003] exist Figure 1 In the scheme, each queue pair (QP) on the request side needs to have a CRRL, and each queue pair (QP) on the response side needs to have an SRRL. The cache of the RDMA subsystem needs to allocate resources for these CRRLs and SRRLs. CRRLs and SRRLs, like other RDMA resources, are stored in the RDMA subsystem's cache, which increases the cache capacity. CRRLs and SRRLs exist in the cache together with other RDMA resources, which increases the complexity of the cache. If the resources in the cache are insufficient, the cache needs to write some infrequently used resources to the host's DDR through the PCIE bus, and then read them back from the host's DDR through the PCIE bus when they are needed. This takes up valuable PCIE bandwidth and increases power consumption. Summary of the Invention

[0004] The purpose of this application is to propose an RDMA read operation method, a request end, a responder end, and an RDMA system to reduce the cache capacity and complexity of the RDMA subsystem, reduce unnecessary PCIE read and write operations, and save valuable PCIE bandwidth and power consumption.

[0005] To achieve the above object, according to a first aspect of the present application, a RDMA read operation method is provided, the method comprising: The receiving processor of the requesting end receives the SQ WQE sent by the host, caches the SQ WQE in a cache, obtains a CRRL tag and CRRL data according to the SQ WQE, and sends a CRRL creation command to the storage control module of the requesting end; the CRRL creation command includes the CRRL tag and CRRL data; The storage control module of the requesting end stores the CRRL data in an idle storage unit in the SRAM memory according to the CRRL creation command, and stores the address of the storage unit and the CRRL tag in the TCAM memory; The sending processor of the requesting end generates a read request according to the SQ WQE, and sends the read request to the responding end, so that the responding end returns read response data according to the read request; The receiving processor of the requesting end receives the read response data returned by the responding end, and sends a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag; The storage control module of the requesting end responds to the CRRL query command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, reads the CRRL data from the SRAM memory according to the address of the storage unit, and returns the CRRL data to the receiving processor of the requesting end; The receiving processor of the requesting end processes the read response data returned by the responding end according to the CRRL data, generates an SQ CQE according to the SQ WQE stored in the cache, and sends the SQ CQE to the host.

[0006] The RDMA read operation method of the first aspect of the present application strips the CRRL data from the requesting side cache, stores it separately in the SRAM memory, and uses the TCAM memory to store the CRRL tag and storage unit address corresponding to the CRRL data. By changing the data storage method, the dependence on the host memory (DDR) and the requesting side cache is reduced, the types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the requesting side cache are reduced, unnecessary PCIE read and write operations are reduced, and valuable PCIE bandwidth and power consumption are saved.

[0007] Preferably, the CRRL tag includes a queue pair function identifier, a queue pair index and a CRRL index.

[0008] Preferably, the method further comprises: When the receiving processor of the requesting end completes processing of the last read response data corresponding to the read request, the receiving processor sends a CRRL deletion command to the storage control module of the requesting end; the CRRL deletion command includes the CRRL tag; The storage control module of the requesting end responds to the CRRL deletion command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the CRRL tag from the TCAM memory.

[0009] Preferably, the method further comprises: The storage control module of the requesting end responds to the CRRL creation command and returns a storage unit full signal to the receiving processor of the requesting end when there is no free storage unit in the SRAM memory; The receiving processor of the requesting end suspends sending the read request to the responding end according to the storage unit full signal.

[0010] According to a second aspect of the present application, another RDMA read operation method is provided, the method comprising: The receiving processor of the responding end receives the read request sent by the requesting end, obtains the SRRL tag and SRRL data according to the read request, and sends an SRRL creation command to the storage control module of the responding end; the SRRL creation command includes the SRRL tag and SRRL data; The storage control module of the responding end responds to the SRRL creation command, stores the SRRL data in an idle storage unit in the SRAM memory, and stores the address of the storage unit and the SRRL tag in the TCAM memory; The receiving processor of the responding end sends a doorbell to the sending processor of the responding end to request to send the read response data; The sending processor of the responding end responds to the doorbell and sends an SRRL query command to the storage control module of the responding end; the SRRL query command includes the SRRL tag; The storage control module of the responding end responds to the SRRL query command, queries the TCAM memory according to the SRRL tag to obtain the address of the storage unit, reads the SRRL data from the SRAM memory according to the address of the storage unit, and returns the SRRL data to the sending processor of the responding end; The sending processor of the responding end obtains the read response data according to the SRRL data, and sends the read response data to the requesting end.

[0011] The RDMA read operation method provided in the second aspect of the present application strips the SRRL data from the cache of the requesting end and stores it separately in the SRAM memory. At the same time, the SRRL tag and storage unit address corresponding to the SRRL data are stored in the TCAM memory. By changing the data storage method, the dependence on the host memory (DDR) and the responding end cache is reduced, the types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the responding end cache are reduced, unnecessary PCIE read and write operations are reduced, and valuable PCIE bandwidth and power consumption are saved.

[0012] Preferably, the SRRL tag includes a queue pair function identifier, a queue pair index and a SRRL index.

[0013] Preferably, storing the SRRL data in an idle storage unit in an SRAM memory, and storing the address of the storage unit and the SRRL tag in a TCAM memory, comprises: The SRRL index is represented in binary format; When the lowest bit of the SRRL index is 0, storing the SRRL data in a first portion of storage space of an idle storage unit in an SRAM memory, and storing the address of the storage unit and the SRRL tag in a TCAM memory; When the lowest bit of the SRRL index is 1, the TCAM memory is queried according to the SRRL tag to obtain the address of the storage unit, and the SRRL data is stored in the second part of the storage space of the storage unit corresponding to the address of the storage unit in the SRAM memory.

[0014] Preferably, the method further comprises: When the sending processor of the responding end completes sending the last read response data corresponding to the read request, and the lowest bit of the SRRL index of the last read response data is 1, sending an SRRL delete command to the storage control module of the responding end; the SRRL delete command includes the SRRL tag; The storage control module on the responding end responds to the SRRL deletion command, queries the TCAM memory according to the SRRL label to obtain the address of the storage unit, marks the address of the storage unit as idle, and deletes the address of the storage unit and the SRRL label from the TCAM memory.

[0015] Preferably, the method further comprises: The storage control module of the responding end responds to the SRRL creation command, and when there is no free storage unit in the SRAM memory, returns a storage unit full signal to the sending processor of the responding end; The sending processor of the responding end sends a negative acknowledgement signal to the requesting end according to the storage unit full signal, so as to notify the requesting end to wait for a preset time before sending the read request.

[0016] According to a third aspect of the present application, there is provided a request end for implementing an RDMA read operation, comprising a sending processor, a storage control module, a receiving processor, a cache, an SRAM memory, and a TCAM memory; The receiving processor of the requesting end is configured to receive the SQ WQE sent by the host, cache the SQ WQE in a cache, obtain a CRRL tag and CRRL data according to the SQ WQE, and send a CRRL creation command to the storage control module of the requesting end; the CRRL creation command includes the CRRL tag and CRRL data; The storage control module of the requesting end is used to store the CRRL data in an idle storage unit in the SRAM memory according to the CRRL creation command, and store the address of the storage unit and the CRRL tag in the TCAM memory; The sending processor of the requesting end is used to generate a read request according to the SQ WQE, and send the read request to the responding end, so that the responding end returns read response data according to the read request; The receiving processor of the requesting end is further configured to receive the read response data returned by the responding end, and send a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag; The storage control module of the requesting end is further configured to respond to the CRRL query command, query the TCAM memory according to the CRRL tag to obtain the address of the storage unit, read the CRRL data from the SRAM memory according to the address of the storage unit, and return the CRRL data to the receiving processor of the requesting end; The receiving processor of the requesting end is further configured to process the read response data returned by the responding end according to the CRRL data, generate an SQ CQE according to the SQ WQE stored in the cache, and send the SQ CQE to the host.

[0017] Preferably, the CRRL tag includes a queue pair function identifier, a queue pair index and a CRRL index.

[0018] According to a fourth aspect of the present application, there is provided a responder for implementing an RDMA read operation, comprising a sending processor, a storage control module, a TCAM memory, an SRAM memory, and a receiving processor; The receiving processor of the responding end is used to receive the read request sent by the requesting end, obtain the SRRL tag and SRRL data according to the read request, and send an SRRL creation command to the storage control module of the responding end; the SRRL creation command includes the SRRL tag and SRRL data; The storage control module of the responding end is used to respond to the SRRL creation command, store the SRRL data in an idle storage unit in the SRAM memory, and store the address of the storage unit and the SRRL tag in the TCAM memory; The receiving processor of the responding end is used to send a doorbell to the sending processor of the responding end to request to send the read response data; The sending processor of the responding end is used to respond to the doorbell and send an SRRL query command to the storage control module of the responding end; the SRRL query command includes the SRRL tag; The storage control module of the responding end is used to respond to the SRRL query command, query the TCAM memory according to the SRRL label to obtain the address of the storage unit, read the SRRL data from the SRAM memory according to the address of the storage unit, and return the SRRL data to the sending processor of the responding end; The sending processor of the responding end is used to obtain read response data according to the SRRL data, and send the read response data to the requesting end.

[0019] Preferably, the SRRL tag includes a queue pair function identifier, a queue pair index and an SRRL index.

[0020] Preferably, the SRRL index is represented in binary format; The storage control module of the responding end is used to store the SRRL data in the first part of the storage space of the idle storage unit in the SRAM memory when the lowest bit of the SRRL index is 0, and store the address of the storage unit and the SRRL label in the TCAM memory; when the lowest bit of the SRRL index is 1, query the TCAM memory according to the SRRL label to obtain the address of the storage unit, and store the SRRL data in the second part of the storage space of the storage unit corresponding to the address of the storage unit in the SRAM memory.

[0021] According to a fifth aspect of the present application, an RDMA system is provided, comprising the requesting end described in the third aspect and the responding end described in the fourth aspect.

[0022] Other features and advantages of the present application will be set forth in the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 This is a schematic diagram of an RDMA read operation method disclosed in the background technology.

[0025] Figure 2 This is a flowchart of an RDMA read operation method in Example 1 of the present application.

[0026] Figure 3 This is an example diagram of an RDMA system in Example 1 of the present application.

[0027] Figure 4 This is a flowchart of an RDMA read operation method in the second embodiment of the present application. DETAILED DESCRIPTION

[0028] The detailed description of the accompanying drawings is intended to serve as an illustration of the presently preferred embodiment of the present application, and is not intended to represent the only form in which the present application can be implemented. It should be understood that the same or equivalent functions can be accomplished by different embodiments intended to be included within the scope of the present application.

[0029] See Figure 2 , Embodiment 1 of the present application provides an RDMA read operation method, the method comprising: Step S11: The receiving processor of the requesting end receives the SQ WQE sent by the host, caches the SQ WQE in a cache, obtains a CRRL tag and CRRL data according to the SQ WQE, and sends a CRRL creation command to the storage control module of the requesting end; the CRRL creation command includes the CRRL tag and CRRL data; Specifically, the send processor (TXP) of the requester receives the SQWQE (Send Queue Work Completion Unit) driven by the host, corresponding to Figure 3 Step ① in the above description; the sending processor of the requesting end parses the SQ WQE to obtain the CRRL tag and CRRL data, where the CRRL tag can be understood as the identity of the CRRL data and is used to distinguish different CRRL data; in this embodiment, the requesting end is provided with a dedicated storage module for storing CRRL data, which is independent of the cache of the requesting end, and the SQ WQE caches the CRRL data in the cache of the requesting end for subsequent use.

[0030] Step S12: The storage control module of the requesting end stores the CRRL data in an idle storage unit in the SRAM memory according to the CRRL creation command, and stores the address of the storage unit and the CRRL tag in the TCAM memory; Specifically, this embodiment sets up an SRAM memory and a TCAM memory (Ternary Content Addressable Memory). The use of TCAM can quickly realize the query of the storage unit address; after the storage control module of the request end receives the CRRL creation command, it queries the current status of each storage unit of the SRAM memory, selects a storage unit that is currently idle, stores the CRRL data in the idle storage unit, and stores the address of this storage unit and the corresponding CRRL tag in the TCAM memory, which is used to quickly find the address of the storage unit where the CRRL data is located according to the CRRL tag.

[0031] Step S13: The sending processor of the requesting end generates a read request according to the SQ WQE, and sends the read request to the responding end, so that the responding end returns read response data according to the read request; Specifically, the sending processor of the requesting end generates a standard RDMA read request according to SQ WQE (SQ WQE) and sends it to the responding end for processing. Figure 3In steps ② and ③, after receiving the RDMA read request, the responder executes the method steps of embodiment 2 disclosed below and returns at least one read response data corresponding to the read request.

[0032] Step S14: the receiving processor of the requesting end receives the read response data returned by the responding end, and sends a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag; Specifically, the receiving processor (RXP) of the requesting end receives the read response data (read response) from the responding end (responder), corresponding to Figure 3 In step ⑦, the receiving processor of the requesting end needs to process the read response data according to the CRRL data (for example, where the read response data should be stored). The receiving processor of the requesting end sends a CRRL query command to the storage control module of the requesting end. The CRRL query command contains the CRRL tag corresponding to the CRRL data, and obtains the CRRL data returned by the storage control module of the requesting end. Figure 3 Step ⑧ in the Figure 3 The storage module in the system includes a storage control module, an SRAM memory and a TCAM memory.

[0033] Step S15: The storage control module of the requesting end responds to the CRRL query command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, reads the CRRL data from the SRAM memory according to the address of the storage unit, and returns the CRRL data to the receiving processor of the requesting end; Specifically, the storage control module of the requesting end uses the CRRL tag to search in the TCAM memory, obtains the storage unit address of the SRAM memory where the CRRL data is located, reads the previously stored CRRL data from the SRAM memory according to the storage unit address, and returns the read CRRL data to the receiving processor of the requesting end.

[0034] Step S16: The receiving processor of the requesting end processes the read response data returned by the responding end according to the CRRL data, generates an SQ CQE according to the SQ WQE stored in the cache, and sends the SQ CQE to the host.

[0035] Specifically, after receiving the CRRL data returned from the storage control module, the receiving processor of the requesting end correctly processes the received read response data according to the CRRL data (for example, copies the data to the specified memory address), and finally generates an SQ CQE (send queue work completion unit) to notify the host driver that the SQ WQE has been completed, corresponding to Figure 3 Step 9 in .

[0036] The RDMA read operation method of the first embodiment of the present invention separates the CRRL data from the requesting-side cache and stores it separately in the SRAM memory. The TCAM memory is used to store the CRRL tag and storage unit address corresponding to the CRRL data. By changing the data storage method, the dependence on the host memory (DDR) and the requesting-side cache is reduced. The types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the requesting-side cache are reduced, unnecessary PCIE read and write operations are reduced, and valuable PCIE bandwidth and power consumption are saved.

[0037] In some embodiments, the CRRL tag includes a queue pair function identifier (function_id), a queue pair index (qpn, queue pair number), a data type (type) and a CRRL index (entry_idx), and the data type of the CRRL tag is CRRL.

[0038] Specifically, qpn is used to uniquely identify a queue pair (QP). A QP can support multiple RDMA operations (read, write, etc.); function_id is used to distinguish which virtual machine the QP belongs to; type is used to distinguish CRRL data from SRRL data. In RDMA communication, the RDMA device acts as both a receiver and a responder to communicate with other devices. That is, when it acts as a responder, it involves the processing of SRRL data, such as Figure 3 As shown, the storage modules of the receiving end and the responding end both include storage for CRRL data and SRRL data. Therefore, in this embodiment, the storage module distinguishes CRRL data and SRRL data according to type and performs corresponding storage operations; each CRRL data has a serial number (entry_idx), and based on function_id, qpn, type and entry_idx, a CRRL data can be uniquely identified.

[0039] like Figure 3As shown in the figure, assuming a 512-bit CRRL data width and a 256-bit SRRL data width, a 512-bit storage unit can hold one CRRL data item or two SRRL data items. Two adjacent SRRL data items with the same function and QP are placed in the same storage unit. The number of required CRRL and SRRL entries is calculated based on the packet rate of a typical scenario. For example, if the RTT (Round-Trip Time) is 10us and the RDMA read request packet rate (pps) is 100M, the required CRRL entries are 10us * 100M = 1000, requiring 1000 CRRL entries. After adding a certain design margin, the required number of CRRL entries can be set at 1200. The required number of SRRL entries can be calculated similarly. Note that two SRRL data items occupy one storage unit, but not every storage unit can store two valid SRRL data items, so a larger design margin is required.

[0040] In some embodiments, the method of embodiment 1 further includes: Step S17, when the receiving processor of the requesting end completes processing of the last read response data corresponding to the read request, it sends a CRRL deletion command to the storage control module of the requesting end; the CRRL deletion command includes the CRRL tag; Specifically, when the receiving processor (Requester RXP) on the requesting side completes processing the last read response data corresponding to the read request, the entire RDMA read operation is completed and the CRRL data is no longer needed to associate or process subsequent data packets or status. At this time, the receiving processor on the requesting side generates a CRRL delete command and sends the delete command containing the CRRL tag to the storage control module on the requesting side.

[0041] In step S18, the storage control module of the requesting end responds to the CRRL deletion command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the CRRL tag from the TCAM memory.

[0042] Specifically, the storage control module on the requesting end uses the CRRL tag (including functionid, qpn, type=CRRL, entry_idx) to query the TCAM memory (TCAM) according to the CRRL deletion command to find the address of the CRRL data stored in the SRAM memory; according to the queried address, the corresponding CRRL data item is deleted (cleared or marked as invalid) from the SRAM memory, which frees up the SRAM memory address and can be reused by subsequent RDMA operations; at the same time, the storage control module also deletes the entry corresponding to the CRRL tag in the TCAM (TCAM memory) (that is, the address of the storage unit and the CRRL tag itself), which frees up the storage space of the TCAM, allowing the search space of the TCAM to be restored, improving the availability and query efficiency of the TCAM, and preventing the TCAM from affecting performance or running out of space due to too many entries.

[0043] In some embodiments, although the number of required CRRL entries and SRRL entries is calculated based on typical scenarios and the design margin is increased, insufficient CRRL entry and SRRL entry resources may still occur. To address this issue, the method of embodiment 1 further includes: Step S19, the storage control module of the requesting end responds to the CRRL creation command, and when there is no free storage unit in the SRAM memory, returns a storage unit full signal to the receiving processor of the requesting end; Specifically, when the system load is high or the SRAM storage resources are very limited, there may be no available free storage units. In this case, the storage control module cannot complete the creation of CRRL data (because the storage space cannot be allocated), so it will return a specific signal to the receiving processor on the requesting end - "storage unit full signal".

[0044] Step S110 : The receiving processor of the requesting end suspends sending the read request to the responding end according to the storage unit full signal.

[0045] Specifically, if the receiving processor on the requesting side receives a "storage unit full signal" from the storage control module, this indicates that the system cannot currently process more RDMA read requests because the storage unit resources used to store CRRL data have been exhausted. The receiving processor on the requesting side pauses sending subsequent read requests to the responding side, and new RDMA read WQEs issued by the driver will not be processed (or the processing will be blocked) until CRRL resources are released in the system (for example, the CRRL deletion mechanism mentioned above).

[0046] See Figure 4, Embodiment 2 of the present application provides an RDMA read operation method, the method comprising: Step S21: The receiving processor of the responding end receives the read request sent by the requesting end, obtains the SRRL tag and SRRL data according to the read request, and sends an SRRL creation command to the storage control module of the responding end; the SRRL creation command includes the SRRL tag and SRRL data; Specifically, the responder RXP receives the RDMA read request sent by the requester, corresponding to Figure 3 Step ③ in the response end; the receiving processor obtains the SRRL tag and SRRL data according to the received read request, corresponding to Figure 3 In step ④, the receiving processor of the responding end sends an SRRL creation command to the storage control module of the responding end, where the SRRL creation command includes an SRRL tag and SRRL data; Figure 3 The storage module in the system includes a storage control module, an SRAM memory and a TCAM memory.

[0047] Step S22: The storage control module of the responding end responds to the SRRL creation command, stores the SRRL data in an idle storage unit in the SRAM memory, and stores the address of the storage unit and the SRRL tag in the TCAM memory; Specifically, after receiving the SRRL creation command, the storage control module on the responding end queries the current status of each storage unit of the SRAM memory, selects a storage unit that is currently idle, stores the SRRL data in the idle storage unit, and stores the address of this storage unit and the corresponding SRRL label in the TCAM memory, which is used to quickly find the address of the storage unit where the SRRL data is located according to the SRRL label.

[0048] Step S23, the receiving processor of the responding end sends a doorbell to the sending processor of the responding end to request to send the read response data; Specifically, after creating the SRRL, the responder's receiving processor sends a receive queue request (RQ DB, Receive queue doorbell) to the responder's sending processor (Responder TXP), corresponding to Figure 3 Step ⑤ in the above is used to notify the sending processor of the responding end that new read response data needs to be sent.

[0049] Step S24: the sending processor of the responding end responds to the doorbell and sends an SRRL query command to the storage control module of the responding end; the SRRL query command includes the SRRL tag; Specifically, the receiving processor (RXP) of the responding end receives the data from the RQ DB and Figure 3 In step ⑦, the receiving processor of the responding end needs to generate and send read response data according to the SRRL data. The receiving processor of the responding end sends an SRRL query command to the storage control module of the responding end. The SRRL query command contains the SRRL tag corresponding to the SRRL data, and obtains the SRRL data returned by the storage control module of the responding end. Figure 3 Step ⑥ in .

[0050] Step S25: The storage control module of the responding end responds to the SRRL query command, queries the TCAM memory according to the SRRL tag to obtain the address of the storage unit, reads the SRRL data from the SRAM memory according to the address of the storage unit, and returns the SRRL data to the sending processor of the responding end; Specifically, this embodiment sets up an SRAM memory and a TCAM memory (Ternary Content Addressable Memory). The use of TCAM can quickly implement the query of the storage unit address; the storage control module of the responding end uses the SRRL tag to search in the TCAM memory, obtains the storage unit address of the SRAM memory where the SRRL data is located, reads the previously stored SRRL data from the SRAM memory according to the storage unit address, and returns the read SRRL data to the receiving processor of the responding end.

[0051] Step S26, the sending processor of the responding end obtains read response data according to the SRRL data, and sends the read response data to the requesting end; Specifically, the receiving processor of the responding end prepares read response data according to the information in the SRRL data, and returns at least one read response data to the requesting end, and the RDMA read operation processing of the responding end is completed.

[0052] The RDMA read operation method of the second embodiment of the present invention separates the SRRL data from the requesting end's cache and stores it separately in the SRAM memory. At the same time, the SRRL tag and storage unit address corresponding to the SRRL data are stored in the TCAM memory. By changing the data storage method, the dependence on the host memory (DDR) and the responding end's cache is reduced. The types of resources managed in the cache are reduced, the management logic can be simplified, the capacity and complexity of the responding end's cache are reduced, unnecessary PCIE read and write operations are reduced, and valuable PCIE bandwidth and power consumption are saved.

[0053] In some embodiments, the SRRL tag includes a queue pair function identifier, a queue pair index, and an SRRL index.

[0054] Specifically, qpn is used to uniquely identify a queue pair (QP). A QP can support multiple RDMA operations (read, write, etc.); function_id is used to distinguish the virtual machine to which the QP belongs; type is used to distinguish CRRL data from SRRL data. In RDMA communication, the RDMA device acts as both the receiver and responder to communicate with other RDMA devices. That is, when it acts as the responder, it involves the processing of SRRL data, such as Figure 3 As shown, the storage modules of the receiving end and the responding end both include storage for CRRL data and SRRL data. Therefore, in this embodiment, the storage control module distinguishes CRRL data and SRRL data according to type and performs corresponding storage operations; each SRRL data has a serial number (entry_idx), and based on function_id, qpn, type and entry_idx, an SRRL data can be uniquely identified.

[0055] In some embodiments, storing the SRRL data in an idle memory cell in an SRAM memory, and storing the address of the memory cell and the SRRL tag in a TCAM memory, comprises: The SRRL index is represented in binary format; When the lowest bit of the SRRL index is 0, storing the SRRL data in a first portion of storage space of an idle storage unit in an SRAM memory, and storing the address of the storage unit and the SRRL tag in a TCAM memory; When the lowest bit of the SRRL index is 1, the TCAM memory is queried according to the SRRL tag to obtain the address of the storage unit, and the SRRL data is stored in the second part of the storage space of the storage unit corresponding to the address of the storage unit in the SRAM memory.

[0056] Specifically, assuming the bit width of CRRL data is 512 bits and the bit width of SRRL data is 256 bits, a storage unit is 512 bits in size and can hold one CRRL data item or two SRRL data items. Two adjacent SRRL data items with the same function and the same QP are placed in the same storage unit. In this embodiment, the SRRL index is represented in binary to determine the location of the SRRL data in the storage unit based on the lowest bit (0 / 1) of the sequence number.

[0057] In some embodiments, the method further comprises: Step S27: When the sending processor of the responding end completes sending the last read response data corresponding to the read request, and the lowest bit of the SRRL index of the last read response data is 1, sending an SRRL delete command to the storage control module of the responding end; the SRRL delete command includes the SRRL tag; Specifically, when the receiving processor (Responder RXP) on the responding end completes processing of the last read response data corresponding to the read request, and the lowest bit of the SRRL index of the last read response data is 1, the entire RDMA read operation corresponding to a storage unit has been completed, and the SRRL data is no longer needed to associate or process subsequent data packets or status. At this time, the receiving processor on the responding end generates an SRRL delete command and sends the delete command containing the SRRL tag to the storage control module on the responding end.

[0058] In step S28, the storage control module of the responding end responds to the SRRL deletion command, queries the TCAM memory according to the SRRL label to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the SRRL label from the TCAM memory.

[0059] Specifically, the storage control module (storage module) on the responding end uses the SRRL tag (including functionid, qpn, type=SRRL, entry idx) to query the TCAM memory (TCAM) according to the SRRL delete command to find the storage unit address storing the SRRL data in the SRAM memory; based on the queried address, the corresponding SRRL data item is deleted (cleared or marked as invalid) from the SRAM memory, thereby freeing up the SRAM storage unit resources / address, which can be reused by subsequent RDMA operations; at the same time, the storage control module also deletes the entry corresponding to the SRRL tag in the TCAM (TCAM memory) (i.e., the address of the storage unit and the SRRL tag itself), thereby freeing up the storage space of the TCAM, allowing the search space of the TCAM to be restored, improving the availability and query efficiency of the TCAM, and preventing the TCAM from affecting performance or exhausting space due to too many entries.

[0060] In some embodiments, the method further comprises: Step S29, the storage control module of the responding end responds to the SRRL creation command, and when there is no free storage unit in the SRAM memory, returns a storage unit full signal to the sending processor of the responding end; Specifically, when the storage control module on the responder side receives the SRRL creation command, it tries to find a free storage unit in the SRAM memory to store the new SRRL data. If it finds no free unit (resource exhaustion), the storage control module on the responder side will generate a storage unit full signal. This signal is returned to the responder receiving processor and is sent to the responder sending processor (Responder TXP) along with the doorbell.

[0061] In step S210 , the sending processor of the responding end sends a negative acknowledgement signal to the requesting end according to the storage unit full signal, so as to notify the requesting end to wait for a preset time before sending the read request.

[0062] Specifically, after receiving the "storage unit is full" signal, the responder's sending processor knows it cannot process new read requests or prepare corresponding read response data. It needs to notify the requester of this situation to prevent further read requests from the requester, which could lead to data loss or system disruption. The responder's sending processor then sends a negative acknowledgement signal (RNR NAK) to the requester, instructing the requester to suspend sending read requests. Upon receiving the RNR NAK signal, the requester suspends sending new RDMA read requests to the responder for a preset period of time. After this period, the requester can attempt to resend the read request.

[0063] The third embodiment of the present application proposes a requester for executing the steps of the RDMA read operation method described in the first embodiment, such as Figure 3 As shown, the requester includes a transmit processor (Requester TXP), a storage control module, an SRAM memory, a TCAM memory, and a transmit processor (Requester RXP); The receiving processor of the requesting end is configured to receive the SQ WQE sent by the host, cache the SQ WQE in a cache, obtain a CRRL tag and CRRL data according to the SQ WQE, and send a CRRL creation command to the storage control module of the requesting end; the CRRL creation command includes the CRRL tag and CRRL data; The storage control module of the requesting end is used to store the CRRL data in an idle storage unit in the SRAM memory according to the CRRL creation command, and store the address of the storage unit and the CRRL tag in the TCAM memory; The sending processor of the requesting end is used to generate a read request according to the SQ WQE, and send the read request to the responding end, so that the responding end returns read response data according to the read request; The receiving processor of the requesting end is further configured to receive the read response data returned by the responding end, and send a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag; The storage control module of the requesting end is further configured to respond to the CRRL query command, query the TCAM memory according to the CRRL tag to obtain the address of the storage unit, read the CRRL data from the SRAM memory according to the address of the storage unit, and return the CRRL data to the receiving processor of the requesting end; The receiving processor of the requesting end is further configured to process the read response data returned by the responding end according to the CRRL data, generate an SQ CQE according to the SQ WQE stored in the cache, and send the SQ CQE to the host.

[0064] In some embodiments, the CRRL tag includes a queue pair function identifier, a queue pair index, and a CRRL index.

[0065] The fourth embodiment of the present application proposes a responder for executing the steps of the RDMA read operation method described in the second embodiment, such as Figure 3 As shown, the responder includes a transmit processor (Responder TXP), a storage control module, a TCAM memory, an SRAM memory, and a transmit processor (Responder RXP); The receiving processor of the responding end is used to receive the read request sent by the requesting end, obtain the SRRL tag and SRRL data according to the read request, and send an SRRL creation command to the storage control module of the responding end; the SRRL creation command includes the SRRL tag and SRRL data; The storage control module of the responding end is used to respond to the SRRL creation command, store the SRRL data in an idle storage unit in the SRAM memory, and store the address of the storage unit and the SRRL tag in the TCAM memory; The receiving processor of the responding end is used to send a doorbell to the sending processor of the responding end to request to send the read response data; The sending processor of the responding end is used to respond to the doorbell and send an SRRL query command to the storage control module of the responding end; the SRRL query command includes the SRRL tag; The storage control module of the responding end is used to respond to the SRRL query command, query the TCAM memory according to the SRRL label to obtain the address of the storage unit, read the SRRL data from the SRAM memory according to the address of the storage unit, and return the SRRL data to the sending processor of the responding end; The sending processor of the responding end is used to obtain read response data according to the SRRL data, and send the read response data to the requesting end.

[0066] In some embodiments, the SRRL tag includes a queue pair function identifier, a queue pair index, and an SRRL index.

[0067] In some embodiments, the SRRL index is represented in binary format; The storage control module of the responding end is used to store the SRRL data in the first part of the storage space of the idle storage unit in the SRAM memory when the lowest bit of the SRRL index is 0, and store the address of the storage unit and the SRRL label in the TCAM memory; when the lowest bit of the SRRL index is 1, query the TCAM memory according to the SRRL label to obtain the address of the storage unit, and store the SRRL data in the second part of the storage space of the storage unit corresponding to the address of the storage unit in the SRAM memory.

[0068] The fifth embodiment of the present application proposes an RDMA system, such as Figure 3 As shown, it includes the request end described in the above embodiment three and the response end described in the above embodiment four.

[0069] While various embodiments of the present application have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many updates and modifications will be readily apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An RDMA read operation method, characterized in that: The method comprises: The receiving processor of the requesting end receives the SQ WQE sent by the host, caches the SQ WQE in a cache, obtains a CRRL tag and CRRL data according to the SQ WQE, and sends a CRRL creation command to the storage control module of the requesting end; the CRRL creation command includes the CRRL tag and CRRL data; The storage control module of the requesting end stores the CRRL data in an idle storage unit in the SRAM memory according to the CRRL creation command, and stores the address of the storage unit and the CRRL tag in the TCAM memory; The sending processor of the requesting end generates a read request according to the SQ WQE, and sends the read request to the responding end, so that the responding end returns read response data according to the read request; The receiving processor of the requesting end receives the read response data returned by the responding end, and sends a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag; The storage control module of the requesting end responds to the CRRL query command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, reads the CRRL data from the SRAM memory according to the address of the storage unit, and returns the CRRL data to the receiving processor of the requesting end; The receiving processor of the requesting end processes the read response data returned by the responding end according to the CRRL data, generates an SQ CQE according to the SQ WQE stored in the cache, and sends the SQ CQE to the host.

2. The method according to claim 1, characterized in that The CRRL tag includes a queue pair function identifier, a queue pair index, and a CRRL index.

3. The method according to claim 2, characterized in that The method further comprises: When the receiving processor of the requesting end completes processing of the last read response data corresponding to the read request, sending a CRRL delete command to the storage control module of the requesting end; the CRRL delete command includes the CRRL tag; The storage control module of the requesting end responds to the CRRL deletion command, queries the TCAM memory according to the CRRL tag to obtain the address of the storage unit, marks the address of the storage unit as free, and deletes the address of the storage unit and the CRRL tag from the TCAM memory.

4. The method according to claim 1, wherein The method further comprises: The storage control module of the requesting end responds to the CRRL creation command and returns a storage unit full signal to the receiving processor of the requesting end when there is no free storage unit in the SRAM memory; The receiving processor of the requesting end suspends sending the read request to the responding end according to the storage unit full signal.

5. An RDMA read operation method, characterized in that: The method comprises: The receiving processor of the responding end receives the read request sent by the requesting end, obtains the SRRL tag and SRRL data according to the read request, and sends an SRRL creation command to the storage control module of the responding end; the SRRL creation command includes the SRRL tag and SRRL data; The storage control module of the responding end responds to the SRRL creation command, stores the SRRL data in an idle storage unit in the SRAM memory, and stores the address of the storage unit and the SRRL tag in the TCAM memory; The receiving processor of the responding end sends a doorbell to the sending processor of the responding end to request to send the read response data; The sending processor of the responding end responds to the doorbell and sends an SRRL query command to the storage control module of the responding end; the SRRL query command includes the SRRL tag; The storage control module of the responding end responds to the SRRL query command, queries the TCAM memory according to the SRRL tag to obtain the address of the storage unit, reads the SRRL data from the SRAM memory according to the address of the storage unit, and returns the SRRL data to the sending processor of the responding end; The sending processor of the responding end obtains the read response data according to the SRRL data, and sends the read response data to the requesting end.

6. The method according to claim 5, characterized in that The SRRL tag includes a queue pair function identifier, a queue pair index, and an SRRL index.

7. The method according to claim 6, characterized in that The step of storing the SRRL data in an idle storage unit in an SRAM memory, and storing the address of the storage unit and the SRRL tag in a TCAM memory includes: The SRRL index is represented in binary format; When the lowest bit of the SRRL index is 0, storing the SRRL data in a first portion of storage space of an idle storage unit in an SRAM memory, and storing the address of the storage unit and the SRRL tag in a TCAM memory; When the lowest bit of the SRRL index is 1, the TCAM memory is queried according to the SRRL tag to obtain the address of the storage unit, and the SRRL data is stored in the second part of the storage space of the storage unit corresponding to the address of the storage unit in the SRAM memory.

8. The method according to claim 5, characterized in that The method further comprises: When the sending processor of the responding end completes sending the last read response data corresponding to the read request, and the lowest bit of the SRRL index of the last read response data is 1, sending an SRRL delete command to the storage control module of the responding end; the SRRL delete command includes the SRRL tag; The storage control module on the responding end responds to the SRRL deletion command, queries the TCAM memory according to the SRRL label to obtain the address of the storage unit, marks the address of the storage unit as idle, and deletes the address of the storage unit and the SRRL label from the TCAM memory.

9. The method according to claim 5, characterized in that The method further comprises: The storage control module of the responding end responds to the SRRL creation command, and when there is no free storage unit in the SRAM memory, returns a storage unit full signal to the sending processor of the responding end; The sending processor of the responding end sends a negative acknowledgement signal to the requesting end according to the storage unit full signal, so as to notify the requesting end to wait for a preset time before sending the read request.

10. A request end for implementing an RDMA read operation, characterized in that: Includes a sending processor, a storage control module, a receiving processor, a cache, an SRAM memory, and a TCAM memory; The receiving processor of the requesting end is configured to receive the SQ WQE sent by the host, cache the SQ WQE in a cache, obtain a CRRL tag and CRRL data according to the SQ WQE, and send a CRRL creation command to the storage control module of the requesting end; the CRRL creation command includes the CRRL tag and CRRL data; The storage control module of the requesting end is used to store the CRRL data in an idle storage unit in the SRAM memory according to the CRRL creation command, and store the address of the storage unit and the CRRL tag in the TCAM memory; The sending processor of the requesting end is used to generate a read request according to the SQ WQE, and send the read request to the responding end, so that the responding end returns read response data according to the read request; The receiving processor of the requesting end is further configured to receive the read response data returned by the responding end, and send a CRRL query command to the storage control module of the requesting end; the CRRL query command includes the CRRL tag; The storage control module of the requesting end is further configured to respond to the CRRL query command, query the TCAM memory according to the CRRL tag to obtain the address of the storage unit, read the CRRL data from the SRAM memory according to the address of the storage unit, and return the CRRL data to the receiving processor of the requesting end; The receiving processor of the requesting end is further configured to process the read response data returned by the responding end according to the CRRL data, generate an SQ CQE according to the SQ WQE stored in the cache, and send the SQ CQE to the host.

11. The requesting end according to claim 10, characterized in that: The CRRL tag includes a queue pair function identifier, a queue pair index, and a CRRL index.

12. A responder for implementing an RDMA read operation, characterized in that: It includes a sending processor, a storage control module, a TCAM memory, an SRAM memory and a receiving processor; The receiving processor of the responding end is used to receive the read request sent by the requesting end, obtain the SRRL tag and SRRL data according to the read request, and send an SRRL creation command to the storage control module of the responding end; the SRRL creation command includes the SRRL tag and SRRL data; The storage control module of the responding end is used to respond to the SRRL creation command, store the SRRL data in an idle storage unit in the SRAM memory, and store the address of the storage unit and the SRRL tag in the TCAM memory; The receiving processor of the responding end is used to send a doorbell to the sending processor of the responding end to request to send the read response data; The sending processor of the responding end is used to respond to the doorbell and send an SRRL query command to the storage control module of the responding end; the SRRL query command includes the SRRL tag; The storage control module of the responding end is used to respond to the SRRL query command, query the TCAM memory according to the SRRL label to obtain the address of the storage unit, read the SRRL data from the SRAM memory according to the address of the storage unit, and return the SRRL data to the sending processor of the responding end; The sending processor of the responding end is used to obtain read response data according to the SRRL data, and send the read response data to the requesting end.

13. The responding end according to claim 12, characterized in that: The SRRL tag includes a queue pair function identifier, a queue pair index, and an SRRL index.

14. The responding end according to claim 13, characterized in that: The SRRL index is represented in binary format; The storage control module of the responding end is used to store the SRRL data in the first part of the storage space of the idle storage unit in the SRAM memory when the lowest bit of the SRRL index is 0, and store the address of the storage unit and the SRRL label in the TCAM memory; when the lowest bit of the SRRL index is 1, query the TCAM memory according to the SRRL label to obtain the address of the storage unit, and store the SRRL data in the second part of the storage space of the storage unit corresponding to the address of the storage unit in the SRAM memory.

15. An RDMA system, characterized in that: The method comprises the request end according to claim 10 or 11, and the response end according to any one of claims 12 to 14.

Images