Method, device and system for realizing random address PCIE DMA read and write

Abstract

The application provides an arbitrary address PCIE DMA read-write implementation method, device and system, and the method comprises the following steps: a FPGA encapsulates a plurality of TLP read request packets in DW units according to a start address and length information, and sends the plurality of TLP read request packets to a PCIE IP CORE, and records effective information; when a TLP read completion packet is received by a data receiving module, the data of the TLP read completion packet is cached into a FIFO first, then the received data is read out from the cache FIFO according to the cached effective information, the received data is adjusted, and finally, the complete and continuous effective data is sent to a downstream module, and the DMA read operation is completed. The application can realize obtaining memory data in an arbitrary address or writing data to a memory in an arbitrary address.

Description

Technical Field

[0001] This invention relates to the field of FPGA technology, and in particular to a method, device and system for implementing arbitrary address PCIe DMA read and write. Background Technology

[0002] With the increasingly widespread application of heterogeneous acceleration, the development of FPGA-based accelerator cards is also accelerating rapidly. FPGA accelerator cards connect to the server host through the PCIe interface and use DMA read operations to move the data that needs to be accelerated from the server host memory to the FPGA accelerator card. After the FPGA accelerator card finishes processing, it moves the data back to memory through DMA write operations. During the data interaction process, the CPU is rarely involved and does not need to copy data, which reduces the CPU's utilization and improves the efficiency of data transmission.

[0003] However, in some application scenarios, it is required that the memory where the server host stores data starts from an arbitrary address, such as (FPGA-based) network card devices. However, the data length and address of the PCIe interface are DW aligned, which makes it difficult for (FPGA-based) network card devices to obtain data from or write data to arbitrary memory addresses on the host. Summary of the Invention

[0004] To address the aforementioned issues, it is necessary to provide a method, device, and system for arbitrary address PCIe DMA read / write, capable of acquiring data from or writing data to any address of the host memory.

[0005] The first aspect of this invention provides a method for implementing arbitrary address PCIe DMA read / write, comprising the following steps:

[0006] Step 1: Write the starting address of memory space I and the length of the data to be processed into the status word cmd_fifo of the FPGA;

[0007] The starting address of memory space I and the length of the data to be processed are both sent to the FPGA by the host through PIO mode. Memory space I is a continuous memory space requested by the host to store the data to be processed.

[0008] Step 2: When the FPGA's DMA read control module finds that the status word cmd_fifo is not empty, it calculates the byte enable signal based on the starting address and the length of the data to be processed, and encapsulates N TLP read request packets in DW units according to preset rules, and caches the starting address and the length of the data to be processed for this DMA read operation in the status word DMA_FIFO.

[0009] Each TLP read request packet includes a byte enable signal, the requested data size, the request start address, and a read request tag;

[0010] Step 3: The FPGA's data transmission module I sends the TLP read request packet to the PCIe IP CORE via the PCIe root union RC to access the host memory;

[0011] Step 4: When all TLP read request packets for this DMA read operation have been sent, the FPGA DMA read control module will sequentially cache the request data size and read request tag in each sent TLP read request packet into the status word TAG_FIFO.

[0012] The size of the request data in the TLP read request packet is set to correspond one-to-one with the read request tag;

[0013] Step 5: When the FPGA data receiving module receives the TLP read complete packet, it parses out the load data, DW valid signal and read complete tag of the TLP read complete packet, and sends the DW valid signal and load data of the read complete packet to the corresponding write FIFO control module according to the read complete tag.

[0014] The read completion tag corresponds one-to-one with the read request tag in step 4;

[0015] Step 6: The write FIFO control module continuously writes the load data into the first data buffer FIFO in the backend in DW units; wherein, the number of write FIFO control modules, the number of first data buffer FIFOs, and the number of read request tags are equal, and the first data buffer FIFO adopts a 32-bit width.

[0016] When the FPGA's data receiving module detects that the TLP read completion packet corresponding to a certain read request tag has been received, it generates a corresponding receive completion identifier and sends it to the FPGA's read FIFO control module; wherein, the receive completion identifier is used to indicate that all data of a certain read request has been received.

[0017] Step 7: After receiving the receive completion identifiers corresponding to all read request tags, the read FIFO control module reads the load data in the corresponding first data buffer FIFO in a loop according to the order of the obtained read completion tags, and sorts the read load data according to the obtained starting address and the length of the data to be processed, and deletes redundant data in the first DW and / or the last DW.

[0018] A second aspect of the present invention provides an FPGA device, the FPGA device comprising a status word cmd_fifo, a DMA read control module, a data transmission module I, a status word DMA_FIFO, a status word TAG_FIFO, a data receiving module, a write FIFO control module, a first data buffer FIFO, and a read FIFO control module, wherein,

[0019] The status word cmd_fifo is used to store the starting address of memory space I and the length of the data to be processed; wherein, the starting address of memory space I and the length of the data to be processed are both sent to the FPGA by the host through PIO mode, and memory space I is a contiguous memory space allocated by the host to store the data to be processed;

[0020] The DMA read control module is used to calculate a byte enable signal based on the starting address and the length of the data to be processed when the status word cmd_fifo is not empty, and encapsulate N TLP read request packets in DW units according to preset rules, and cache the starting address and the length of the data to be processed in the status word DMA_FIFO; it is also used to cache the request data size and read request tag in each TLP read request packet in the status word TAG_FIFO when all TLP read request packets of the current DMA read operation have been sent.

[0021] Each TLP read request packet includes a byte enable signal, the requested data size, the request start address, and a read request tag;

[0022] The data sending module I is used to send the TLP read request packet to the PCIe IP CORE via the PCIe Root Consortium RC to access the host memory;

[0023] The data receiving module is configured to, upon receiving a TLP read completion packet, parse out the payload data, DW valid signal, and read completion tag of the TLP read completion packet, and send the DW valid signal and payload data of the read completion packet to the corresponding write FIFO control module according to the read completion tag; it is also configured to, upon detecting that the TLP read completion packet corresponding to a certain read completion tag has been received, generate a corresponding reception completion identifier and send it to the read FIFO control module; wherein, the read completion tag corresponds one-to-one with the read request tag;

[0024] The write FIFO control module is used to continuously write the load data into the first data buffer FIFO in the backend in units of DW.

[0025] The read FIFO control module is used to read the load data in the corresponding first data buffer FIFO in a loop according to the order of the read completion tags after receiving the receive completion identifiers corresponding to all read completion tags. The module also sorts the read load data according to the starting address and the length of the data to be processed in the status word DMA_FIFO, and deletes redundant data in the first DW and / or the last DW.

[0026] A third aspect of the present invention provides an FPGA-based communication system, the communication system comprising a host and the aforementioned FPGA device, wherein,

[0027] The host is used to monitor whether there is data to be processed. If so, it requests a contiguous memory space to store the data to be processed and sends the starting address of the contiguous memory space and the length of the data to be processed to the FPGA device through PIO mode. It is also used to receive TLP read request packets from the FPGA device and process them to generate corresponding TLP read completion packets and return them to the FPGA device. It is also used to receive TLP write request packets from the FPGA device.

[0028] The beneficial effects of this invention are as follows:

[0029] 1) This invention enables FPGA devices to obtain data from any address of the host memory without any special requirements on the memory address; it enables FPGA devices to write data of any length to any starting address of the host memory without any special requirements on the memory address, reducing CPU copying of memory and saving CPU resources.

[0030] 2. The FPGA does not need to adjust the data to be sent in the second data buffer FIFO to adapt to DMA write operations, thus achieving independence in the engineering structure;

[0031] 3) Based on the actual number of read request tags used, the same number of write FIFO control modules and data cache FIFOs are used, achieving design flexibility;

[0032] 4) A large-width buffer space is formed by using a write FIFO control module and multiple small-width first data buffer FIFOs to buffer the received data; a 32-bit-width FIFO is used as the data buffer FIFO, and the write FIFO control module can handle the impact of RCB.

[0033] 5) When initiating a DMA read operation, the address and length information are cached. When reading the data from the FIFO cache, redundant data is deleted and valid data is rearranged based on the information cached when initiating the DMA read operation, thus ensuring the reliability of transmission.

[0034] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0035] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0036] Figure 1 A block diagram of the DMA read structure of the FPGA device of the present invention is shown;

[0037] Figure 2 A block diagram of the DMA write structure of the FPGA device of the present invention is shown;

[0038] Figure 3 The DMA write TLP data structure of the FPGA device of the present invention is shown;

[0039] Figure 4 The timing diagram of the arbitrary address DMA read implementation method of the present invention is shown;

[0040] Figure 5 The timing diagram of the arbitrary address DMA write implementation method of the present invention is shown;

[0041] Figure 6 A flowchart illustrating the arbitrary address DMA read implementation method of the present invention is shown. Detailed Implementation

[0042] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0044] PCIe (Peripheral Component Interconnect Express) is the third generation of I / O bus after ISA and PCI buses. PCIe has two data transfer methods:

[0045] (1) DMA (Direct Memory Access) mode: In this mode, data transfer is not handled by the CPU, but by a DMA controller, which consumes less CPU resources;

[0046] (2) PIO (Programmed Input-Output) mode: Programmable I / O is a data exchange mode in which the CPU executes I / O port instructions to read and write data. In this mode, data transfer is handled by the CPU executing I / O port instructions, which consumes a lot of CPU resources and the data transfer speed is much lower than that of DMA mode.

[0047] The RC (PCI Express root complex) is equivalent to the PCIe main bridge and is also known as the PCIe bus controller.

[0048] RCB (Read Completion Boundary) means that when the receiver returns read response data to the sender, the size of multiple TLP packets must be a multiple of the RCB, that is, they must be aligned to the RCB address boundary.

[0049] FIFO (First In First Out) is a first-in-first-out data buffer.

[0050] DW (Double Word), 1DW=4Byte=32bit.

[0051] Example 1

[0052] Appendix Figure 4 An arbitrary address PCIe DMA read / write implementation method is shown, which includes the following steps:

[0053] Step 0: The host detects whether there is data to be processed. If so, it requests a contiguous memory space I to store the data to be processed.

[0054] Step 1: Write the starting address of memory space I and the length of the data to be processed into the status word cmd_fifo of the FPGA;

[0055] The starting address of memory space I and the length of the data to be processed are both sent to the FPGA by the host through PIO mode. Memory space I is a continuous memory space requested by the host to store the data to be processed.

[0056] Step 2: The FPGA's DMA read control module checks in real time whether the status word com_fifo is empty.

[0057] When the status word cmd_fifo is found to be non-empty, the FPGA's DMA read control module calculates the byte enable signal based on the starting address and the length of the data to be processed, and encapsulates N TLP read request packets in DW units according to preset rules, and caches the starting address and the length of the data to be processed for this DMA read operation in the status word DMA_FIFO;

[0058] Each TLP read request packet includes a byte enable signal, a request data size, a request start address, and a read request tag; the read request tag is used to record the order of the TLP read request packets.

[0059] For each TLP read request packet sent, the read request tag is incremented by 1, the valid start address for reading is increased by the request data size of the previous TLP read request packet, and the length of the data to be processed is decreased by the request data size of the previous TLP read request packet.

[0060] Step 3: The TLP read request packets are sequentially sent to the PCIE root union RC via the FPGA data transmission module I to access the host memory;

[0061] Step 4: The FPGA's DMA read control module monitors in real time whether all TLP read request packets for this DMA read operation have been sent (whether the length of the data to be processed has been reduced to 0).

[0062] If so, the DMA read control module will cache the request data size (length information) and read request tag of each TLP read request packet in the status word TAG_FIFO in sequence;

[0063] The size of the request data in the TLP read request packet is set to correspond one-to-one with the read request tag;

[0064] Step 5: The FPGA's data receiving module monitors in real time whether there is a TLP read completion packet from the host.

[0065] When the data receiving module receives the TLP read complete packet, it parses out the payload data, DW valid signal (byte enable signal), and read complete tag of the TLP read complete packet, and sends the DW valid signal and payload data of the TLP read complete packet to the corresponding write FIFO control module in sequence according to the order of the read complete tags.

[0066] The number of write FIFO control modules is equal to the number of read completion tags, and each read completion tag corresponds one-to-one with the read request tag in step 4. The read completion tag is used to record the order of the TLP read completion packets so that the FPGA can sort the valid data according to the read completion tags in the TLP read completion packets.

[0067] Step 6: The FPGA's write FIFO control module continuously writes the load data into the first data buffer FIFO at the back end in units of DW.

[0068] The FPGA's data receiving module monitors in real time whether the TLP read completion packet corresponding to each read completion tag has been received. After detecting that the valid data corresponding to a certain read completion tag has been written to the first data buffer FIFO in the backend, it determines that the TLP read completion packet corresponding to the read completion tag has been received, generates the corresponding receive completion identifier, and sends it to the FPGA's read FIFO control module.

[0069] The receiving completion identifier and the reading completion tag are set to correspond one-to-one;

[0070] Step 7: After receiving the receive completion identifiers corresponding to all read completion tags, the read FIFO control module reads the load data in the corresponding first data buffer FIFO in a loop according to the obtained read completion tag order, and sorts the read load data according to the obtained starting address and the length of the data to be processed, and deletes redundant data in the first DW and / or the last DW.

[0071] It should be noted that steps 1 to 7 of the present invention are DMA read operation steps, which realizes redundant reading of host memory data at any accessible address by the FPGA, has the function of requesting data from memory at any starting address, and does not have special requirements on memory address.

[0072] It is understandable that the host sends the starting address and data length information of the memory data to the FPGA via PIO mode. The FPGA encapsulates multiple memory TLP read request packets in DW units according to the starting address and length information and sends them to the PCIe IPCORE, and records the valid information (starting address and requested data size, etc.). When the data receiving module receives the TLP read completion packet, it first caches the load data in the read completion packet into the first data buffer FIFO, then reads the received data from the first data buffer FIFO according to the cached valid information, adjusts the received data, and finally sends the complete and continuous valid data to the downstream module to complete the DMA read operation on arbitrary address memory (single byte), and does not require the starting address to be 16-byte aligned or 4K aligned.

[0073] Specifically, the byte enable signal is 0 or 1, where 1 represents valid data and 0 represents redundant data. It can be understood that adjusting the received data means that the read FIFO control module identifies valid and redundant data in the load data according to the byte enable signal, deletes redundant data, and sorts and splices the remaining valid data according to the order of the read completion tags to obtain complete and continuous valid data.

[0074] Furthermore, the number of write FIFO control modules, the number of first data buffer FIFOs, and the number of read request tags are equal, and the first data buffer FIFO uses a 32-bit width. It can be understood that the first data buffer FIFO includes data buffer FIFO0 to FIFOOn, and uses write FIFO control modules and multiple small-width FIFOs to form a large-width buffer space to buffer the received data.

[0075] This invention uses a 32-bit wide FIFO as the first data buffer IFO. The write FIFO control module can handle the impact of the RCB (Read Completion Boundary). It can be understood that after the TLP read completion packet is split by the RCB, the data is in units of DW (Data Wounds). The DMA data width can be 2DW, 4DW, or 8DW. Therefore, when writing to the 32-bit wide first data buffer FIFO, the write FIFO control module can arbitrarily concatenate the data according to the DW valid signal. For example, if the DMA data width is 4DW, and the RCB splits into two read completion packets (with the same tag): read completion packet 1 corresponds to a 1DW load, and read completion packet 2 corresponds to a 4DW load, then the write FIFO control module will first write the 1DW data from read completion packet 1 and the first 3DW data from read completion packet 2 to the data buffer FIFO, and then write the remaining DW from read completion packet 2 to the data buffer FIFO.

[0076] It should be noted that this invention employs a write FIFO control module and a first data buffer FIFO with the same number of read request tags as the actual number used, achieving design flexibility. When initiating a DMA read operation, this invention caches the starting address and length information, serving as the basis for data modification and adjustment in the read FIFO control module. After sorting the received data, the received data is uniformly modified and adjusted to ultimately obtain all continuous valid data. When reading from the first data buffer FIFO, redundant data is deleted and valid data is rearranged based on the information cached when initiating the DMA read operation, ensuring transmission reliability. Furthermore, this method has a clear structure, greatly reducing the development difficulty for reading data from arbitrary memory addresses.

[0077] Furthermore, in step 2, when the FPGA's DMA read control module calculates the byte enable signal based on the starting address and the length of the data to be processed, and encapsulates N TLP read request packets in DW units according to preset rules, it executes:

[0078] Determine whether the starting address in the status word cmd_fifo satisfies DW alignment. If DW alignment is satisfied, pad the ending address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal I to ensure that the last TLP read request packet satisfies DW alignment. If DW alignment is not satisfied, pad the starting address and ending address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal II to ensure that each TLP read request packet satisfies DW alignment.

[0079] Determine whether the length of each TLP read request packet exceeds the maximum read request length negotiated by the PCIe link. If it does, split the TLP read request packet into multiple sub-TLP read request packets with the maximum read request length as the request length, and the length of each sub-TLP read request packet will not cross the 4K boundary. If it does not exceed the maximum read request length, determine whether the TLP read request packet crosses the 4K boundary. If it does, split the TLP read request packet with the request data address as the boundary.

[0080] It should be noted that, for byte enable signal I, the valid end address is first obtained based on the start address and the length of the data to be processed, and then address padding is performed based on the valid end address to ensure that the valid end address satisfies DW alignment; for byte enable signal II, the valid end address is first obtained based on the start address and the length of the data to be processed, and then address padding is performed based on the start address and the valid end address respectively to ensure that both the start address and the valid end address satisfy DW alignment.

[0081] Specifically, the maximum read request length negotiated for the PCIe link can be any of 128DW (512 bytes), 64DW (256 bytes), 32DW (128 bytes), and 16DW (64 bytes).

[0082] In one specific implementation, taking an FPGA accelerator card as an example, the maximum read request length for PCIe link negotiation is 64 DW (256 bytes), and the RCB parameter is 64 bytes; the implementation method for arbitrary address DMA read operations is described, as follows: Figure 1 , 4 As shown in Figure 6:

[0083] (1) When the host has data that needs to be accelerated by the FPGA acceleration card, first apply for a continuous memory space to store the data to be accelerated as memory space I; assume that the starting address of memory space I is 0f46h, the ending address is 1016h, and the length of the data to be processed is 208 bytes.

[0084] (2) The host configures the starting address (0f46h) and data length information (208 bytes) of the forward memory to the FPGA in PIO mode and writes them into the status word cmd_fifo;

[0085] (3) When the DMA read control module finds that cmd_fifo is not empty (i.e., data is cached), it reads cmd_fifo to obtain the starting address (0f46h) and data length (208 bytes) of memory space I. Based on the maximum read request size negotiated by the PCIE link, 64 DW (256 bytes), and the processing across 4K pages, the data length is fragmented (in DW units):

[0086] It was determined that the start address (0f46h) and the end address (1016h) did not meet the DW alignment requirement;

[0087] Address padding is performed based on the starting address (0f46h) to ensure that the starting address of the first TLP read request packet satisfies DW alignment (0f44h), and the first DW byte enable signal II (0011) is generated; 0011 indicates that the first and second bytes are redundant data, and the third and fourth bytes are valid data;

[0088] Address padding is performed based on the end address (1016h) to ensure that the end address of the last TLP read request packet satisfies DW alignment (1018h), and the byte enable signal II (1100) of the last DW is generated; 1100 indicates that the first and second bytes are valid data, and the third and fourth bytes are redundant data;

[0089] It is understandable that the byte enable signal between the first DW and the last DW is 1111;

[0090] Since the TLP read request crosses the 4K boundary, the TLP read request is encapsulated into two TLP read request packets and sent to the PCIe IP CORE (PCIe IP core) through the data sending module. The PCIe IP CORE sends TLP read request packet 1 and TLP read request packet 2 to the PCIe root union RC, so that the PCIe root union RC can process the received TLP read request packets to form a TLP read completion packet. Among them, the request (read) data size of TLP read request packet 1 is 47DW (188Byte), and the request start address of TLP read request packet 1 is 0f44h. The request (read) data size of TLP read request packet 2 is 6DW (24Byte), and the request start address of TLP read request packet 2 is 1001h.

[0091] For each TLP read request packet sent, the read request tag (TAG number) is incremented by 1, the read address is increased by the request data size of the previous read request TLP, and the data length is decreased by the request data size of the previous read request TLP, until the data length is reduced to 0, at which point all read requests for this DMA read operation have been sent.

[0092] Because the TLP read completion packet corresponding to TLP read request packet 1 crosses the RCB boundary, the TLP read completion packet corresponding to TLP read request packet 1 is encapsulated into TLP read completion packet 1 (start address 0f44h, end address 0080h, length information 15DW) and TLP read completion packet 2 (start address 0081h, end address 1000h, length information 32DW).

[0093] The TLP read completion packet corresponding to TLP read request packet 2 did not cross the RCB boundary, so the TLP read completion packet corresponding to TLP read request packet was encapsulated into TLP read completion packet 3 (start address is 1001h, end address is 1018h, length information is 6DW).

[0094] (4) Cache the starting address (0f46h) and data length information (208 bytes) of this DMA read operation into DMA_FIFO, and cache the length information of the sent TLP read request packets (read request tag and length information 47DW for TLP read request packet 1, and read request tag and length information 6DW for TLP read request packet 2) into TAG_FIFO;

[0095] (5) When the data receiving module receives TLP read completion packet 1, TLP read completion packet 2 and TLP read completion packet 3, the data receiving module extracts the load data and read completion tag (TAG number), sets the byte enable signal, and sends it to the corresponding write FIFO control module according to the read completion tag (TAG number); the number of TAGs used for DMA read operation is equal to the number of write FIFO control modules and back-end data buffer FIFO;

[0096] Specifically, TLP read completion packet 1 and TLP read completion packet 2 are transmitted to write FIFO control module 1, and TLP read completion packet 3 is transmitted to write FIFO control module 2;

[0097] When the data receiving module detects that the TLP read completion packet of a certain read completion tag (TAG) has been received, it will send a read completion packet reception completion indication for that read completion tag (TAG) to the read FIFO control module.

[0098] (6) When the write FIFO control module receives the load data and byte enable signal, it writes the data continuously into the back-end data cache FIFO0 ~ FIFOon in DW units; where the bit width of data cache FIFO0 ~ FIFOon is 32 bits; the data cache FIFO is consistent with the area of ​​the host memory to be processed, forming a mapping relationship, which effectively improves storage efficiency;

[0099] Specifically, 47 DW data from TLP read completion packet 1 and TLP read completion packet 2 (corresponding to read request tag 0) are cached sequentially in the first data cache FIFO1, and 6 DW data from TLP read completion packet 3 (corresponding to read request tag 1) are cached sequentially in the first data cache FIFO2.

[0100] (7) The read FIFO control module receives the receive completion identifier 1 corresponding to read completion tag 1 and the receive completion identifier 2 corresponding to read completion tag 2, and reads the information in TAG_FIFO and DMA_FIFO, and sequentially reads the data buffer FIFO of the corresponding TAG starting from TAG value 0.

[0101] Based on the start address and length information in the DMA_FIFO, the read data is adjusted, invalid data is deleted, and valid data is concatenated and arranged before being sent to the downstream module.

[0102] Specifically, read 47 DW load data from the first data buffer FIFO1 and delete the redundant data of the first and second bytes in the first DW; read 6 DW data from the first data buffer FIFO2 and delete the redundant data of the third and fourth bytes in the last DW; concatenate and arrange the remaining 186 bytes of valid data in the first data buffer FIFO1 and the remaining 22 bytes of valid data in the first data buffer FIFO2 to obtain 208 bytes of valid data.

[0103] It is understood that this embodiment uses a 32-bit wide FIFO to buffer the received data, which is an effective method for handling read completion splits caused by RCB; it cleverly achieves the purpose of FPGA requesting data from memory at any starting address with less redundant data.

[0104] Example 2

[0105] It should be noted that the arbitrary address PCIe DMA read / write implementation method includes, in addition to the method for obtaining arbitrary address memory data in Embodiment 1, a method for writing data to arbitrary address memory; as shown in the appendix. Figure 5 The specific steps of a DMA write operation are shown below:

[0106] Step a: Configure the starting address of memory space II in the FPGA as the starting address of the DMA write operation;

[0107] The starting address of memory space II is configured by the host to the FPGA through PIO mode. Memory space II is a contiguous memory space requested by the host for storing data from the FPGA.

[0108] Step b: When the FPGA has data to send to the host, write the data to be sent into the second data buffer FIFO, and write the length of the data to be sent and the starting address of memory space II into the status word state_fifo.

[0109] Step c: The FPGA's DMA write control module reads the length of the data to be sent and the starting address of memory space II from the status word state_fifo, calculates the effective starting address of DMA write and the byte enable signal, and divides the data to be sent into several sub-data to be sent according to the maximum load of the PCIe link negotiation and the processing across 4k pages, and sends the relevant information of DMA write start to the FPGA's data transmission module II.

[0110] The relevant information for DMA write initiation includes byte enable signal, write request start address, and write data size;

[0111] In step d, the data sending module II obtains the relevant information of DMA write initiation, reads the data to be sent from the second data buffer FIFO in sequence, adds redundant data to fill it, splices the redundant data and valid data, encapsulates M TLP write request packets in DW unit and sends them to the PCIE root union RC, and writes them into the host memory.

[0112] It should be noted that in the method of writing data to memory at any address: the host sends the starting address of the memory to the FPGA through PIO mode. When the FPGA has data to send to the host, it first segments the data and generates byte enable signals according to the starting address, the length of the data to be sent and the maximum load negotiated by the PCIe link. Then, it encapsulates the data into TLP write request packets in DW units and sends them to the PCIe IP CORE to complete the DMA write operation at any address (single byte).

[0113] Further, in step c, the length and starting address of the data to be sent are read from the state_fifo, the effective starting address and byte enable signal for DMA write are calculated, and the data to be sent is fragmented into several sub-data segments based on the maximum load of the PCIe link negotiation and the processing across 4k pages. Then, the following steps are executed:

[0114] Determine whether the start address in the state word state_fifo satisfies DW alignment. If DW alignment is satisfied, pad the end address according to the start address and the length of the data to be sent, and generate a byte enable signal III to ensure that the last TLP write request packet satisfies DW alignment. If DW alignment is not satisfied, pad the start address and end address according to the start address and the length of the data to be sent in the state word state_fifo, and generate a byte enable signal IV to ensure that each TLP write request packet satisfies DW alignment.

[0115] Determine whether the length of each TLP write request packet exceeds the maximum write request length negotiated by the PCIe link. If it does, split the TLP write request packet into multiple sub-TLP write request packets in units of DW, and the length of each sub-TLP write request packet will not cross the 4K boundary. If it does not exceed the limit, determine whether the TLP write request packet crosses the 4K boundary. If it does, split the TLP write request packet in units of DW.

[0116] In another specific implementation, taking an FPGA accelerator card as an example, the maximum write request length for PCIe link negotiation is 64 DW (256 bytes), and the RCB parameter is 64 bytes; the implementation method for arbitrary address DMA write operations is described, as follows: Figure 2 and 5 As shown:

[0117] (1) First, the host requests a contiguous memory space II (starting address is 0f46h) and configures the starting address (0f46h) of memory space II to the FPGA through PIO mode;

[0118] (2) When the FPGA has data to send to the host, the FPGA writes the data to be sent into the second data buffer FIFO and writes the length (108 bytes) and starting address (0f46h) of the data to be sent into state_fifo;

[0119] (3) The DMA write control module reads the length of the data to be sent (108 bytes) and the start address (0f46h) in the state_fifo, and calculates the valid end address and byte enable signal for the DMA write:

[0120] Specifically, the starting address of the DMA write is 0f46h, and the effective ending address of the DMA write is 0fb2h;

[0121] It is determined that the starting address (0f46h) does not meet the DW alignment requirement. Address padding is performed based on the starting address (0f46h) to obtain the padded starting address (0f44h). The first DW byte enable signal Ⅳ (0011) is generated, where 0011 indicates that the first and second bytes are redundant data and the third and fourth bytes are valid data.

[0122] If the valid end address (0fb2h) is determined to be not DW aligned, address padding is performed based on the valid end address (0fb2h) to obtain the padded end address (0fb4); and the byte enable signal Ⅳ (1100) of the last DW is generated; 1100 indicates that the first and second bytes are valid data, and the third and fourth bytes are redundant data;

[0123] Based on the maximum PCIe link negotiation payload of 64DW (256 bytes) and the requirement to process across 4k pages, the data length is fragmented, and the DMA write initiation information is sent to the data transmission module:

[0124] Specifically, the padded start address (0f44h) and the padded end address (0fb4) do not cross the 4K boundary, and the data length is less than 64DW (256 bytes). Therefore, there is no need to fragment the data length. Only one TLP write request packet needs to be sent (write request start address is 0f44h, end address is 0fb4, and write data size is 28DW).

[0125] (4) Data transmission module II obtains the relevant information for DMA write, reads data from the second data buffer FIFO in sequence, and adds redundant data to fill it (such as...). Figure 3 (Shaded area) The redundant data and valid data are spliced ​​together, encapsulated into a memory TLP write request packet, and sent to the PCIe IP CORE to write the data into the host memory;

[0126] Specifically, the first DW header has 2 bytes of redundant data, and the last DW tail has 2 bytes of redundant data. The host obtains the location and size of the redundant data through the byte enable signal, deletes the redundant data, and then concatenates the remaining data into continuous and complete valid data.

[0127] The DMA write operation implementation method of the present invention can write data to memory at any starting address without any special requirements on the memory address; it enables the host to directly obtain the original data of the FPGA device without requiring the FPGA device to adjust the data in the second data buffer FIFO to adapt to the DMA write operation, thus achieving the independence of the engineering structure; the data transmission module reads the data in the second data buffer FIFO, and according to the byte enable signal, data length and memory address, adds invalid redundant data and original valid data to concatenate them and encapsulate them into a memory write TLP request packet.

[0128] The DMA write method of the present invention realizes the DMA operation of writing data of arbitrary length to memory at any starting address. By analyzing the memory address and combining the data length, the byte enable signal required for the DMA write operation at any address (single byte transmission) is generated, and the payload data of the corresponding length is added and encapsulated into a memory TLP write request packet.

[0129] Example 3

[0130] Based on Embodiments 1 and 2, this embodiment provides a specific implementation of an FPGA device; the FPGA device includes a DMA read operation management unit, which includes a status word cmd_fifo, a DMA read control module, a data transmission module I, a status word DMA_FIFO, a status word TAG_FIFO, a data receiving module, a write FIFO control module, a first data buffer FIFO, and a read FIFO control module, wherein,

[0131] The status word cmd_fifo is used to store the starting address of memory space I and the length of the data to be processed; wherein, the starting address of memory space I and the length of the data to be processed are both sent to the FPGA by the host through PIO mode, and memory space I is a contiguous memory space allocated by the host to store the data to be processed;

[0132] The DMA read control module is used to calculate a byte enable signal based on the starting address and the length of the data to be processed when the status word cmd_fifo is not empty, and encapsulate N TLP read request packets in DW units according to preset rules, and cache the starting address and the length of the data to be processed in the status word DMA_FIFO; it is also used to cache the request data size and read request tag in each TLP read request packet in the status word TAG_FIFO when all TLP read request packets of the current DMA read operation have been sent.

[0133] Each TLP read request packet includes a byte enable signal, the requested data size, the request start address, and a read request tag;

[0134] The data sending module I is used to send the TLP read request packet to the PCIe IP CORE via the PCIe Root Complex (RC) to access the host memory;

[0135] The data receiving module is used to parse the payload data, byte enable signal, and read completion tag of the TLP read completion packet when it receives the TLP read completion packet, and send the byte enable signal and payload data of the read completion packet to the corresponding write FIFO control module according to the read completion tag; it is also used to generate a corresponding reception completion identifier after detecting that the TLP read completion packet corresponding to a certain read request tag has been received, indicating that all the data of the read request has been received, and send it to the read FIFO control module.

[0136] The write FIFO control module is used to continuously write the load data into the first data buffer FIFO in the backend in units of DW.

[0137] The read FIFO control module is used to read the load data in the corresponding first data buffer FIFO in a loop according to the order of the read request tags after receiving the receive completion identifiers corresponding to all read completion tags, and sort the read load data according to the obtained starting address and the length of the data to be processed, and delete redundant data in the first DW and / or the last DW.

[0138] Furthermore, the FPGA device also includes a DMA write operation management unit, which comprises a status word state_fifo, a second data buffer FIFO, a DMA write control module, and a data transmission module II.

[0139] The state word state_fifo is used to store the starting address of memory space II and the length of the data to be sent; wherein, the starting address of memory space II is configured to the FPGA by the host through PIO mode, and memory space II is a contiguous memory space requested by the host.

[0140] The second data buffer FIFO is used to store data to be sent;

[0141] The DMA write control module is used to read the length of the data to be sent and the starting address of memory space II from the status word state_fifo, calculate the effective start address of the DMA write and the byte enable signal; it is also used to divide the data to be sent into several sub-data according to the maximum load of the PCIe link negotiation and the processing across 4k pages, and send the relevant information of DMA write initiation to the data sending module II; wherein, the relevant information of DMA write initiation includes the byte enable signal, the write request start address and the write data size;

[0142] The data transmission module II is used to obtain relevant information about DMA write initiation, read the data to be sent from the second data buffer FIFO in sequence, add redundant data to fill it, splice the redundant data and valid data, encapsulate M TLP write request packets in DW units, and send the TLP write request packets to the PCIE root union RC through the PCIE IP CORE and write them into the host memory.

[0143] Furthermore, the DMA read control module is specifically used for:

[0144] Determine whether the starting address in the status word cmd_fifo satisfies DW alignment. If DW alignment is satisfied, pad the ending address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal I to ensure that the last TLP read request packet satisfies DW alignment. If DW alignment is not satisfied, pad the starting address and ending address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal II to ensure that each TLP read request packet satisfies DW alignment.

[0145] Determine whether the length of each TLP read request packet exceeds the maximum read request length negotiated by the PCIe link. If it does, split the TLP read request packet into multiple sub-TLP read request packets with the maximum read request length as the request length, and the length of each sub-TLP read request packet will not cross the 4K boundary. If it does not exceed the maximum read request length, determine whether the TLP read request packet crosses the 4K boundary. If it does, split the TLP read request packet with the request data address as the boundary.

[0146] Furthermore, the DMA write control module is specifically used for:

[0147] Determine whether the start address in the state word state_fifo satisfies DW alignment. If DW alignment is satisfied, pad the end address according to the start address and the length of the data to be sent, and generate a byte enable signal III to ensure that the last TLP write request packet satisfies DW alignment. If DW alignment is not satisfied, pad the start address and end address according to the start address and the length of the data to be sent in the state word state_fifo, and generate a byte enable signal IV to ensure that each TLP write request packet satisfies DW alignment.

[0148] Determine whether the length of each TLP write request packet exceeds the maximum write request length negotiated by the PCIe link. If it does, split the TLP write request packet into multiple sub-TLP write request packets in units of DW, and the length of each sub-TLP write request packet will not cross the 4K boundary. If it does not exceed the limit, determine whether the TLP write request packet crosses the 4K boundary. If it does, split the TLP write request packet in units of DW.

[0149] Furthermore, the number of write FIFO control modules, the number of the first data cache FIFO, and the number of read request tags are equal, and the data cache FIFO adopts a 32-bit width.

[0150] Example 4

[0151] Based on Embodiment 3, this embodiment provides an FPGA-based communication system, which includes a host and the FPGA device from Embodiment 3, wherein...

[0152] The host is used to monitor whether there is data to be processed. If so, it requests a contiguous memory space to store the data to be processed and sends the starting address of the contiguous memory space and the length of the data to be processed to the FPGA device through PIO mode. It is also used to receive TLP read request packets from the FPGA device and process them to generate corresponding TLP read completion packets and return them to the FPGA device. It is also used to receive TLP write request packets from the FPGA device.

[0153] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for implementing arbitrary address PCIe DMA read / write, characterized in that, Includes the following steps: Step 1: Write the starting address of memory space I and the length of the data to be processed into the status word cmd_fifo of the FPGA; The starting address of memory space I and the length of the data to be processed are both sent to the FPGA by the host through PIO mode. Memory space I is a continuous memory space requested by the host to store the data to be processed. Step 2: When the FPGA's DMA read control module finds that the status word cmd_fifo is not empty, it calculates the byte enable signal based on the starting address and the length of the data to be processed, and encapsulates N TLP read request packets in DW units according to preset rules, and caches the starting address and the length of the data to be processed for this DMA read operation in the status word DMA_FIFO. Each TLP read request packet includes a byte enable signal, the requested data size, the request start address, and a read request tag; Step 3: The FPGA's data transmission module I sends the TLP read request packet to the PCIe IP CORE via the PCIe root union RC to access the host memory; Step 4: When all TLP read request packets for this DMA read operation have been sent, the FPGA DMA read control module will sequentially cache the request data size and read request tag in each sent TLP read request packet into the status word TAG_FIFO. Step 5: When the FPGA data receiving module receives the TLP read complete packet, it parses out the load data, DW valid signal and read complete tag of the TLP read complete packet, and sends the DW valid signal and load data of the TLP read complete packet to the corresponding write FIFO control module according to the read complete tag. The read completion tag corresponds one-to-one with the read request tag in step 4; Step 6: The FPGA's write FIFO control module continuously writes the load data into the first data buffer FIFO at the back end in units of DW. When the FPGA's data receiving module detects that the TLP read completion packet corresponding to a certain read completion tag has been received, it generates the corresponding receive completion identifier and sends it to the FPGA's read FIFO control module. Step 7: After receiving the receive completion identifiers corresponding to all read completion tags, the read FIFO control module reads the load data in the corresponding first data buffer FIFO in a loop according to the order of the obtained read completion tags, and sorts the read load data according to the starting address and the length of the data to be processed obtained from the status word DMA_FIFO, and deletes redundant data in the first DW and / or the last DW.

2. The arbitrary address PCIe DMA read / write implementation method according to claim 1, characterized in that, The method also includes the specific steps of a DMA write operation: Step a: Configure the starting address of memory space II for DMA write operations on the FPGA; The starting address of memory space II is configured to the FPGA by the host through PIO mode, and memory space II is a contiguous memory space requested by the host. Step b: When the FPGA has data to send to the host, write the data to be sent into the second data buffer FIFO, and write the length of the data to be sent and the starting address of memory space II into the status word state_fifo. Step c: The FPGA's DMA write control module reads the length and starting address of the data to be sent from the state_fifo, calculates the effective starting address and byte enable signal for DMA write, and divides the data to be sent into several sub-data to be sent based on the maximum load negotiated by the PCIe link and the processing across 4k pages. The module then sends the relevant information for DMA write initiation to the FPGA's data transmission module II. The relevant information for DMA write initiation includes byte enable signal, write request start address, and write data size; Step d: The data sending module II obtains the relevant information of DMA write start, reads the data to be sent from the second data buffer FIFO in sequence, adds redundant data to fill, splices the redundant data and valid data, encapsulates M TLP write request packets in DW unit, and sends the TLP write request packets to the PCIE root union RC through PCIE IP CORE and writes them into the host memory.

3. The arbitrary address PCIe DMA read / write implementation method according to claim 1, characterized in that, In step 2, when the FPGA's DMA read control module calculates the byte enable signal based on the starting address and the length of the data to be processed, and encapsulates N TLP read request packets in DW units according to preset rules, it executes: Determine whether the starting address in the status word cmd_fifo satisfies DW alignment. If DW alignment is satisfied, pad the address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal I to ensure that the last TLP read request packet satisfies DW alignment. If DW alignment is not satisfied, pad the address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal II to ensure that each TLP read request packet satisfies DW alignment. Determine whether the length of each TLP read request packet exceeds the maximum read request length negotiated by the PCIe link. If it does, split the TLP read request packet into multiple sub-TLP read request packets with the maximum read request length as the request length, and the length of each sub-TLP read request packet will not cross the 4K boundary. If it does not exceed the maximum read request length, determine whether the TLP read request packet crosses the 4K boundary. If it does, split the TLP read request packet with the request data address as the boundary.

4. The arbitrary address PCIe DMA read / write implementation method according to claim 1, characterized in that, The number of write FIFO control modules, the number of the first data cache FIFO, and the number of read request tags are equal, and the first data cache FIFO uses a 32-bit width.

5. An FPGA device, characterized in that, This includes a status word cmd_fifo, a DMA read control module, a data transmission module I, a status word DMA_FIFO, a status word TAG_FIFO, a data reception module, a write FIFO control module, a first data buffer FIFO, and a read FIFO control module. The status word cmd_fifo is used to store the starting address of memory space I and the length of the data to be processed; wherein, the starting address of memory space I and the length of the data to be processed are both sent to the FPGA by the host through PIO mode, and memory space I is a contiguous memory space allocated by the host to store the data to be processed; The DMA read control module is used to calculate a byte enable signal based on the starting address and the length of the data to be processed when the status word cmd_fifo is not empty, and encapsulate N TLP read request packets in DW units according to preset rules, and cache the starting address and the length of the data to be processed in the status word DMA_FIFO; it is also used to cache the request data size and read request tag in each TLP read request packet in the status word TAG_FIFO when all TLP read request packets of the current DMA read operation have been sent. Each TLP read request packet includes a byte enable signal, the requested data size, the request start address, and a read request tag; The data sending module I is used to send the TLP read request packet to the PCIe IP CORE via the PCIe Root Consortium RC to access the host memory; The data receiving module is configured to, upon receiving a TLP read completion packet, parse out the payload data, DW valid signal, and read completion tag of the TLP read completion packet, and send the DW valid signal and payload data of the read completion packet to the corresponding write FIFO control module according to the read completion tag; it is also configured to, upon detecting that the TLP read completion packet corresponding to a certain read completion tag has been received, generate a corresponding reception completion identifier and send it to the read FIFO control module; wherein, the read completion tag corresponds one-to-one with the read request tag; The write FIFO control module is used to continuously write the load data into the first data buffer FIFO in the backend in units of DW. The read FIFO control module is used to read the load data in the corresponding first data buffer FIFO in a loop according to the order of the read completion tags after receiving the receive completion identifiers corresponding to all read completion tags. The module also sorts the read load data according to the starting address and the length of the data to be processed in the status word DMA_FIFO, and deletes redundant data in the first DW and / or the last DW.

6. The FPGA device according to claim 5, characterized in that, It also includes the status word state_fifo, the second data buffer FIFO, the DMA write control module, and the data transmission module II, among which, The state word state_fifo is used to store the starting address of memory space II and the length of the data to be sent; wherein, the starting address of memory space II is configured to the FPGA by the host through PIO mode, and memory space II is a contiguous memory space requested by the host; The second data buffer FIFO is used to store data to be sent; The DMA write control module is used to read the length of the data to be sent and the starting address of memory space II from the status word state_fifo, calculate the effective start address of the DMA write and the byte enable signal; it is also used to divide the data to be sent into several sub-data according to the maximum load of the PCIe link negotiation and the processing across 4k pages, and send the relevant information of DMA write initiation to the data sending module II; wherein, the relevant information of DMA write initiation includes the byte enable signal, the write request start address and the write data size; The data sending module II is used to read the data to be sent from the second data buffer FIFO in sequence according to the relevant information of DMA write start, add redundant data to fill, splice the redundant data and valid data, encapsulate M TLP write request packets, and send the TLP write request packets to the PCIe root union RC through the PCIe IP CORE and write them into the host memory.

7. The FPGA device according to claim 5, characterized in that, The DMA read control module is specifically used for: Determine whether the starting address in the status word cmd_fifo satisfies DW alignment. If DW alignment is satisfied, pad the address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal I to ensure that the last TLP read request packet satisfies DW alignment. If DW alignment is not satisfied, pad the address according to the starting address and the length of the data to be processed in the status word cmd_fifo, and generate byte enable signal II to ensure that each TLP read request packet satisfies DW alignment. Determine whether the length of each TLP read request packet exceeds the maximum read request length negotiated by the PCIe link. If it does, split the TLP read request packet into multiple sub-TLP read request packets with the maximum read request length as the request length, and the length of each sub-TLP read request packet will not cross the 4K boundary. If it does not exceed the maximum read request length, determine whether the TLP read request packet crosses the 4K boundary. If it does, split the TLP read request packet with the request data address as the boundary.

8. The FPGA device according to claim 5, characterized in that, The number of write FIFO control modules, the number of the first data cache FIFO, and the number of read request tags are equal, and the first data cache FIFO uses a 32-bit width.

9. A communication system based on FPGA, characterized in that: Includes a host computer and the FPGA device according to any one of claims 5 to 8, wherein, The host is used to monitor whether there is data to be processed. If so, it requests a contiguous memory space to store the data to be processed and sends the starting address of the contiguous memory space and the length of the data to be processed to the FPGA device through PIO mode. It is also used to receive TLP read request packets from the FPGA device and process them to generate corresponding TLP read completion packets and return them to the FPGA device. It is also used to receive TLP write request packets from the FPGA device.

Images