PCI Express data transfer

Abstract

PCIe devices and corresponding procedures are provided, whereby the length of data to be transferred is aligned with a multiple of a double-word length.

Description

TECHNICAL FIELD

[0001] This application relates to data transmission via PCI Express (Peripheral Component Interconnect Express). For example, this application may relate to corresponding devices or methods. BACKGROUND

[0002] PCI Express (PCIe) is a high-speed serial computer expansion bus standard designed to connect peripheral devices to a main processor chipset (e.g., a central processing unit). The PCI Express electrical interface is also used in various other standardized interfaces, such as ExpressCard, a laptop expansion card interface, and SATA Express, a computer storage interface. PCI Express essentially defines separate point-to-point serial connections. Different endpoints can be connected via switches. This allows for direct connections between PCI Express devices.

[0003] Applications running at a higher layer than PCI Express, such as applications that reference a peripheral device, can use different data unit sizes, such as different frame sizes or different packet sizes. Conventional implementations of PCI Express and point devices read a precise amount of data available over a bus. Such data units do not need to be aligned to the double words (DWORDs) used for PCI data transfer. In some cases, this can lead to performance degradation.

[0004] The PCIe-DMA ConfigFPGA Design User Interface Manual Rev. 1.19, dated August 27, 2012, discloses adapting data in DWORD format to a 64-bit data channel for DMA addressing. To adapt addresses, dummy data can be used.

[0005] US 2007 / 0263629 A1 discloses the use of descriptors in data transmission.

[0006] US 2004 / 0123013 A1 discloses including additional filler bytes for calculating a debug code, wherein a descriptor can be used to describe which filler bytes are to be included in the calculation.

[0007] US 2013 / 082 851 A1 discloses dictionary-based compression for PCIe. SHORT SUMMARY

[0008] In one embodiment, an apparatus is provided as defined in claim 1. In another embodiment, a method is provided as defined in claim 10. The dependent claims define further embodiments. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic diagram of a PCI system according to one embodiment. Fig. Figure 2 illustrates a DWORD alignment. Fig. 3 shows a detailed block diagram of a device according to an embodiment. Fig. 4 shows the operation of the embodiment of the Fig. 3 when sending data. Fig. 5 shows the operation of the embodiment of the Fig. 3 when receiving data. Fig. 6 shows an embodiment of a method for sending data. Fig. 7 shows an embodiment of a method for receiving data. Fig. 8 and Fig. 9 show the performance in comparison systems. Fig. 10 shows the performance in a system according to an embodiment. DETAILED DESCRIPTION

[0009] Various embodiments are discussed in detail below with reference to the attached drawings. It should be noted that the embodiments shown in the drawings or described herein serve merely as illustrative examples and are not to be interpreted as limiting. For example, while embodiments may be described as including a plurality of features or elements, in other embodiments some of these features or elements may be omitted and / or replaced with alternative features or elements. In other embodiments, additional features or elements may be provided. Features of different embodiments may be combined to form further embodiments, unless otherwise indicated.

[0010] Any direct connection or couplings between elements shown in the drawings or described herein may be a direct connection or coupling, i.e. a connection or coupling without additional intervening elements, or an indirect connection or coupling, i.e. a connection or coupling with one or more intervening elements, as long as the basic function of the connection or coupling, for example to transmit a certain type of information or a certain type of signal, is substantially retained.

[0011] Embodiments refer to PCI Express devices. All terminology used herein is intended to have the meaning generally used in the field of PCI Express devices, for example, as defined in various applicable PCI Express standards, unless otherwise specified. PCI Express devices may be devices that include or utilize a PCI Express-based interface, including, for example, ExpressCard devices or SATA Express devices.

[0012] Turning now to the figures, in Fig. 1 shows an embodiment of a PCIe communication system. In Fig. 1, a first PCIe device 10 is shown communicating with a second PCIe device 11 via one or more lanes 110. In some embodiments, each of the one or more lanes 110 includes a first differential line, such as a first twisted pair, for sending data from device 10 to device 11 and a second differential line, such as a second twisted pair, for communicating from PCIe device 11 to PCIe device 10. One or more lanes 110 connecting PCIe devices 10 and 11 are also referred to as a link. In other systems, more than two PCIe devices may be included, and switches may be used to route data between desired PCIe devices.

[0013] For example, PCIe device 10 may be an endpoint device, and PCIe device 11 may be a PCIe host device, sometimes referred to as a root complex. In other embodiments, communication may be between two PCIe endpoints.

[0014] The first PCIe device 10 includes a transaction layer 13, a data link layer 14, and a physical layer 15. The second PCIe device 11 includes a transaction layer 17, a data link layer 18, and a physical layer 19. These layers may, in some embodiments, be defined or implemented substantially as in conventional PCIe devices.

[0015] Furthermore, the PCIe device 10 includes one or more higher layers or other circuits 12 to implement a desired functionality of the PCIe device 10 and / or techniques disclosed herein, and the PCIe device 11 includes one or more higher layers or other circuits 16 to implement a desired functionality of the PCIe device 11 and / or techniques disclosed herein.

[0016] The PCIe functionality, ie, communicating, for example, according to a PCIe standard or a PCIe-related standard, is implemented in the PCIe device 10 by layers 13-15 and by layers 17-19 of the PCIe device 11. Apart from differences discussed in more detail below, the communication may essentially be standard PCIe communication.

[0017] Physical layer 15 or 19 corresponds to the physical implementation of the PCIe interface and can be divided into two sublayers according to an electrical and logical specification. Physical layer 15 or 19 can, for example, include a next switch.

[0018] The link layer 14 or 18 sequences transaction layer packets (TLPs) received from the transaction layer 13 or 17, respectively, ensures reliable delivery of transaction layer packets between two PCIe endpoints via an acknowledgment protocol (e.g., between devices 10 and 11), and, in some embodiments, manages flow control credits. The transaction layer 13 or 17 transports payload data between a logical sender and receiver (e.g., between circuits 12 and 16), ignoring, for example, switches between them. For example, transaction layer packets (TLPs) can be formed in the transaction layers 13 or 17.

[0019] The above is intended to be only a brief overview of how the layers work, and any functionality known in the field of PCI Express devices can be implemented.

[0020] As mentioned, the transaction layer 13 or 17 generates third transaction layer packets (TLPs) for sending data. These packets contain a plurality of 32-bit words, also referred to below as DWORDs (double words). A TLP may include a header, which may be, for example, three or four DWORDs long, and one or more DWORDs containing the actual data, such as data to be written.

[0021] For example, when sending data, the transaction layer 13 receives data to be written (sent) from higher layers or other circuits 12. The higher layers or other circuits 12 may use data units that are not aligned with DWORDs, for example, having a length that does not correspond to an integer number of DWORDs. Such data units may be, for example, frames or packets. In embodiments, in the event of a misalignment, the transaction layers 13, 17 or an entity above the transaction layers 13, 17 within the PCIe device 10 add additional data, e.g., dummy data, in the event of a misalignment, for example, so that the data to be written is aligned with a DWORD. After the data is transmitted, this additional data may be removed or may not be processed. Similar techniques may be used when receiving data.In some embodiments, after the data transmission, an associated data descriptor used for the transmission does not include the additional data such as overhead bytes or overhead bits, so that further processing stages that process received data ignore the additional data.

[0022] A schematic example of such an alignment is shown in Fig. 2. In the example of Fig. 2, three data units 20 to 22 are provided, for example, from a high layer. The length of the three data units 20 to 22 is in the example of the Fig. 2 longer than the length of three DWORDs 23 to 25, but shorter than the length of four DWORDs 23 to 26, ie, misaligned to the DWORD length. In some embodiments, additional bits or bytes 27 are used as dummy data so that the length is aligned. For example, since a DWORD comprises 32 bits corresponding to four bytes and data units such as data units 20 to 23 in Fig. 2 often comprise an integer number of bytes, the additional dummy data 27 in some embodiments comprise one, two or three bytes. It should be noted that the relative length and the numbers of the data units and DWORDs in Fig. 2 is merely an example, and other relationships, including data units being shorter than DWORDs, are also possible. In some embodiments, the alignment may be dynamic, e.g., continuously adjusted to the data being sent.

[0023] In some embodiments, such DWORD length alignment may improve a transfer indicated by a PCI Express endpoint. While in some embodiments, in the case of non-DWORD length aligned data, as in Fig. 2, a few additional bytes or bits were read or written, this may nevertheless speed up the overall data transfer in some embodiments. In some embodiments, buffer stages of a PCI Express host system, as well as those of the endpoint, may be designed to accommodate up to a maximum transfer size, and therefore, alignment according to techniques disclosed herein may be implemented in some embodiments without significant hardware changes.

[0024] In Fig. 3 shows an example architecture of a system embodiment. The system of Fig. Figure 3 illustrates communication between a PCI endpoint device (elements 30 to 35) and a host device (elements 37 to 310) or root complex. The endpoint device may comprise, for example, a peripheral device, and the host device may comprise a PCI chipset and / or a central processing unit (CPU) of a computer. In other embodiments, other devices communicating according to PCI Express or a PCI Express-based standard may be used.

[0025] The PCIe endpoint device includes a PCIe endpoint 35, for example, hardware associated with sending or receiving data, an endpoint memory direct access (DMA) engine 33 for initiating and performing data transfers, an endpoint descriptor handling engine 34, an endpoint packet memory 30, an endpoint descriptor memory 31, and an intelligent transfer adaptation block 32. Apart from the intelligent transfer adaptation block 32 and its interaction with the other components, the endpoint device may be implemented in any conventional manner using hardware, software, firmware, or combinations thereof. Various functions of the Fig. The components illustrated in Figure 3 may be implemented using hard-wired logic and / or one or more appropriately programmed processors. The memories 30, 31 may be implemented as separate memories (e.g., random access memory (RAM) or any other conventional memory) or may be implemented as a common memory. In general, describing devices in terms of distinct blocks, machines, circuits, etc., should not be construed to indicate that these elements must be implemented in a physically separate manner.

[0026] The host device includes PCIe host hardware 37, system memory 38, host packet memory 39, and host descriptor memory 310, and may be implemented in a conventional manner.

[0027] In the embodiment of the Fig. 3, for data units not aligned to a DWORD length, such as packets or frames, the intelligent transfer adaptation block 32 can calculate a next DWORD-aligned length and adjust a transfer descriptor used by the endpoint DMA engine 33. For data within the endpoint device, the intelligent transfer adaptation block 32 can present an actual data unit length for a next processing stage to remain consistent with the requested transfer, i.e., additional data (dummy data) that has been added can be deleted. A descriptor on the host device side, stored, for example, in the host descriptor memory 310, can be maintained at an original data unit length in embodiments.In some embodiments, data transmission may be fully customized to a desired transmission, but performance may be improved in some embodiments. PCIe devices using techniques disclosed herein may communicate with legacy devices, e.g., hosts.

[0028] Next, a data flow for sending data from an endpoint to a host is described with reference to the Fig. 4 and Fig. 6 shown. Fig. 4 shows the system of Fig. 3 with arrows marked AI. Fig. 6 shows a method according to an embodiment, which in the system of Fig. 3 and Fig. 4 can be implemented. The procedure of Fig. 6 can also be implemented in other devices. In addition, while the method of Fig. 6 as a sequence of operations or events 60 to 68, in other embodiments, other orders of these operations or events may be used, some operations or events may be omitted, and / or operations and events may be implemented substantially simultaneously. When applied to the system of Fig. 3, the processes 60 to 68 arrows AI in Fig. 4, as explained in detail below. For better illustration, the procedure of Fig. 6 with reference to Fig. 4 described.

[0029] At 60 in Fig. 4 receives, as indicated by an arrow A in Fig. 4, the intelligent transmission adaptation block 32 receives data from previous processing stages in the endpoint device, for example from higher layers or other circuits as indicated by 12 or 16 in Fig. 1. The intelligent transfer adjustment block 32 then checks an alignment of the received data (e.g., data to be transferred), ie, checks whether the data is aligned to DWORDs. If alignment is necessary, the intelligent transfer adjustment block 32 can align the data by adjusting a pointer to the next DWORD-length aligned address in a memory in which the data is stored, plus adding additional data, e.g., one to three bytes, as described with reference to Fig. 2. The pointer can be provided in a descriptor.

[0030] At 61, this pointer is passed to the endpoint descriptor handling engine 34, as indicated by an arrow B in Fig. 4 is displayed.

[0031] Next, at 62, the endpoint descriptor handling engine 34 prepares a packet descriptor for the endpoint DMA engine 33 and stores it in the memory 31, which in this case serves to store transmit (TX) DMA descriptors. This is indicated by an arrow C in Fig. 4 is displayed.

[0032] At 63, as indicated by an arrow B in the figure, the endpoint DMA engine 33 reads the descriptor from the memory 31. Next, the data, which is now aligned with the DWORD length, is transferred over the PCIe link to the packet memory 39 on the host side. This is indicated by an arrow E in Fig. 4 is displayed.

[0033] The transmission of the aligned data can essentially be accomplished using conventional PCIe techniques, using a transmission layer, a link layer, and a physical layer. For example, the aligned data can be used to form transaction layer packets (TLPs), which can then be processed in the normal way. The functions implemented by the intelligent transmission adaptation block 32 can be implemented in a layer above the transaction layer.

[0034] After the transfer at 64, the DMA engine updates the descriptor stored in the descriptor memory 31 at 65 according to the transfer of the data, as indicated by an arrow F in Fig. 4. For example, the descriptor can be updated with information regarding the fact that data has been transferred.

[0035] Next, at 66, the transfer adjustment block 32 modifies this descriptor to adjust the data length back to the original length, without the data added for alignment. In other words, after this adjustment, the descriptor describes the transferred data as if it had been transferred without the additional data, for example, without the additional bytes. This is shown in Fig. 4 indicated by an arrow G.

[0036] At 67, the descriptor just adapted is passed to the endpoint descriptor handling engine 34, as indicated by an arrow H in Fig. 4. At 61, the endpoint descriptor handling engine 34 then exchanges the descriptor with the host side, such that the descriptor is stored, for example, in the host descriptor memory 310. This exchange can be performed as in conventional systems.

[0037] In some embodiments, by providing the adjusted descriptor corresponding to the original length, e.g., without the added dummy data, the transfer of the dummy data is transparent to blocks, devices, or layers that subsequently handle the data, e.g., on the host side. In other words, the DMA hardware receives a descriptor including the additional data, but a subsequent instance, e.g., on the host side, receives an adjusted descriptor corresponding to the actual data length without the added dummy data. Therefore, subsequent instances essentially do not perceive the added data but process the data in a regular way.

[0038] Next, with reference to the Fig. 5 and Fig. 7 receiving data in the Express endpoint from a host using the system of Fig. 3 as an example. While a method of Fig. 7 is presented as a sequence of processes or events, is, as already explained for the procedure of Fig. 6 explains that the order in which the processes or events are described is not to be interpreted as limiting. Furthermore, Fig. 5 is only referred to in order to explain the procedure of Fig. 7 to further illustrate, and the procedure of Fig. 7 can also be used on systems other than the one in Fig. 3 may be applicable. In other embodiments, some of the illustrated processes or events may be omitted.

[0039] In 70 of the Fig. 7 receives, as indicated by an arrow K in Fig. 5, the endpoint device receives a descriptor from a host page. For example, as shown in Fig. 5, the descriptor may be transferred from the host descriptor memory 310 to the endpoint descriptor handling engine 34. This may be done in a conventional manner, ie, the descriptor may be a regular descriptor as in conventional PCIe systems.

[0040] In conventional systems, this descriptor would then be provided to the endpoint DMA engine 33 to enable the endpoint DMA engine 33 to retrieve data corresponding to the descriptor from the host. In embodiments, the descriptor is instead provided to the adaptation block 32 at 71, as indicated by an arrow L in Fig. 5. In the case where the transfer adjustment block detects that the length of the data to be transferred, as indicated by the descriptor, does not match a multiple of a DWORD length, the transfer adjustment block adjusts the descriptor by adding dummy data, for example, dummy data bytes, so that the descriptor refers to data corresponding to a multiple of a DWORD length. In other words, a data pointer of the descriptor can be adjusted to the DWORD length.

[0041] At 72, the adapted descriptor is provided to the endpoint memory 31, which in this case serves as a receive descriptor memory, as indicated by an arrow M in Fig. 5. At 73, the DMA engine 33 then reads the adjusted descriptor, as indicated by an arrow N.

[0042] At 74, the DMA engine then transfers data corresponding to the adjusted descriptor, ie, data aligned to the DWORD length, from a host-side memory such as the host packet memory 39 to an endpoint memory such as the endpoint packet memory 30. This is indicated by an arrow O in Fig. 5 is displayed.

[0043] At 75, the descriptor is then updated to reflect the completed transfer, as indicated by an arrow P in Fig. 5 is displayed.

[0044] At 76, as indicated by an arrow Q in Fig. 5, the intelligent transmission adjustment block 32 uses this updated descriptor and adjusts the data length back to the original length (without dummy data).

[0045] At 77, this adapted descriptor is then passed to the endpoint descriptor handling engine, as indicated by an arrow R in Fig. 5. Based on the updated descriptor, the endpoint descriptor handling engine passes, as indicated by an arrow S in Fig. 5, then at 78 a corresponding data pointer corresponding to the original length is passed to subsequent processing stages that use the data. By adjusting the data length back to the original length, the DWORD alignment is transparent to subsequent processing stages; that is, due to the adjusted pointer passed at 76, subsequent stages, for example, only read data corresponding to the original (unadjusted) length from the endpoint packet memory 30. Therefore, the DWORD alignment is used only for transmission, essentially unnoticed by subsequent stages.

[0046] While in the Fig. 3 to 7 illustrate communication between a host device (e.g., a root complex) and an endpoint, the techniques disclosed herein may also be applicable to communication between PCIe endpoints.

[0047] To illustrate the effect of the techniques disclosed here, the Fig. 8 and Fig. 9 shows an example of the performance of comparison systems without DWORD alignment adjustment. Fig. 8 and Fig. Figure 9 shows the read performance of a PCIe device versus the packet size in bytes of packets to be transmitted. Curves 80 and 81 in Fig. 8 and curves 90 and 91 in Fig. 9 show results for two different comparison systems. As can be seen, the performance varies depending on the packet size (corresponding, for example, to the data unit sizes in Fig. 2), and for certain sizes, significant performance degradations occur. Fig. Figure 10 shows an example of an embodiment using DWORD-length-aligned data transfer as discussed here. Curves 100 and 101 again show results for two different systems. As can be seen, performance degradations, as in the Fig. 8 and Fig. 9 for certain packet sizes, essentially suppressed.

[0048] Conventional PCIe devices enable data to be written to and read from one device, e.g., an endpoint, to another device, e.g., a root complex, via an address map. In embodiments, as described above, this address map is substantially modified for aligned data transfer.

[0049] In a PCIe system, for example, a root complex checks the PCIe connections for the presence of devices, also known as endpoints, upon startup. Data transfer between these PCIe devices is based on transaction-layer packets. Data written or read over PCIe connections is therefore partitioned into transaction-layer packets. In some embodiments, the data is aligned to a multiple of a DWORD length, for example, by inserting dummy bytes, for example, before executing the transaction-layer packets.

[0050] The embodiments described above serve only as examples and are not to be interpreted as limiting.

Claims

[1] Peripheral Component Interconnect Express (PCIe) device comprising: a transmission adaptation device configured to adapt a length of data to be transmitted over a PCIe link to a multiple of a double word (DWORD) length, wherein aligning the length comprises modifying a descriptor. [2] The apparatus of claim 1, wherein the transmission adaptation means is configured to operate above a transaction layer of the PCIe device. [3] The device of claim 1 or 2, wherein the device comprises a PCIe endpoint device. [4] Apparatus according to any one of claims 1 to 3, wherein the transmission adaptation means is arranged to adapt the descriptor to an original length of the data after the transmission, and to provide the adapted descriptor for further processing of the data after the transmission. [5] The apparatus of any one of claims 1 to 4, wherein the apparatus comprises a direct memory access (DMA) device configured to perform the transfer based on the aligned length. [6] Apparatus according to any one of claims 1 to 5, wherein the transmission adjustment means is arranged to align the length by adding a number of dummy bytes to the data to be transmitted. [7] The apparatus of claim 6, wherein the number of dummy bytes is between 1 and 3. [8] The device of any one of claims 1 to 7, wherein the aligning comprises dynamically aligning the data to be transmitted over a PCIe link to a multiple of a double word (DWORD) length. [9] Device according to one of claims 1-8, wherein the transmission adaptation means is arranged to adapt the data to be transmitted which are present as data units having a length which does not correspond to an integer number of double words. [10] Method comprising: Aligning a length of data to be transferred to an integer multiple of a double word (DWORD) length into a Peripheral Component Interconnect Express device, wherein the aligning comprises modifying a descriptor. [11] The method of claim 10, wherein the method comprises transmitting the data based on the modified descriptor. [12] The method of claim 11, further comprising, after transmitting, adapting the modified descriptor to an original length of the data, and providing the adapted descriptor for processing the transmitted data. [13] The method of any one of claims 10 to 12, wherein the aligning comprises adding dummy bytes. [14] A method according to any one of claims 10 to 13, wherein aligning comprises modifying an address card. [15] The method of any one of claims 10 to 14, wherein the aligning comprises dynamically aligning the length of the data to be transmitted over a PCIe link to a multiple of a double word (DWORD) length.

Images