PCIe (Peripheral Component Interconnect Express) exchange chip with storage and network acceleration functions

Abstract

The invention belongs to the field of PCIe exchange chips, and relates to a PCIe exchange chip with storage and network acceleration functions, a data acceleration function module is added in the PCIe exchange chip, and the data acceleration function module comprises a built-in iEP, a control unit, a built-in CPU and firmware, a DMA and an address translation unit ATU. According to the invention, the load which is originally operated on the host CPU and is interactively related with the network or the storage equipment hardware can be unloaded to the PCIe exchange chip for execution, so that the host CPU can be ensured to more effectively operate the application program. After the PCIe exchange chip with the data acceleration function is used for connecting a network or a storage device to a host CPU, all computing power of the CPU can be used for running an application program, the operation capability of the CPU does not need to be spent on state switching between a user state and a kernel state of an operating system related to hardware, processing context switching is interrupted, and the operation efficiency is greatly improved. And the processing capability and the use efficiency of the CPU can be obviously improved on the aspects of data processing such as data processing, polling and the like.

Description

Technical Field

[0001] The invention relates to a PCIe switching chip with storage and network acceleration functions, belonging to the technical field of switching chips. Background Art

[0002] PCIe is a high-speed serial computer expansion bus, widely used in connecting computer hosts to peripheral devices. The PCIe controller on the host side is called RC, and the PCIe controller on the device side is called EP. Each EP device has a standard configuration space register, which indicates the basic properties and configurable information of the device. The BAR register identifies the address space resources such as registers and memory of the device, referred to as BAR space. After the PCIe physical layer establishes a link and is enumerated, the RC maps the configuration space and BAR space of the EP device to the address space of the host, and then the host driver can access and operate the device. Under the PCIe bus topology, each PCIe device has a unique BDF number.

[0003] PCIe uses TLP to transmit configuration requests and data requests. There are four types of TLP: memory request TLP is used to read and write device registers or memory, IO TLP is used to access IO space under certain architectures, configuration TLP is used to access the device's configuration space registers, and message TLP is used to implement user-defined messages.

[0004] PCIe is a point-to-point connection. When the host has insufficient RC controllers and needs to connect more devices, a PCIe switch chip is required. Figure 1 As shown in the figure, the PCIe switch chip usually has an upstream port USP, which is used to connect to the RC on the host side, and multiple downstream ports DSP, which are used to connect to EP devices. During PCIe enumeration, a lookup table is generated inside the PCIe switch chip, which records the bus number and address range of each port. After the enumeration is completed, the TLP coming in from each port can be routed to the corresponding port by looking up the table according to its different types, thereby achieving the purpose of connecting multiple EP devices to one RC, and also realizing communication between EP devices.

[0005] PCIe switching chips can usually be used to connect storage devices, network devices, and various data computing devices to the host.

[0006] At present, mainstream storage devices are divided into two categories according to their storage media: traditional mechanical disks and NVMe disks using flash memory chips. The read and write speed of NVMe disks is much better than that of traditional mechanical disks, and they require higher CPU processing performance.

[0007] Currently, the common mainstream network devices are IB network cards and Ethernet cards. IB network cards have their own RDMA acceleration technology. The network devices in this article are all Ethernet cards unless otherwise specified.

[0008] The CPU of a host is usually connected to an external network or storage device through a PCIe interface. If the CPU has fewer PCIe interfaces and needs to connect more devices, a PCIe switch chip is needed for expansion.

[0009] The network device, namely the network card, needs to be used with its driver. Generally, in Linux or other operating systems, the application runs in user state, while the driver runs in kernel state. When the application uses the driver to send and receive data, it must go through the state switching between the user state and kernel state of the operating system, and the data copying process between the user state and kernel state, which will bring a certain amount of additional load to the CPU. In addition, after the network card receives the data, it usually generates an interrupt to the CPU. The driver responds to the interrupt and processes the data. The operating system will also bring additional overhead when entering and exiting the interrupt context. At the same time, due to the repeated switching between the user state and kernel state, and the frequent entry and exit of the system interrupt, the system cache will be constantly refreshed, affecting the cache hit rate, and then affecting the system efficiency. As the network card speed increases, the above performance impact becomes more significant.

[0010] To solve the above problems, some special software solutions have been developed to replace traditional drivers. For example, Intel's DPDK is a software solution specifically designed to accelerate network data processing. It mainly uses the following technologies to improve network processing efficiency: mapping network card devices to the user state of the operating system; using polling to replace interrupts; fixing network processing threads to special CPU cores; using large page technology to improve cache hit rates; using lock-free queues to reduce resource competition, etc. Using DPDK can greatly improve the efficiency of network data processing.

[0011] The same situation also exists for storage devices, especially NVMe storage devices. That is, as the device speed increases, the large number of operating system user state and kernel state switches brought by traditional drivers, as well as frequent operations such as entering and exiting interrupt handlers, greatly affect data processing performance.

[0012] Correspondingly, in order to solve the performance problems of storage devices, there are also some specialized software solutions, such as Intel's SPDK, which uses a similar approach to DPDK, such as mapping the device to user mode and using polling instead of interrupts to improve system efficiency.

[0013] Traditional network or storage device drivers cause a decrease in the effective CPU utilization rate due to the large amount of switching between the operating system user state and kernel state, as well as frequent entry and exit of interrupt contexts.

[0014] Data acceleration processing software solutions represented by DPDK and SPDK can avoid the shortcomings of the working mode of traditional drivers and greatly improve the processing efficiency of the system. However, there are also obvious disadvantages: dedicated CPU cores are required to process data. If a system is connected to multiple networks or storage devices, multiple CPU cores must be occupied. In addition, when there is no data to be processed, these CPU cores are still idling and cannot be used to process other tasks. This will cause a waste of CPU utilization. Usually, the server's CPU should be used to run more valuable applications, rather than being occupied or idling by hardware devices.

[0015] In short, existing solutions will have additional waste of CPU utilization, resulting in low effective CPU utilization.

[0016] Based on this, the present invention is proposed. Summary of the invention

[0017] The inventor of the present application has conducted multiple tests and analyses and found that after using a PCIe switch chip to connect a network or storage device to a host CPU, in order to reduce the extra load caused by the CPU processing hardware devices, the traditional driver or data acceleration processing software running on the CPU can be offloaded to the PCIe switch chip. By adding specialized hardware and software to the PCIe switch chip, the PCIe switch chip has the acceleration function of the network or storage device.

[0018] After the network or storage device is connected to the CPU, the state switching between the user state and the kernel state of the operating system brought about by the traditional device driver, the frequent entry and exit, and the interrupt processing context will bring about certain performance losses; and the data processing acceleration technology that maps the hardware device to the user state and mainly uses software polling will cause the dedicated CPU core to be occupied, even when there is no data to be processed. Especially after using the PCIe switching chip to connect more network or storage devices, the CPU occupation causes the performance loss of the application running on the CPU to be more obvious, so the goal of the present invention is to solve the above problems. Make the PCIe switching chip have the data processing and acceleration capabilities of the network and storage, so that the CPU's computing power can be more effectively focused on the user's application, rather than being overly occupied by the operation of the hardware device.

[0019] The present invention provides a PCIe switching chip with storage and network acceleration functions, and the specific technical solution is as follows: A PCIe switching chip with storage and network acceleration functions, by adding a data acceleration function module in the PCIe switching chip, the data acceleration function module includes a built-in iEP, a control unit, a built-in CPU and firmware, a DMA and an address translation unit ATU; In the PCIe switch chip, an EP is connected to the DSP through the DSP integrated EP, which is called built-in iEP; The control unit includes a status register, a page table and a pointer register; The status register displays the information of the internal CPU on the network or storage device to the host CPU, so that the host CPU knows whether the device can perform read and write operations; The page table is a conversion table of virtual addresses and physical addresses used in the host, which records the conversion relationship between the virtual addresses and physical addresses of the memory area where the data to be read and written is located, for the address translation unit ATU to search and use; The pointer register is used to indicate the starting and ending positions of the data area requested in the host memory and the head and tail pointers of the data, and also includes the read and write instructions of the data; after the host of the data performs a read or write operation on the data, the head or tail pointer register is updated, and the DMA continues to copy the data according to the read or write instruction; The data read from or to be written to the storage or network device is cached in the PCIe switch chip, and DMA completes the data transmission between the PCIe switch chip cache and the host memory; When DMA accesses the host memory, it must go through the address translation unit ATU. The address translation unit ATU searches the pre-written page table in the control unit, converts the virtual address into a physical address, and then performs memory read and write access; The firmware can enumerate the network or storage device connected to the switch chip, and the built-in CPU replaces the external host CPU to directly manage the external network or storage device.

[0020] As a further improvement, the information of the storage device includes the type, name, and current operating status of the device.

[0021] As a further improvement, the built-in iEP is enumerated by the host CPU, and the built-in iEP maps registers and memory resources in the control unit to the address space of the host CPU through the BAR space, so that the host CPU can access the control unit.

[0022] As a further improvement, the use process of the data acceleration function module includes the following steps: Step S1, the system is connected to a network or storage device, and after startup, the firmware on the built-in CPU runs and enumerates the device; Step S2, the firmware configures the status register, indicating that its function and the device are ready; Step S3: The program on the host CPU enumerates the built-in iEP device and reads its function and status; Step S4: the program on the host allocates address space in the host memory and locks it, which means that the memory will not generate page switching behavior, that is, the correspondence between the virtual address and the physical address will remain unchanged; Step S5, the host writes the allocated memory start and end positions into the pointer register of the control unit; Step S6: The host writes the conversion table of the virtual address and physical address of the used memory space into the page table of the control unit; Step S7: Before the data read and write operation starts, the host sets the head and tail pointers of the pointer register to the starting address of the memory; Step S8: The host sends a read data instruction to the pointer register. or, The host prepares the data, updates the tail pointer of the data to the pointer register, and issues a write data instruction; Step S9: for the host's data read instruction, the internal firmware starts, polls the device data, and stores the data in the cache; or, For the host's write data instruction, the internal firmware starts, and when there is data in the cache, it writes it to the device; Step S10, DMA moves data from the PCIe switch chip cache to the host memory, and updates the tail pointer of the data at the same time; or, DMA moves data from the host memory to the switch chip cache and updates the data head pointer at the same time; Step S11, the host application reads data from the host memory and updates the data head pointer; or, The data in the host memory is written out, and other data can be written in and the data tail pointer is updated; Step S12: The above steps 8 to 11 are repeated until all data are read and written. In addition, data reading and writing are performed in parallel.

[0023] As a further improvement, a data acceleration function module is deployed above each DSP port and below the bus in the PCIe switch chip.

[0024] As a further improvement, a data acceleration function module is deployed under the USP port, below the bus in the PCIe switch chip.

[0025] Beneficial effects of the present invention: 1. The present invention adds a data acceleration function module on the basis of a conventional PCIe switching chip, and the data acceleration function module includes a built-in iEP, a control unit, a built-in CPU and firmware, a DMA and an address translation unit ATU, so that the load related to the interaction with the network or storage device hardware originally running on the host CPU can be unloaded to the PCIe switching chip for execution, ensuring that the host CPU can run the application more efficiently. Existing PCIe switching chips do not have this function.

[0026] 2. After using the PCIe switching chip with data acceleration function provided by the present invention to connect the network or storage device to the host CPU, all the computing power of the CPU can be used to run the application program. There is no need to spend the CPU's computing power on hardware-related operating system user state and kernel state state switching, interrupt processing context switching, or polling and other data processing, which can significantly improve the CPU's processing power and utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of connecting multiple devices to a host CPU using a PCIe switch chip; Figure 2 This is a block diagram of the data acceleration function module built into the PCIe switch chip; Figure 3 This is the data acceleration function module deployment diagram shown in Method 1; Figure 4 This is the data acceleration function module deployment diagram shown in Method 2. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] Definitions of Abbreviations and Key Terms PCIe: Peripheral Component Interconnect express, a high-speed serial bus interface; RC: Root Complex, PCIe root complex; EP: Endpoint, PCIe endpoint device; BAR: Base Address Register, which specifies the register of the PCIe device memory resources; BDF: Bus Device Function, PCIe device unique ID; TLP: Transaction Layer Packets, the data packet transmission format specified by the PCIe protocol; USP: Upstream Port, upstream port of PCIe switch chip, used to connect RC devices; DSP: Downstream Port, the downstream port of the PCIe switch chip, used to connect to the EP device; Enumeration: The host on the PCIe RC side discovers the PCIe bus topology and maps its devices to the host address space; iEP: DSP integrated EP, refers to the EP integrated inside the chip; CPU: Central Processing Unit, the core of computer system calculation and control; Flash Memory: Flash Memory is an electronic memory that can be erased or written many times, which is different from a mechanical disk; NVMe: Non Volatile Memory express, a non-volatile memory specification that uses flash memory chips; IB: InfiniBand, a computer network communication standard for high-performance computing; RDMA: Remote Direct Memory Access, a mechanism for directly accessing remote host memory; Ethernet: Ethernet, a widely used computer local area network technology; Kernel state: refers to the state with higher permissions in the operating system, which can directly access more hardware resources; User state: refers to the state with lower permissions in the operating system. The resources that can be accessed are limited, and other resources are accessed through kernel state; DPDK: Data Plane Development Kit, a software solution for fast processing of network packets; SPDK: Storage Performance Development Kit, a fast data processing solution for the storage field; RTOS: Real Time Operation System, real-time operating system; DMA: Direct Memory Access, direct memory access; ATU: Adress Translation Unit, address translation unit, used for conversion between virtual address and physical address; IO TLP: Transaction Layer Packet, a type of message used to transmit input / output requests in the PCIe protocol. IO TLP is mainly used to process read and write operations of input / output devices; IO space: an address space independent of the memory space, mainly used to access registers of external devices; Linux: A free to use and free to distribute operating system. Example 1

[0030] The PCIe switching chip with storage and network acceleration functions described in the present invention adds a data acceleration function module in the PCIe switching chip, wherein the data acceleration function module includes a built-in iEP, a control unit, a built-in CPU and firmware, a DMA and an address translation unit ATU, such as Figure 2 shown. Figure 2 The dual-line path in the middle is the high-speed data channel of the data acceleration function module built into the PCIe switch chip, which can efficiently transmit data between storage or network devices and host memory without occupying host CPU resources.

[0031] 1. Built-in iEP In the PCIe architecture, a functional unit needs to be connected to an EP to implement PCIe data packet routing; and in the PCIe switch chip, an EP needs to be connected to a DSP to comply with the specification, which is implemented in practice by integrating the EP with the DSP, which is called built-in iEP in the present invention.

[0032] This solution uses a built-in iEP, which can be enumerated by the host CPU. The built-in iEP maps the registers, memory and other resources in the control unit to the address space of the host CPU through the BAR space, so that the host CPU can access the control unit and then control and use the data acceleration processing function.

[0033] EP devices such as network or storage that were originally connected through PCIe switching chips are replaced by this iEP device.

[0034] 2. Control unit The control unit mainly includes three parts: status register, page table and pointer register.

[0035] The status register can display the internal CPU's information about the network or storage device (such as the device's type, name, current operating status, etc.) to the host CPU, so that the host CPU can know whether the device can be read or written.

[0036] The page table is a conversion table of virtual addresses and physical addresses used in the host. It only records the conversion relationship between the virtual address and the physical address of the memory area where the data to be read or written is located, for the address translation unit ATU to search and use.

[0037] The pointer register is used to indicate the start and end positions of the data area requested in the host memory, as well as the head and tail pointers of the data, and also includes the read and write instructions of the data. After the host of the data reads or writes the data, it updates the head or tail pointer register, and the DMA can continue to copy the data according to the read or write instructions.

[0038] 3. DMA and address translation unit ATU Data read from or to be written to storage or network devices is cached in the PCIe switch chip, and DMA completes the data transfer between the PCIe switch chip cache and the host memory. To improve efficiency, user-mode applications can directly use user-mode virtual addresses for DMA.

[0039] When DMA accesses the host memory, it must pass through the address translation unit ATU, which searches the pre-written page table in the control unit, converts the virtual address into a physical address, and then performs memory read and write access. The use of DMA virtual addresses allows the PCIe switch chip to directly transfer data between the virtual addresses of the device and the application, bypassing the host CPU's operating system, greatly improving transmission efficiency.

[0040] 4. Built-in CPU and firmware In order to flexibly adapt to various network or storage devices, a solution with a built-in CPU and its running firmware is adopted. The firmware can enumerate the network or storage device connected to the switch chip, and the built-in CPU replaces the external host CPU to directly manage the external network or storage device.

[0041] The internal firmware uses an efficient real-time operating system to eliminate the extra overhead caused by the switching between user mode and kernel mode of the traditional operating system; the polling method is used to send and receive data from the network or storage device, which can eliminate the performance loss caused by interrupt context switching. After the data is temporarily stored in the cache area, DMA is used to transfer it to and from the host memory. The overall effect is equivalent to the efficient data processing program of the firmware on the built-in CPU, which can access the host memory and perform efficient data transmission between the external device and the host memory. Example 2

[0042] against Figure 2 The usage process of each module in the data acceleration function module is as follows: Step S1: The system is connected to a network or storage device. After power-on, the firmware on the built-in CPU runs and enumerates the device. Step S2, the firmware configures the status register, indicating that its function and the device are ready; Step S3: The program on the host CPU enumerates the built-in iEP device and reads its function and status; Step S4: the program on the host allocates address space in the host memory and locks it, which means that the memory will not generate page switching behavior, that is, the correspondence between the virtual address and the physical address will remain unchanged; Step S5, the host writes the allocated memory start and end positions into the pointer register of the control unit; Step S6: The host writes the conversion table of the virtual address and physical address of the used memory space into the page table of the control unit; Step S7: Before the data read and write operation starts, the host sets the head and tail pointers of the pointer register to the starting address of the memory; Step S8: The host sends a read data instruction to the pointer register. or, The host prepares the data, updates the tail pointer of the data to the pointer register, and issues a write data instruction; Step S9: for the host's data read instruction, the internal firmware starts, polls the device data, and stores the data in the cache; or, For the host's write data instruction, the internal firmware starts, and when there is data in the cache, it writes it to the device; Step S10, DMA moves data from the PCIe switch chip cache to the host memory, and updates the tail pointer of the data at the same time; or, DMA moves data from the host memory to the switch chip cache and updates the data head pointer at the same time; Step S11, the host application reads data from the host memory and updates the data head pointer; or, The data in the host memory is written out, and other data can be written in and the data tail pointer is updated; Step S12: The above steps 8 to 11 can be repeated until all data are read and written. In addition, data reading and writing can be performed in parallel.

[0043] After the above process, the host application only needs to read and write its own local host memory and update the relevant pointer register to read and write data from the device. Other processing processes related to the hardware device are completed by the PCIe switch chip, which greatly improves the efficiency of the host CPU running the application. Example 3

[0044] Deployment method: Figure 2 The data acceleration function module shown is illustrated by taking only one DSP port, that is, one device, as an example.

[0045] The actual PCIe switching chip has multiple DSP ports, and there are two deployment methods for the data acceleration function module.

[0046] like Figure 3 The first method presented deploys a data acceleration function module above each DSP port and below the bus in the PCIe switch chip.

[0047] like Figure 4 The second method shown is to deploy a data acceleration function module under the USP port, below the bus in the PCIe switch chip. In this case, the built-in CPU needs to have the same number of cores as the DSP ports, and other hardware units also need to increase their processing capabilities accordingly.

[0048] Regardless of which solution is adopted, the data acceleration function module can be turned off individually as needed, restoring the PCIe switching chip to its original function.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A PCIe switch chip with storage and network acceleration functions, characterized in that: By adding a data acceleration function module in the PCIe switching chip, the data acceleration function module includes a built-in iEP, a control unit, a built-in CPU and firmware, a DMA and an address translation unit ATU; In the PCIe switch chip, an EP is connected to the DSP through the DSP integrated EP, which is called built-in iEP; The control unit includes a status register, a page table and a pointer register; The status register displays the information of the internal CPU on the network or storage device to the host CPU, so that the host CPU knows whether the device can perform read and write operations; The page table is a conversion table of virtual addresses and physical addresses used in the host, which records the conversion relationship between the virtual addresses and physical addresses of the memory area where the data to be read and written is located, for the address translation unit ATU to search and use; The pointer register is used to indicate the starting and ending positions of the data area requested in the host memory and the head and tail pointers of the data, and also includes the read and write instructions of the data; after the host of the data performs a read or write operation on the data, the head or tail pointer register is updated, and the DMA continues to copy the data according to the read or write instruction; The data read from or to be written to the storage or network device is cached in the PCIe switch chip, and DMA completes the data transmission between the PCIe switch chip cache and the host memory; When DMA accesses the host memory, it must go through the address translation unit ATU. The address translation unit ATU searches the pre-written page table in the control unit, converts the virtual address into a physical address, and then performs memory read and write access; The firmware can enumerate the network or storage device connected to the switch chip, and the built-in CPU replaces the external host CPU to directly manage the external network or storage device.

2. The PCIe switch chip with storage and network acceleration functions according to claim 1, characterized in that: The information of the storage device includes the type, name, and current operating status of the device.

3. The PCIe switch chip with storage and network acceleration functions according to claim 1, characterized in that: The built-in iEP is enumerated by the host CPU, and the built-in iEP maps registers and memory resources in the control unit to the address space of the host CPU through the BAR space, so that the host CPU can access the control unit.

4. The PCIe switch chip with storage and network acceleration functions according to claim 1, characterized in that: The use process of the data acceleration function module includes the following steps: Step S1, the system is connected to a network or storage device, and after startup, the firmware on the built-in CPU runs and enumerates the device; Step S2, the firmware configures the status register, indicating that its function and the device are ready; Step S3: The program on the host CPU enumerates the built-in iEP device and reads its function and status; Step S4: the program on the host allocates address space in the host memory and locks it, which means that the memory will not generate page switching behavior, that is, the correspondence between the virtual address and the physical address will remain unchanged; Step S5, the host writes the allocated memory start and end positions into the pointer register of the control unit; Step S6: The host writes the conversion table of the virtual address and physical address of the used memory space into the page table of the control unit; Step S7: Before the data read and write operation starts, the host sets the head and tail pointers of the pointer register to the starting address of the memory; Step S8: The host sends a read data instruction to the pointer register. or, The host prepares the data, updates the tail pointer of the data to the pointer register, and issues a write data instruction; Step S9: for the host's data read instruction, the internal firmware starts, polls the device data, and stores the data in the cache; or, For the host's write data instruction, the internal firmware starts, and when there is data in the cache, it writes it to the device; Step S10, DMA moves data from the PCIe switch chip cache to the host memory, and updates the tail pointer of the data at the same time; or, DMA moves data from the host memory to the switch chip cache and updates the data head pointer at the same time; Step S11, the host application reads data from the host memory and updates the data head pointer; or, The data in the host memory is written out, and other data can be written in and the data tail pointer is updated; Step S12: The above steps 8 to 11 are repeated until all data are read and written. In addition, data reading and writing are performed in parallel.

5. The PCIe switch chip with storage and network acceleration functions according to claim 1, characterized in that: A data acceleration function module is deployed above each DSP port and below the bus in the PCIe switch chip.

6. The PCIe switch chip with storage and network acceleration functions according to claim 1, characterized in that: A data acceleration function module is deployed under the USP port, below the bus in the PCIe switch chip.

Images