Method, device and electronic equipment for DPDK performance optimization based on multi-numa architecture processor
By virtualizing network interface card (NIC) functionality on multi-NUMA architecture processors and enabling transaction sorting mechanisms for IDs, the performance degradation of NIC DPDK under multi-NUMA architecture is solved, achieving multi-node parallel processing and performance improvement.
Patent Information
- Application Number
- CN202611230482.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-13
- Publication Date
- 2026-11-03
AI Technical Summary
On multi-NUMA architecture processors, existing technologies struggle to effectively utilize the resources of multiple NUMA nodes, leading to a decline in network card DPDK performance. This is especially true for high-speed network cards of 100G and above, where the computing cores and memory bandwidth within a single NUMA node cannot meet the line-speed forwarding requirements. Furthermore, simply distributing data flows across multiple NUMA nodes can cause queue congestion due to the strong ordering rules of the PCIe protocol.
By acquiring the hardware topology information of the target multi-NUMA architecture processor, the physical function of the virtualized network card is divided into multiple virtual functions, and a request identifier is configured for each virtual function. An ID-based transaction ordering mechanism is enabled, allowing virtual functions to be independently ordered and executed based on the request identifier. SR-IOV technology and IDO mechanism are used to achieve multi-node parallel processing.
It accelerates the performance of DPDK network cards on multi-NUMA architecture processors, avoids performance loss caused by cross-node access, eliminates queue blocking, and improves data transmission efficiency.
Smart Images
Figure CN123030518A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and electronic device for DPDK performance optimization based on multi-NUMA architecture processors. Background Technology
[0002] With the rapid development of cloud computing, artificial intelligence, and big data technologies, Ethernet card speeds have evolved from 10 Gigabit to 100G and even 400G. Meanwhile, constrained by the slowdown of Moore's Law, modern processors generally adopt Non-Uniform Memory Access (NUMA) architecture, integrating multiple memory controllers to improve memory access bandwidth. Taking processors using multi-chip modular packaging technology as an example, multiple compute dies and I / O dies are packaged in a single physical slot. Each compute die is configured with an independent memory controller and mounts local memory, forming an independent NUMA node. Under this architecture, memory access latency exhibits significant locality: accessing local die memory has low latency, while accessing remote die memory requires traversing the on-chip interconnect bus, significantly increasing latency. Therefore, when a PCIe device initiates a direct memory access operation, if the target address is not in the NUMA node where the network card resides, a remote access problem will be triggered, leading to performance degradation.
[0003] In the field of data plane processing, solutions based on the Data Plane Development Kit (DPDK), with their user-space polling mechanism and kernel bypass technology, have become the core benchmark for measuring system forwarding performance. The current mainstream optimization approach is to bind the DPDK's processing cores and memory resources to the NUMA node where the network interface card (NIC) resides, to avoid performance degradation caused by cross-node access. However, this approach can only utilize the resources of a single NUMA node. For high-speed NICs of 100G and above, the computing cores and memory bandwidth within a single NUMA node are insufficient to meet line-rate forwarding requirements. Simply distributing the NIC data stream across multiple NUMA nodes for processing necessitates adherence to the strong ordering rules of the PCIe protocol, meaning all device transactions under the same PCIe root port must be executed strictly in sequence. This leads to memory access requests to different NUMA nodes blocking each other, causing severe queue congestion and a sharp decline in performance.
[0004] Therefore, there is an urgent need for a solution to accelerate the performance of the network interface card (NIC) DPDK on a multi-NUMA architecture processor. Summary of the Invention
[0005] In view of this, this application provides a method, apparatus and electronic device for DPDK performance optimization based on a multi-NUMA architecture processor, which can accelerate the performance of network interface card DPDK on a multi-NUMA architecture processor.
[0006] In a first aspect, this application provides a method for DPDK performance optimization based on multi-NUMA architecture processors, including: Obtain the hardware topology information of the target multi-NUMA architecture processor, wherein the hardware topology information includes at least the NUMA node information where the network interface card is located and the resource information of each NUMA node in the multi-NUMA architecture; The physical functions of the network interface card are virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function; Based on the hardware topology information, configure the startup parameters of DPDK; and, Enable an ID-based transaction ordering mechanism so that virtual functions with different request identifiers are ordered and executed independently based on the request identifier.
[0007] Optionally, obtaining the hardware topology information of the target multi-NUMA architecture processor includes: The NUMA node information of the network card is obtained by reading the NUMA node information file in the network card device directory of the system file; and / or The resource information of each NUMA node in the multi-NUMA architecture is obtained by querying the system using a query command.
[0008] Optionally, after obtaining the hardware topology information of the target multi-NUMA architecture processor, the method further includes: Query the capability set information of the network interface card; If the capability set information contains single root I / O virtualization identification information, then it is determined that the network card supports single root I / O virtualization function, the physical function of the network card is virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function. If the capability set information does not contain single root I / O virtualization identification information, it is determined that the network card does not support single root I / O virtualization function, and the DPDK processing core and memory resources of the network card are bound to the NUMA node where the network card is located for processing.
[0009] Optionally, the step of virtualizing the physical function of the network interface card into at least two virtual functions and configuring a corresponding request identifier for each virtual function includes: Enable the Input / Output Memory Management Unit and Single Root I / O Virtualization in the Basic Input / Output System, and set the kernel parameters that support the Input / Output Memory Management Unit when the operating system starts. Write the target number value to the single root I / O virtualization virtual function quantity control interface of the target network card in the system file, and create the corresponding number of virtual functions. Each virtual function has an independent request identifier.
[0010] Optionally, configuring the DPDK startup parameters based on the hardware topology information includes: Based on the hardware topology information, core binding configuration information is generated; the core binding configuration information is used to limit the processing thread of each virtual function to the set of CPU cores specified by the NUMA node corresponding to the virtual function. Based on the hardware topology information, memory allocation configuration information is generated; the memory allocation configuration information is used to allocate independent memory resources for each NUMA node. Based on the hardware topology information, port mapping configuration information is generated; the port mapping configuration information is used to establish the mapping relationship between the request identifier of each virtual function and the corresponding target NUMA node.
[0011] Optionally, enabling the ID-based transaction ordering mechanism to allow virtual functions with different request identifiers to be ordered and executed independently based on the request identifier includes: Based on the processor's architecture type, a PCIe transaction ordering register in the processor is set by writing a predetermined value to a predetermined register address corresponding to the architecture type, thereby enabling an ID-based transaction ordering mechanism.
[0012] Optionally, the method further includes: When the processor is of the first architecture type, the predetermined register address is the first predetermined register address, and the predetermined value is the first predetermined value; wherein, the first architecture type is a processor architecture in which the PCIe controller is set on each computing die, and the predetermined register address corresponding to each computing die is the same.
[0013] Optionally, the method further includes: When the processor is of the second architecture type, the predetermined register address is a plurality of second predetermined register addresses, and each second predetermined register address corresponds to each computing die of the processor, and the predetermined value is written to each second predetermined register address respectively; wherein, the second architecture type is a processor architecture in which the PCIe controller is set on the IO die.
[0014] Optionally, the number of the second predetermined register addresses corresponds to the number of compute dies of the processor.
[0015] Secondly, this application also provides an apparatus for DPDK performance optimization based on a multi-NUMA architecture processor, comprising: The hardware topology detection module is used to obtain the hardware topology information of the target multi-NUMA architecture processor. The hardware topology information includes at least the NUMA node information where the network card is located and the resource information of each NUMA node in the multi-NUMA architecture. A virtual function creation module is used to virtualize the physical functions of the network interface card into at least two virtual functions, and configure a corresponding request identifier for each virtual function; The binding configuration module is used to configure the startup parameters of DPDK according to the hardware topology information; and, The IDO enabling module is used to enable an ID-based transaction ordering mechanism so that virtual functions with different request identifiers are ordered and executed independently based on the request identifiers.
[0016] Optionally, the hardware topology detection module includes: The node information acquisition submodule is used to obtain the NUMA node information of the network card by reading the NUMA node information file in the network card device directory of the system file system; and / or The core distribution acquisition submodule is used to obtain the resource information of each NUMA node in the multi-NUMA architecture through system query commands.
[0017] Optionally, following the hardware topology detection module, the following may also be included: A single-root I / O virtualization capability determination module is used to query the capability set information of the network interface card (NIC). If the capability set information contains single-root I / O virtualization identification information, it is determined that the NIC supports single-root I / O virtualization, and the physical function of the NIC is virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function. If the capability set information does not contain single-root I / O virtualization identification information, it is determined that the NIC does not support single-root I / O virtualization, and the DPDK processing core and memory resources of the NIC are bound to the NUMA node where the NIC is located for processing.
[0018] Optionally, the virtual function creation module includes: The Basic Input / Output System and Kernel Parameter Configuration Submodule is used to enable the Input / Output Memory Management Unit and Single Root I / O Virtualization in the Basic Input / Output System, and to set the kernel parameters that support the Input / Output Memory Management Unit when the operating system starts. The virtual function creation submodule is used to write the target number value to the single root I / O virtualization virtual function quantity control interface of the target network card in the system file, and create a corresponding number of virtual functions. Each virtual function has an independent request identifier.
[0019] Optionally, the binding configuration module includes: The core binding submodule is used to generate core binding configuration information based on the hardware topology information; the core binding configuration information is used to limit the processing thread of each virtual function to the set of CPU cores specified by the NUMA node corresponding to the virtual function. The memory allocation submodule is used to generate memory allocation configuration information based on the hardware topology information; the memory allocation configuration information is used to allocate independent memory resources for each NUMA node. The port configuration submodule is used to generate port mapping configuration information based on the hardware topology information; the port mapping configuration information is used to establish the mapping relationship between the request identifier of each virtual function and the corresponding target NUMA node.
[0020] Optionally, the IDO enabling module includes: The register configuration submodule is used to set the registers related to PCIe transaction ordering in the processor by writing predetermined values to predetermined register addresses corresponding to the processor architecture type, thereby enabling an ID-based transaction ordering mechanism.
[0021] Optionally, the IDO enabling module further includes: A first architecture type processing submodule is configured to, when the processor is of the first architecture type, have the predetermined register address as a first predetermined register address and the predetermined value as a first predetermined value; wherein, the first architecture type is a processor architecture in which the PCIe controller is set on each computing die and the predetermined register address corresponding to each computing die is the same.
[0022] Optionally, the IDO enabling module further includes: The second architecture type processing submodule is used to write the predetermined value to each of the second architecture types when the processor is of the second architecture type, wherein the predetermined register address is a plurality of second predetermined register addresses, and each second predetermined register address corresponds to a computing die of the processor; wherein the second architecture type is a processor architecture in which the PCIe controller is set on the IO die.
[0023] Thirdly, this application also provides an electronic device comprising: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed within the space enclosed by the housing, and the processor and memory are disposed on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing the DPDK performance optimization method based on a multi-NUMA architecture processor described in the first aspect above.
[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the DPDK performance optimization method based on a multi-NUMA architecture processor described in the first aspect.
[0025] This application achieves performance acceleration of the network interface card (NIC) DPDK on multi-NUMA architecture processors by acquiring hardware topology information of the target processor, including at least the NUMA node information where the network interface card (NIC) resides and the resource information of each NUMA node in the multi-NUMA architecture; virtualizing the physical function of the NIC into at least two virtual functions and configuring a corresponding request identifier for each virtual function; configuring startup parameters for the DPDK based on the hardware topology information; and enabling an ID-based transaction ordering mechanism so that the virtual functions with different request identifiers are ordered and executed independently based on the request identifiers. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 A schematic diagram of the method flow for DPDK performance optimization based on multi-NUMA architecture processors provided in this application embodiment; Figure 2 A schematic diagram illustrating the principle of changing the order preservation granularity from PCIe RP to BDF after enabling IDO in an embodiment of this application; Figure 3 This is a schematic diagram of a traditional single NUMA node binding scheme provided in an embodiment of this application; Figure 4A schematic diagram illustrating queue blocking caused by PCIe strong order preservation rules provided in this application embodiment; Figure 5 This is a schematic diagram illustrating the non-blocking parallel transmission of each virtual function bound to a different NUMA node after IDO is enabled, as provided in the embodiments of this application. Figure 6 A schematic diagram of the device structure for DPDK performance optimization based on a multi-NUMA architecture processor provided in this application embodiment; Figure 7 This is a schematic block diagram of the electronic device architecture provided in the embodiments of this application. Detailed Implementation
[0028] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0029] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0030] See Figure 1 , Figure 1 This is a schematic flowchart illustrating a method for optimizing DPDK performance based on a NUMA architecture processor, provided in an embodiment of this application. The method includes the following steps: S110. Obtain the hardware topology information of the target multi-NUMA architecture processor. The hardware topology information includes at least the NUMA node information where the network card is located and the resource information of each NUMA node in the multi-NUMA architecture.
[0031] In some embodiments, the NUMA node information of the network card is obtained by reading the NUMA node information file in the network card device directory of the system file system. Specifically, the NUMA node number of the network card can be read using the following command: cat / sys / class / net / eth0 / device / numa_node Replace eth0 with the actual network interface name (e.g., ens33, enp1s0f0, etc.). The number returned by this command is the NUMA node number where the network interface is located. For example, if the return value is 0, it means that the network interface is physically connected to NUMA node 0.
[0032] In some embodiments, resource information of each NUMA node in a multi-NUMA architecture is obtained through system query commands. This resource information includes at least the CPU core distribution information of each NUMA node. Specifically, the range of CPU core numbers contained in each NUMA node can be obtained using the following command: lscpu | grep -E 'NUMA|Core|CPU' The output of this command includes a list of CPU cores for each NUMA node, for example: NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 NUMA node2 CPU(s): 32-47 NUMA node3 CPU(s): 48-63 The above information is used in subsequent steps to bind and configure virtual functions with the CPU cores of each NUMA node.
[0033] See Figure 3 , Figure 3 This is a schematic diagram of a traditional single NUMA node binding scheme. (Example:) Figure 3 As shown, the processor includes four compute dies (CDD0, CDD1, CDD2, CDD3) and one IO die (IOD). CDD0 to CDD3 represent the first to fourth compute dies, respectively; IOD represents the input / output die; NIC represents the network interface card; UMC represents the unified memory controller; and DDR0 to DDR3 represent the DDR memory mounted on each compute die. Each compute die is connected to the IO die via an on-chip interconnect bus, and the network interface card is connected to the IO die. Each compute die is configured with a unified memory controller (UMC) and a corresponding DDR memory controller, mounting independent DDR memory (DDR0, DDR1, DDR2, DDR3), forming an independent NUMA node. In existing technologies, to avoid performance losses caused by cross-NUMA node access, the processing core and memory resources driven by the DPDK polling mode are typically bound to the NUMA node where the network interface card resides, ensuring that all data traffic flows within the same NUMA domain and uses local memory.
[0034] This solution can only utilize the resources of a single NUMA node. For high-speed network cards of 100G and above, the computing cores and memory bandwidth within a single NUMA node are insufficient to meet line-speed forwarding requirements. The hardware topology information obtained in this step provides the hardware basis for subsequently binding each virtual function to different NUMA nodes.
[0035] In some embodiments, after obtaining the hardware topology information of the target multi-NUMA architecture processor, the method further includes querying the capability set information of the network interface card (NIC) device to determine whether the NIC supports single root I / O virtualization functionality.
[0036] Specifically, the capability set information of a network interface card (NIC) device can be queried using the following command: lspci -s<NIC BDF number>-v | grep "SR-IOV" Wherein, <NIC BDF number> is the NIC bus-device-function number seen by the actual system (e.g., 41:00.0). If the query result contains Single Root I / O Virtualization (SR-IOV) identifier information, it is determined that the NIC supports SR-IOV function. Then, the physical function of the NIC is virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function.
[0037] If the query results do not contain SR-IOV identification information, the network card is determined to not support SR-IOV functionality. In this case, since the physical network card cannot be virtualized into multiple independent virtual functions, the multi-VF multi-node parallel acceleration scheme of this application cannot be implemented. Therefore, the DPDK processing core and memory resources of the network card are bound to the NUMA node where the network card resides for processing. Specifically, through the core list parameter and memory allocation parameter in the DPDK startup parameters, both the processing core and memory resources are specified to the resources corresponding to the NUMA node where the network card resides, and operation is carried out according to the traditional single NUMA node binding mode. This processing method ensures that the system can still run normally in hardware environments that do not support SR-IOV, enhancing the compatibility and robustness of this technical solution.
[0038] S120. Virtualize the physical function of the network card into at least two virtual functions, and configure a corresponding request identifier for each virtual function.
[0039] Specifically, the physical network interface card (PF) is virtualized into at least two virtual functions (VF) using SR-IOV technology. Each VF has an independent request identifier (i.e., PCIe bus-device-function number, BDF).
[0040] In some embodiments, the Input / Output Memory Management Unit (IOMMU) and SR-IOV functionality are first enabled in the Basic Input / Output System (BIOS), and kernel parameters supporting the IOMMU are set during operating system startup. For example, the iommu=pt kernel parameter is added to the GRUB boot configuration file to support IOMMU pass-through mode.
[0041] Then, write the target number value to the SR-IOV virtual function quantity control interface of the target network card in the system file (i.e., / sys / bus / pci / devices / <network card's BDF number> / sriov_numvfs) to create the corresponding number of virtual functions. Specifically, write the target number to the following path: echo <target number>> / sys / bus / pci / devices / <network card BDF number> / sriov_numvfs For example, if the target quantity is 4 and the network interface card (NIC) BDF number is 0000:41:00.0, then execute: echo 4> / sys / bus / pci / devices / 0000:41:00.0 / sriov_numvfs After successful execution, the system will create a specified number of virtual functions (VFs) for the physical network interface card (NIC). Each VF has an independent PCIe configuration space and a request identifier (BDF number). Assuming four VFs are generated, their PCIe addresses are 0000:41:00.1, 0000:41:00.2, 0000:41:00.3, and 0000:41:00.4. The BDF number of each VF is its unique request identifier, used to distinguish different data flows in the subsequent IDO mechanism.
[0042] S130. Configure the startup parameters of DPDK according to the hardware topology information.
[0043] Specifically, the startup parameters of DPDK are configured based on the hardware topology information obtained in step S110.
[0044] In some embodiments, core binding configuration information is generated based on the hardware topology information; the core binding configuration information is used to limit the processing thread of each virtual function to the set of CPU cores specified by the NUMA node corresponding to the virtual function.
[0045] Taking a processor with four NUMA nodes (node0~node3) as an example, assuming four Virtual Functions (VFs) have been created (BDF addresses are 0000:41:00.1, 0000:41:00.2, 0000:41:00.3, and 0000:41:00.4 respectively), the configuration strategy is to bind VF0 to node0, VF1 to node1, VF2 to node2, and VF3 to node3. In the DPDK boot parameters, the core list parameter (-l) specifies the list containing the CPU cores of each NUMA node. For example, if node0 contains CPU cores 0-15, node1 contains cores 16-31, node2 contains cores 32-47, and node3 contains cores 48-63, then the core list parameter can be configured as follows: -l 1-3,16-17,32-33,48-49 This configuration ensures that each NUMA node has a corresponding CPU core to participate in packet processing, thus realizing the function of core binding configuration information and limiting the processing threads of each VF to the set of CPU cores of its corresponding NUMA node.
[0046] In some embodiments, memory allocation configuration information is generated based on the hardware topology information; the memory allocation configuration information is used to allocate independent memory resources for each NUMA node.
[0047] In the DPDK startup parameters, memory is allocated to each NUMA node using the memory allocation parameter (--socket-mem). For example: --socket-mem 2048,2048,2048,2048 This configuration allocates 2GB of large page memory to each NUMA node, ensuring that each node has sufficient memory resources for packet buffering and processing. In other words, it implements the function of memory allocation configuration information, allocating independent memory resources to each NUMA node.
[0048] In some embodiments, port mapping configuration information is generated based on the hardware topology information; the port mapping configuration information is used to establish a mapping relationship between the request identifier of each virtual function and the corresponding target NUMA node.
[0049] In the DPDK startup parameters, the port affinity configuration parameter (--port-numa-config) explicitly specifies the mapping between each DPDK port and each NUMA node, binding each virtual function to a different NUMA node. For example: --port-numa-config=(0,0),(1,1),(2,2),(3,3) This configuration means that port0 uses the memory on node0, port1 uses the memory on node1, port2 uses the memory on node2, and port3 uses the memory on node3. Through this configuration, the data streams of each VF are bound to different NUMA nodes, achieving NUMA-local processing of the data streams and avoiding the performance overhead caused by cross-node memory access.
[0050] See Figure 4 , Figure 4 This diagram illustrates queue blocking caused by PCIe strong ordering rules. Figure 4 As shown, without SR-IOV enabled, the physical network interface card (NIC) has only one physical function (PF). All DMA write operations (W0, W1, W2, W3) destined for different NUMA nodes share the same PCIe link. Constrained by PCIe's strong ordering rules, write operations destined for different NUMA nodes must be executed sequentially, leading to queue congestion. In the diagram, CDD0 to CDD3 represent the first to fourth compute dies, IOD represents the input / output die, UMC represents the unified memory controller, DDR0 to DDR3 represent the DDR memory mounted on each compute die, W0, W1, W2, and W3 represent DMA write operations destined for the first to fourth DDR memory (DDR0~DDR3), and NIC represents the network interface card. The inability to parallelize write operations due to ordering rules results in performance degradation.
[0051] Simply distributing network interface card (NIC) data streams across multiple NUMA nodes without enabling IDO (Internet Deployment) requires adhering to the strong ordering rules of the PCIe protocol. This means all device transactions under the same PCIe root port must be executed strictly in sequence, causing memory access requests destined for different NUMA nodes to block each other, resulting in severe queue congestion and a sharp performance degradation. This step, by binding each Virtual Function (VF) to different NUMA nodes, lays the foundation for enabling IDO and achieving non-blocking parallel transmission later.
[0052] S140. Enable an ID-based transaction sorting mechanism so that virtual functions with different request identifiers are sorted and executed independently based on the request identifiers.
[0053] Enable ID-based transaction ordering (IDO) mechanism so that virtual functions with different request identifiers can be ordered and executed independently based on the request identifier.
[0054] The IDO (Independent Ordering) model is a feature added in PCIe V2.1. This model introduces the concept of a data stream: Transaction Layer Packets (TLPs) sent from the same data source belong to the same data stream, while TLPs sent from different data sources belong to different data streams. Devices with different BDFs (Browser Flow Dependencies) belong to different data sources. The IDO model allows TLPs from different data streams to not necessarily adhere to transaction ordering conventions. Without IDO enabled, all devices under the PCIe root port must comply with strong ordering rules; with IDO enabled, the ordering granularity is refined from the root port to individual BDFs, eliminating the dependency between data streams from different BDFs (i.e., different VFs).
[0055] See Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of changing the order preservation granularity from PCIe RP to BDF after enabling IDO. Figure 2 As shown, without IDO enabled, the ordering granularity is PCIe RP. W1 and W2 are dependent on each other; W2 must wait for W1 to complete before it can execute. Simultaneously, subsequent write transactions W3 and W4 are queued, causing a blockage. All VF's TLPs are queued uniformly at the PCIe root port, following strong ordering rules. With IDO enabled, the ordering granularity changes to BDF. Each transaction is routed to an independent data stream channel based on its BDF identifier. Transaction 1 and Transaction 2 are independent and can execute in parallel without blocking each other. Each VF's TLP is routed to its own independent data stream channel based on its BDF identifier. Different data streams do not need to follow ordering constraints and can be transmitted in parallel. In the diagram, IO DIE represents the processor's input / output die, PCIe RP represents the PCIe root port, and write transactions W3 and W4 represent subsequent transactions waiting to be executed.
[0056] In some embodiments, depending on the processor's architecture type, a PCIe transaction ordering-related register in the processor is set by writing a predetermined value to a predetermined register address corresponding to the architecture type to enable an ID-based transaction ordering mechanism. The processor provides the readsmn / writesmn tools to enable IDO by setting the SMN register.
[0057] In some embodiments, when the processor is a first architecture type, the predetermined register address is a first predetermined register address, and the predetermined value is a first predetermined value. The first architecture type is a processor architecture where the PCIe controller is located on each compute die, and the predetermined register address is the same for each compute die. For example, for a processor of the first architecture type (where the PCIe controller is located on each compute die, and the register address is the same for each compute die), the IDO mechanism is enabled by writing a first predetermined value (e.g., 0xa000) to the first predetermined register address (e.g., 0x11280198).
[0058] In some embodiments, when the processor is a second architecture type, the predetermined register address is a plurality of second predetermined register addresses, and each second predetermined register address corresponds one-to-one with each compute die of the processor, and the predetermined value is written to each second predetermined register address respectively. The second architecture type is a processor architecture where the PCIe controller is located on the IO die. For example, for a processor of the second architecture type (where the PCIe controller is located on the IO die, and the register addresses corresponding to each compute die are different), for a processor with four compute dies, a predetermined value (e.g., 0xa000) needs to be written to four register addresses (e.g., 0x11580198, 0x11680198, 0x11780198, 0x11880198) respectively to enable the IDO mechanism.
[0059] In some embodiments, the number of second predetermined register addresses corresponds to the number of computation dies of the processor. That is, for a processor with N computation dies, write operations need to be performed to N second predetermined register addresses, with each address corresponding to one computation die.
[0060] See Figure 5 , Figure 5 This diagram illustrates the non-blocking parallel transmission where each VF is bound to a different NUMA node after IDO is enabled. Figure 5As shown, with IDO enabled, the physical network interface card (NIC) is virtualized into multiple virtual functions (NIC0, NIC1, NIC2, NIC3) using SR-IOV technology. Each virtual function has an independent BDF identifier and is bound to a different NUMA node. After enabling IDO, the order preservation granularity is refined from PCIe RP to a single BDF. The DMA write operations of each VF (W0, W1, W2, W3) are diverted to independent data stream channels according to their respective BDF identifiers, allowing them to be executed in parallel without blocking each other. In the diagram, CDD0 to CDD3 represent the first to fourth compute dies, IOD represents the input / output die, UMC represents the unified memory controller, DDR0 to DDR3 represent the DDR memory mounted on each compute die, W0, W1, W2, and W3 represent DMA write operations sent to the first to fourth DDR memory, and NIC0, NIC1, NIC2, and NIC3 represent the first to fourth virtual functions virtualized by the same physical NIC.
[0061] After enabling IDO, the order preservation granularity changes to BDF. Each VF is bound to a NUMA node domain, and the TLP of each VF is diverted to its own independent data stream channel according to its BDF identifier. This mechanism enables multiple VFs to initiate DMA write operations to their respective bound NUMA nodes simultaneously without blocking each other, thereby eliminating the transmission bottleneck caused by PCIe's strong order preservation.
[0062] It is worth noting that although the PCIe protocol specification states that IDO should not be enabled in certain topologies (e.g., when there is closed-loop communication between two endpoint devices and the CPU, and transaction processing has causal dependencies), this risk does not exist in the DPDK model applied in this application. Specifically: First, in the DPDK architecture, the network interface card (NIC) only interacts with the root complex and does not need to concern itself with the data flow order between other devices; second, each VF based on SR-IOV is independent, and each VF only maintains its own exclusive data flow with the root complex, without creating transaction dependencies with other VFs. Based on this isolation characteristic, breaking the order preservation constraints between different BDFs will not cause logical anomalies. Therefore, enabling IDO in the DPDK scenario is safe and beneficial.
[0063] See Figure 6 This application also provides an apparatus for DPDK performance optimization based on multi-NUMA architecture processors, comprising: The hardware topology detection module 61 is used to obtain the hardware topology information of the target multi-NUMA architecture processor. The hardware topology information includes at least the NUMA node information where the network card is located and the resource information of each NUMA node in the multi-NUMA architecture. The virtual function creation module 62 is used to virtualize the physical function of the network card into at least two virtual functions, and configure a corresponding request identifier for each virtual function; The binding configuration module 63 is used to configure the startup parameters of DPDK according to the hardware topology information; and, IDO enabling module 64 is used to enable an ID-based transaction ordering mechanism so that virtual functions with different request identifiers are ordered and executed independently based on the request identifiers.
[0064] Optionally, the hardware topology detection module 61 includes: The node information acquisition submodule is used to obtain the NUMA node information of the network card by reading the NUMA node information file in the network card device directory of the system file system; and / or The core distribution acquisition submodule is used to obtain the resource information of each NUMA node in the multi-NUMA architecture through system query commands.
[0065] Optionally, after the hardware topology detection module 61, the following is also included: A single-root I / O virtualization capability determination module is used to query the capability set information of the network interface card (NIC). If the capability set information contains single-root I / O virtualization identification information, it is determined that the NIC supports single-root I / O virtualization, and the physical function of the NIC is virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function. If the capability set information does not contain single-root I / O virtualization identification information, it is determined that the NIC does not support single-root I / O virtualization, and the DPDK processing core and memory resources of the NIC are bound to the NUMA node where the NIC is located for processing.
[0066] Optionally, the virtual function creation module 62 includes: The Basic Input / Output System and Kernel Parameter Configuration Submodule is used to enable the Input / Output Memory Management Unit and Single Root I / O Virtualization in the Basic Input / Output System, and to set the kernel parameters that support the Input / Output Memory Management Unit when the operating system starts. The virtual function creation submodule is used to write the target number value to the single root I / O virtualization virtual function quantity control interface of the target network card in the system file, and create a corresponding number of virtual functions. Each virtual function has an independent request identifier.
[0067] Optionally, the binding configuration module 63 includes: The core binding submodule is used to generate core binding configuration information based on the hardware topology information; the core binding configuration information is used to limit the processing thread of each virtual function to the set of CPU cores specified by the NUMA node corresponding to the virtual function. The memory allocation submodule is used to generate memory allocation configuration information based on the hardware topology information; the memory allocation configuration information is used to allocate independent memory resources for each NUMA node. The port configuration submodule is used to generate port mapping configuration information based on the hardware topology information; the port mapping configuration information is used to establish the mapping relationship between the request identifier of each virtual function and the corresponding target NUMA node.
[0068] Optionally, the IDO enabling module 64 includes: The register configuration submodule is used to set the registers related to PCIe transaction ordering in the processor by writing predetermined values to predetermined register addresses corresponding to the processor architecture type, thereby enabling an ID-based transaction ordering mechanism.
[0069] Optionally, the IDO enabling module 64 further includes: A first architecture type processing submodule is configured to, when the processor is of the first architecture type, have the predetermined register address as a first predetermined register address and the predetermined value as a first predetermined value; wherein, the first architecture type is a processor architecture in which the PCIe controller is set on each computing die and the predetermined register address corresponding to each computing die is the same.
[0070] Optionally, the IDO enabling module 64 further includes: The second architecture type processing submodule is used to write the predetermined value to each of the second architecture types when the processor is of the second architecture type, wherein the predetermined register address is a plurality of second predetermined register addresses, and each second predetermined register address corresponds to a computing die of the processor; wherein the second architecture type is a processor architecture in which the PCIe controller is set on the IO die.
[0071] The DPDK performance optimization device based on a multi-NUMA architecture processor provided in this embodiment can be used to execute... Figure 1 The technical solutions of the method embodiments described above are similar in implementation principle and technical effect to those of the foregoing embodiments, and will not be repeated here. They can be referred to each other.
[0072] Figure 7 This is a schematic block diagram illustrating the architecture of one embodiment of the electronic device of this application; based on the same technical concept as the foregoing Embodiment 1, the electronic device provided in this application embodiment, such as... Figure 7 As shown, the steps of the method described in the embodiments of this application can be implemented.
[0073] The aforementioned electronic device may include: a housing 71, a processor 72, a memory 73, a circuit board 74, and a power supply circuit 75, wherein the circuit board 74 is disposed inside the space enclosed by the housing 71, and the processor 72 and the memory 73 are disposed on the circuit board 74; the power supply circuit 75 is used to supply power to the various circuits or devices of the aforementioned electronic device; the memory 73 is used to store executable program code; the processor 72 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 73, for executing the aforementioned method for DPDK performance optimization based on a multi-NUMA architecture processor.
[0074] For details on the specific execution process of the above steps by the processor 72 and the steps further executed by the processor 72 by running executable program code, please refer to the description of the foregoing embodiments of this application, which will not be repeated here.
[0075] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the DPDK performance optimization method based on a multi-NUMA architecture processor described in Embodiment 1 above.
[0076] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0077] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0078] For ease of description, if systems, servers, etc. are involved, they may be described separately as various units / modules based on their functions. Of course, in implementing this application, the functions of each unit / module can be implemented in one or more software and / or hardware.
[0079] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0080] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for performance optimization of DPDK based on multi-NUMA architecture processors, characterized in that, include: Obtain the hardware topology information of the target multi-NUMA architecture processor, wherein the hardware topology information includes at least the NUMA node information where the network interface card is located and the resource information of each NUMA node in the multi-NUMA architecture; The physical functions of the network interface card are virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function; Configure the startup parameters of DPDK based on the hardware topology information; as well as, Enable an ID-based transaction ordering mechanism so that virtual functions with different request identifiers are ordered and executed independently based on the request identifier.
2. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 1, characterized in that, The acquisition of the hardware topology information of the target multi-NUMA architecture processor includes: The NUMA node information of the network card is obtained by reading the NUMA node information file in the network card device directory of the system file; and / or The resource information of each NUMA node in the multi-NUMA architecture is obtained by querying the system using a query command.
3. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 2, characterized in that, After obtaining the hardware topology information of the target multi-NUMA architecture processor, the method further includes: Query the capability set information of the network interface card; If the capability set information contains single root I / O virtualization identification information, then it is determined that the network card supports single root I / O virtualization function, the physical function of the network card is virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function. If the capability set information does not contain single root I / O virtualization identification information, it is determined that the network card does not support single root I / O virtualization function, and the DPDK processing core and memory resources of the network card are bound to the NUMA node where the network card is located for processing.
4. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 1, characterized in that, The step of virtualizing the physical functions of the network interface card into at least two virtual functions, and configuring a corresponding request identifier for each virtual function, includes: Enable the Input / Output Memory Management Unit and Single Root I / O Virtualization in the Basic Input / Output System, and set the kernel parameters that support the Input / Output Memory Management Unit when the operating system starts. Write the target number value to the single root I / O virtualization virtual function quantity control interface of the target network card in the system file, and create the corresponding number of virtual functions. Each virtual function has an independent request identifier.
5. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 1, characterized in that, The step of configuring DPDK startup parameters based on the hardware topology information includes: Based on the hardware topology information, core binding configuration information is generated; the core binding configuration information is used to limit the processing thread of each virtual function to the set of CPU cores specified by the NUMA node corresponding to the virtual function. Based on the hardware topology information, memory allocation configuration information is generated; the memory allocation configuration information is used to allocate independent memory resources for each NUMA node. Based on the hardware topology information, port mapping configuration information is generated; the port mapping configuration information is used to establish the mapping relationship between the request identifier of each virtual function and the corresponding target NUMA node.
6. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 1, characterized in that, The enabling of the ID-based transaction ordering mechanism, so that virtual functions with different request identifiers are ordered and executed independently based on the request identifier, includes: Based on the processor's architecture type, a PCIe transaction ordering register in the processor is set by writing a predetermined value to a predetermined register address corresponding to the architecture type, thereby enabling an ID-based transaction ordering mechanism.
7. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 6, characterized in that, The method further includes: When the processor is of the first architecture type, the predetermined register address is the first predetermined register address, and the predetermined value is the first predetermined value; wherein, the first architecture type is a processor architecture in which the PCIe controller is set on each computing die, and the predetermined register address corresponding to each computing die is the same.
8. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 6, characterized in that, The method further includes: When the processor is of the second architecture type, the predetermined register address is a plurality of second predetermined register addresses, and each second predetermined register address corresponds to each computing die of the processor, and the predetermined value is written to each second predetermined register address respectively; wherein, the second architecture type is a processor architecture in which the PCIe controller is set on the IO die.
9. The method for DPDK performance optimization based on multi-NUMA architecture processors according to claim 8, characterized in that, The number of the second predetermined register addresses corresponds to the number of compute dies of the processor.
10. A device for DPDK performance optimization based on multi-NUMA architecture processors, characterized in that, include: The hardware topology detection module is used to obtain the hardware topology information of the target multi-NUMA architecture processor. The hardware topology information includes at least the NUMA node information where the network card is located and the resource information of each NUMA node in the multi-NUMA architecture. A virtual function creation module is used to virtualize the physical functions of the network interface card into at least two virtual functions, and configure a corresponding request identifier for each virtual function; The binding configuration module is used to configure the startup parameters of DPDK according to the hardware topology information. as well as, The IDO enabling module is used to enable an ID-based transaction ordering mechanism so that virtual functions with different request identifiers are ordered and executed independently based on the request identifiers.
11. The apparatus for DPDK performance optimization based on a multi-NUMA architecture processor according to claim 10, characterized in that, The hardware topology detection module includes: The node information acquisition submodule is used to obtain the NUMA node information of the network card by reading the NUMA node information file in the network card device directory of the system file system; and / or The core distribution acquisition submodule is used to obtain the resource information of each NUMA node in the multi-NUMA architecture through system query commands.
12. The apparatus for DPDK performance optimization based on a multi-NUMA architecture processor according to claim 11, characterized in that, Following the hardware topology detection module, the following is also included: A single-root I / O virtualization capability determination module is used to query the capability set information of the network interface card (NIC). If the capability set information contains single-root I / O virtualization identification information, it is determined that the NIC supports single-root I / O virtualization, and the physical function of the NIC is virtualized into at least two virtual functions, and a corresponding request identifier is configured for each virtual function. If the capability set information does not contain single-root I / O virtualization identification information, it is determined that the NIC does not support single-root I / O virtualization, and the DPDK processing core and memory resources of the NIC are bound to the NUMA node where the NIC is located for processing.
13. The apparatus for DPDK performance optimization based on a multi-NUMA architecture processor according to claim 10, characterized in that, The virtual function creation module includes: The Basic Input / Output System and Kernel Parameter Configuration Submodule is used to enable the Input / Output Memory Management Unit and Single Root I / O Virtualization in the Basic Input / Output System, and to set the kernel parameters that support the Input / Output Memory Management Unit when the operating system starts. The virtual function creation submodule is used to write the target number value to the single root I / O virtualization virtual function quantity control interface of the target network card in the system file, and create a corresponding number of virtual functions. Each virtual function has an independent request identifier.
14. The apparatus for DPDK performance optimization based on a multi-NUMA architecture processor according to claim 10, characterized in that, The binding configuration module includes: The core binding submodule is used to generate core binding configuration information based on the hardware topology information; the core binding configuration information is used to limit the processing thread of each virtual function to the set of CPU cores specified by the NUMA node corresponding to the virtual function. The memory allocation submodule is used to generate memory allocation configuration information based on the hardware topology information; the memory allocation configuration information is used to allocate independent memory resources for each NUMA node. The port configuration submodule is used to generate port mapping configuration information based on the hardware topology information; the port mapping configuration information is used to establish the mapping relationship between the request identifier of each virtual function and the corresponding target NUMA node.
15. The apparatus for DPDK performance optimization based on a multi-NUMA architecture processor according to claim 10, characterized in that, The IDO activation module includes: The register configuration submodule is used to set the registers related to PCIe transaction ordering in the processor by writing predetermined values to predetermined register addresses corresponding to the processor architecture type, thereby enabling an ID-based transaction ordering mechanism.
16. The apparatus for DPDK performance optimization based on a multi-NUMA architecture processor according to claim 15, characterized in that, The IDO activation module also includes: A first architecture type processing submodule is configured to, when the processor is of the first architecture type, have the predetermined register address as a first predetermined register address and the predetermined value as a first predetermined value; wherein, the first architecture type is a processor architecture in which the PCIe controller is set on each computing die and the predetermined register address corresponding to each computing die is the same.
17. The apparatus for DPDK performance optimization based on a multi-NUMA architecture processor according to claim 15, characterized in that, The IDO activation module also includes: The second architecture type processing submodule is used to write the predetermined value to each of the second architecture types when the processor is of the second architecture type, wherein the predetermined register address is a plurality of second predetermined register addresses, and each second predetermined register address corresponds to a computing die of the processor; wherein the second architecture type is a processor architecture in which the PCIe controller is set on the IO die.
18. An electronic device, characterized in that, The electronic device includes: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside the space enclosed by the housing, and the processor and memory are disposed on the circuit board; the power supply circuit is used to supply power to the various circuits or devices of the electronic device; the memory is used to store programs corresponding to executable program code, and is used to execute the DPDK performance optimization method based on a multi-NUMA architecture processor as described in any one of claims 1 to 9.
19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the DPDK performance optimization method based on a multi-NUMA architecture processor as described in any one of claims 1 to 9.