Performance optimization method of NVMe storage card based on tengyun S2500 dual-channel server architecture

Abstract

The application relates to the technical field of computer science, in particular to an NVMe storage card performance optimization method based on a Tengyun S2500 dual-path server architecture, wherein the Tengyun S2500 dual-path server architecture comprises a main path CPU and a slave path CPU; the main path CPU and the slave path CPU are respectively connected with an NVMe storage card; the NVMe storage card comprises a plurality of hardware queues; the method is characterized in that the method comprises the following steps: S1, controlling the number of hardware queues of the NVMe storage card, so that the number of hardware queues of the NVMe storage card does not exceed the core number of the main path CPU or the slave path CPU connected with the NVMe storage card; and S2, controlling the interruption of the hardware queues of the NVMe storage card to be processed on the main path CPU or the slave path CPU connected with the NVMe storage card. The optimization method can avoid the interruption cross-path accumulation problem caused by the interruption processing of the NVMe storage card due to cross paths.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer science, and in particular to an NVMe storage card performance optimization method based on a Fengyun S2500 dual-path server architecture. BACKGROUND

[0002] In recent years, with the rapid development of domestic CPU technology, CPUs supporting dual-path and multi-path interconnection architecture have gradually entered the market. In this rapid development process, the interconnection channel between the physical CPUs has become a major performance bottleneck, and problems such as cross-path interrupt processing have gradually emerged.

[0003] The domestic Fengyun S2500 processor provides a multi-port high-speed low-latency interconnection path. In terms of multi-path expansion capability, the Fengyun S2500 supports dual-path, 4-path, and 8-path direct connection. However, due to the adaptation problems of the Fengyun S2500 architecture and kernel NVMe driver software and hardware, there are still certain performance problems when applying high-performance NVMe storage cards to Fengyun S2500 servers, mainly in:

[0004] To better describe the problem, take Figure 1 For example, a high-performance NVMe storage card is implemented based on multiple hardware queues, that is, there may be at most one hardware queue corresponding to each CPU core, such as on a dual-path Fengyun S2500, there may be at most 128 hardware queues (Fengyun S2500 processor is a 64-core architecture, and dual-path is a 128-core architecture). However, due to the special nature of cross-path interrupt processing in the Fengyun S2500 dual-path architecture, the master CPU and the slave CPU are interconnected, and the interrupt processing of the NVMe storage card connected to the master CPU can be processed through the interconnection channel into the slave CPU, and vice versa. This interrupt processing method may cause interrupt accumulation problems, further more likely to cause interconnection cross-path data accumulation problems, and even cause downtime.

[0005] Therefore, based on the above reasons, an optimization method is needed to specifically solve the problems of NVMe storage cards on dual-path S2500. SUMMARY

[0006] Technical problem to be solved

[0007] In view of the above shortcomings of the prior art, the present application provides an NVMe storage card performance optimization method based on a Fengyun S2500 dual-path server architecture, which can avoid the interrupt cross-path accumulation problem caused by the interrupt processing of the NVMe storage card due to cross-path.

[0008] Technical solution

[0009] To achieve the above object, the present application is realized by the following technical solutions:

[0010] The application provides a performance optimization method of an NVMe storage card based on a Tengyun S2500 dual-path server architecture, wherein the Tengyun S2500 dual-path server architecture comprises a master CPU and a slave CPU, the master CPU and the slave CPU are respectively connected with an NVMe storage card, the NVMe storage card comprises a plurality of hardware queues, and the method comprises the following steps:

[0011] S1, the number of hardware queues of the NVMe storage card is controlled, so that the number of hardware queues of the NVMe storage card does not exceed the core number of the master CPU or the slave CPU connected therewith;

[0012] S2, the interrupt of the hardware queue of the NVMe storage card is controlled to be processed on the master CPU or the slave CPU connected therewith.

[0013] Further, step S2 specifically comprises:

[0014] The node number of the numa node where the NVMe storage card is located is obtained, the node number corresponds to the PCI slot number of the master CPU or the slave CPU, and the master CPU or the slave CPU connected with the NVMe storage card is obtained;

[0015] The affinity of the hardware queue interrupt of the NVMe storage card is set, so that the interrupt of the hardware queue of the NVMe storage card is processed on the master CPU or the slave CPU connected therewith.

[0016] Further, the node number of the numa node where the NVMe storage card is located is obtained by calling the pci_irq_get_node function.

[0017] Further, when the node number is 0-7, it indicates that the NVMe storage card is connected with the master CPU, and when the node number is 8-15, it indicates that the NVMe storage card is connected with the slave CPU.

[0018] Further, the core number of the master CPU and the slave CPU is 64.

[0019] Further, the number of hardware queues of the NVMe storage card connected with the master CPU is 64, and the number of hardware queues of the NVMe storage card connected with the slave CPU is 32.

[0020] Based on the same inventive concept, the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method.

[0021] Based on the same inventive concept, the present application further provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method.

[0022] Advantages

[0023] The technical scheme provided by the present application has the following beneficial effects compared with the known prior art: the NVMe storage card performance optimization method based on the Tengyun S2500 dual-path server architecture provided by the present application optimizes the number of hardware queues of the NVMe storage card according to the characteristics of the Tengyun S2500 dual-path server architecture, and optimizes the interrupt affinity according to whether the NVMe storage card is actually located in the master CPU or the slave CPU, so as to avoid the problem of interrupt cross-path accumulation processing caused by interrupt cross-path, and achieve the purpose of optimizing the performance of the NVMe storage card. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the drawings needed in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0025] Figure 1 Prior art schematic diagram;

[0026] Figure 2 The method for optimizing the performance of the NVMe storage card based on the Tengyun S2500 dual-path server architecture provided by an embodiment of the present application is shown in the flowchart.

[0027] Figure 3 The effect diagram of the method for optimizing the performance of the NVMe storage card based on the Tengyun S2500 dual-path server architecture provided by an embodiment of the present application is shown in the flowchart.

[0028] Figure 4 The flowchart of the method for optimizing the performance of the NVMe storage card based on the Tengyun S2500 dual-path server architecture provided by an embodiment of the present application is shown in the flowchart. DETAILED DESCRIPTION

[0029] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0030] First, the technical terms used in the present application are explained:

[0031] Dual-server: The dual-server refers to a server having two physical CPUs, and of course, there are two CPU card slots on the corresponding server motherboard. Dual is not dual-core, and dual-core refers to installing two CPU cores on one CPU, so dual-core or multi-core CPUs still look like one CPU, and multi-way CPUs have several visible physical CPUs.

[0032] Interrupt crossway: The interrupt crossway refers to the interrupt generated by the physical CPU on the server not being processed on the physical CPU, but being processed on other physical CPUs through an interconnection channel (used for connecting and communicating between two physical CPUs to realize a dual-server architecture), that is, the interrupt generated on the A physical CPU is processed on the B physical CPU, and the interrupt generated on the B physical CPU is processed on the A physical CPU.

[0033] NVMe: NVMe stands for Non-Volatile Memory Express, a non-volatile memory standard, which is a kind of protocol similar to AHCI established on the M.2 interface, and is a protocol specially designed for flash memory storage. NVMe is a kind of specification for SSDs using PCI-E channels, which fully utilizes the low latency and parallelism of PCI-E SSDs from the beginning of its design, as well as the parallelism of contemporary processors, platforms and applications. The parallelism of SSDs can be fully utilized by the hardware and software of the host, and compared with the AHCI standard, the NVMe standard can bring performance improvement in many aspects.

[0034] Hardware queue: The hardware queue refers to the NVMe hardware queue, which is a hardware unit physically formed on the NVMe card. High-performance NVMe cards generally have multiple hardware queues. On a multi-core architecture, each hardware queue is bound to a different CPU core, and each hardware queue has a corresponding PCI MSIx interrupt number. Hardware multi-queue is an important method for improving performance of NVMe. With the help of hardware multi-queue technology, NVMe realizes the allocation of different queues according to tasks, scheduling priorities and CPU cores to complete high-performance storage functions.

[0035] ReferenceFigure 2 An embodiment of the present application provides a performance optimization method of an NVMe storage card based on a cloud S2500 dual-path server architecture, the cloud S2500 dual-path server architecture comprising a main path CPU and a slave path CPU, the main path CPU and the slave path CPU being connected with an NVMe storage card respectively, the NVMe storage card comprising a plurality of hardware queues, and the method comprising the following steps:

[0036] S1, the number of hardware queues of the NVMe storage card is controlled, so that the number of hardware queues of the NVMe storage card does not exceed the core number of the main path CPU or the slave path CPU connected therewith;

[0037] S2, the interrupt of the hardware queue of the NVMe storage card is controlled to be processed on the main path CPU or the slave path CPU connected therewith.

[0038] In the embodiment, step S2 specifically comprises:

[0039] the node number of a numa node where the NVMe storage card is located is acquired, the node number corresponding to the PCI slot number of the main path CPU or the slave path CPU, so that the main path CPU or the slave path CPU connected with the NVMe storage card is obtained;

[0040] the affinity of the hardware queue interrupt of the NVMe storage card is set, so that the interrupt of the hardware queue of the NVMe storage card is processed on the main path CPU or the slave path CPU connected therewith. In specific implementation, firstly, the number of NVMe storage card hardware multi-column queues on the cloud S2500 dual-path server is limited to not more than 64 (the cloud S2500 processor is a 64-core architecture, that is, the main path CPU and the slave path CPU each have 64 cores). Secondly, whether the NVMe storage card is inserted into the PCI slot of the main path CPU or the slave path CPU of the cloud S2500 dual-path server is judged through a specific software interface. Finally, the kernel NVMe driver is modified, so that the hardware queue interrupt is concentrated on the CPU actually accessed by the NVMe storage card (for example, when the NVMe storage card is connected to the PCI slot of the main path CPU, the NVMe hardware queue interrupt is concentrated on the main path CPU for processing, and when the NVMe storage card is connected to the PCI slot of the slave path CPU, the NVMe hardware multi-column queue interrupt is concentrated on the slave path CPU for processing), so that the interrupt across the path is avoided, and the specific effect is as shown in Figure 3 The NVMe storage card connected with the main path CPU comprises 64 hardware queues, and the NVMe storage card connected with the slave path CPU comprises 32 hardware queues. Of course, it should be understood by those skilled in the art that the number of hardware queues of the NVMe storage card is not limited, and the present application is effective for optimization of any number of hardware queues. Figure 1The optimized interrupt mapping framework diagram is shown in Figure 2 As can be seen, the main road CPU supports 128 hardware queues of the NVMe storage card, which limits the hardware queues to 64, and then evenly maps the 64 hardware queue interrupts to the main road 0-63 CPU cores. The slave road supports 32 hardware queues of the NVMe storage card, and in order to evenly map the hardware queue interrupts to the slave road 64-127 CPU, the kernel default mapping of one hardware queue interrupt per 4 CPU cores is changed to one hardware queue interrupt per 2 CPU cores. Figure 2 As can be seen, whether the NVMe storage card is connected to the main road PCI slot or the slave road PCI slot, there is no interrupt cross-road problem, and of course there is no interrupt accumulation on the main road 0 CPU and the slave road 64 CPU. The hardware queue interrupts are processed in the on-road CPU.

[0041] Specifically, for step S1, i.e., limiting the number of NVMe storage card hardware queues: first, it is also necessary to determine whether the current processor platform is a dual-road Tengyun S2500, if so, the following steps are executed; then limit the number of NVMe storage card hardware queues to no more than the number of cores of Tengyun S2500 single CPU, i.e., 64.

[0042] For step S2, i.e., the hardware queue interrupts of the NVMe storage card are concentrated on the CPU accessed by the NVMe storage card for processing: first, it is also necessary to determine whether the current processor platform is a dual-road Tengyun S2500, if so, the following steps are executed; second, get the node number of the numa node where the NVMe storage card is located, according to the node number, it can be inferred whether the NVMe storage card is inserted into the PCI slot of the main road CPU or the slave road CPU; finally, set the affinity of the NVMe hardware queue interrupt to make it concentrated on the inferred main road or slave road CPU for processing; in step S1, the number of NVMe hardware queues is limited to no more than 64 because Tengyun S2500 is a 64-core CPU. The purpose of this operation is to concentrate the NVMe hardware queue interrupts on the main road CPU or the slave road CPU for processing in step S2 below, to achieve the purpose of mapping one hardware queue interrupt per CPU core. In addition, in step S2, the CPU ID can be used to distinguish the processor model, and at the same time, the numa node number of the main road CPU is 0-7, and the numa node number of the slave road CPU is 8-15, so getting the numa node number also determines whether the NVMe storage card is inserted into the PCI slot of the main road CPU or the slave road CPU.

[0043] In this embodiment, for steps S1 and S2, refer to Figure 4 , the following steps can be performed in detail:

[0044] 1) Call a function to obtain the CPU ID, by which it can be determined whether the current platform is S2500; if yes, the following steps are executed;

[0045] 2) Call the function pci_irq_get_node to obtain the numa node number of the NVMe storage card device, that is, nvme_node.

[0046] 3) Further determine whether the NVMe storage card is inserted into the master CPU or the slave CPU through the numa node number nvme_node; when the nvme_node is 0-7, it is the master CPU; when the nvme_node is 8-15, it is the slave CPU.

[0047] 4) When creating a hardware queue, according to the judgment of step 3), the affinity of the hardware queue interrupt is evenly set to the master CPU or the slave CPU through the interface irq_set_affinity_hint.

[0048] Based on the same inventive concept, the application further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method for optimizing the performance of the NVMe storage card based on the S2500 dual-path server architecture of Tencent Cloud.

[0049] The processor can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor (such as a GPU (Graphics Processing Unit)), or other data processing chips in some embodiments. The processor is usually used to control the overall operation of the electronic device. In this embodiment, the processor is used to run the program code or process data stored in the memory, for example, to run the program code of the method for optimizing the performance of the NVMe storage card based on the S2500 dual-path server architecture of Tencent Cloud.

[0050] The memory includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device. Of course, the memory can also include both the internal storage unit and the external storage device of the electronic device. In this embodiment, the memory is generally used to store the operating method and various application software installed on the electronic device, such as the program code of the NVMe storage card performance optimization method based on the Tencent S2500 dual-path server architecture, etc. In addition, the memory can also be used to temporarily store various data that has been output or will be output.

[0051] Based on the same inventive concept, the application also provides a readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the NVMe storage card performance optimization method based on the Tencent S2500 dual-path server architecture.

[0052] The application has the advantage of providing the NVMe storage card performance optimization method based on the Tencent S2500 dual-path server architecture, which optimizes the number of hardware queues of the NVMe storage card according to the characteristics of the Tencent S2500 dual-path server architecture, and optimizes the interrupt affinity according to whether the NVMe storage card is actually located in the master CPU or the slave CPU, so as to avoid the problem of interrupt cross-path accumulation processing caused by interrupt cross-path, and achieve the purpose of optimizing the performance of the NVMe storage card.

[0053] The above embodiments are only used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the application.

Claims

1. A method for optimizing performance of an NVMe storage card based on a Tencent S2500 dual-path server architecture, the Tencent S2500 dual-path server architecture comprising a master path CPU and a slave path CPU, the master path CPU and the slave path CPU being respectively connected with an NVMe storage card, the NVMe storage card comprising a plurality of hardware queues, characterized in that, The method comprises the following steps: S1, controlling the number of hardware queues of the NVMe storage card, so that the number of hardware queues of the NVMe storage card does not exceed the number of cores of the master CPU or the slave CPU connected thereto; S2, controlling the interrupt of the hardware queue of the NVMe storage card to be processed on the master CPU or the slave CPU connected thereto, specifically comprising: obtaining the node number of the numa node where the NVMe storage card is located, the node number corresponding to the PCI slot number of the master CPU or the slave CPU, obtaining the master CPU or the slave CPU connected with the NVMe storage card; setting the affinity of the hardware queue interrupt of the NVMe storage card, so that the interrupt of the hardware queue of the NVMe storage card is processed on the master CPU or the slave CPU connected thereto.

2. The method for performance optimization of NVMe storage card based on cloud S2500 dual-path server architecture according to claim 1, characterized in that, The node number of the numa node where the NVMe storage card is located is obtained by calling the pci_irq_get_node function.

3. The method of claim 2, wherein the cloud S2500 dual-path server architecture-based NVMe storage card performance optimization method is characterized by, When the node number is 0-7, it means that the NVMe storage card is connected to the master CPU, and when the node number is 8-15, it means that the NVMe storage card is connected to the slave CPU.

4. The method of claim 1, wherein the cloud S2500 dual-path server architecture-based NVMe storage card performance optimization method is characterized by, The number of cores of the master CPU and the slave CPU is 64.

5. The method of claim 4, wherein the cloud S2500 dual-path server architecture-based NVMe storage card performance optimization method is characterized by, The number of hardware queues of the NVMe storage card connected to the master CPU is 64, and the number of hardware queues of the NVMe storage card connected to the slave CPU is 32.

6. An electronic device, comprising: The method comprises a processor and a memory, and the memory stores a computer program, and the computer program is executed by the processor to realize the method of any one of claims 1-5.

7. A readable storage medium, characterized by, The computer program is stored in the readable storage medium, and the computer program is executed by the processor to realize the method of any one of claims 1-5.

Images