Methods, systems, devices, and storage media of inter-node pcie link performance testing

Abstract

The application provides a kind of inter-node PCIE link performance test method, system, equipment and storage medium, method includes: using external PCIE link interconnection to constitute four control storage array, each controller is connected two controllers of another IO group by two PCIE links respectively;Each server and a controller in four control storage array are directly connected by FC link;IO is issued to corresponding node through server, and corresponding instruction is executed to volume corresponding node by forwarding to controller inter PCIE link;And by adjusting server end thread parameter, front end pressure is filled with PCIE link bandwidth to test the performance actually used by PCIE link.The application makes full use of the characteristics of storage PCIE four control interconnection and storage IO processing flow, without using professional tools, the interconnection link performance between storage can be tested.

Description

Technical Field

[0001] This invention relates to the field of testing, and more specifically, to a method, system, device, and storage medium for testing the performance of PCIe links between nodes. Background Technology

[0002] The most direct way to test the performance of interconnect links in a multi-controller storage array is to use professional tools. However, these tools are often expensive and cost a lot of money. Furthermore, the performance of the interconnect links only reflects the interconnect links themselves and cannot reflect the storage's processing capabilities in terms of interconnection. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, system, computer device, and computer-readable storage medium for testing the performance of PCIe links between nodes. This invention fully utilizes the characteristics of storage PCIe four-controller interconnection and storage IO processing flow, and can test the performance of interconnection links between storage devices without relying on professional tools. This invention can test a single link to provide data support for product development, or test the overall performance value of IO interconnection forwarding links between storage products, which can better reflect product value and improve product competitiveness.

[0004] To achieve the above objectives, one aspect of this invention provides a method for testing the performance of inter-node PCIe links, comprising the following steps: using external PCIe links to interconnect a quad-controller storage array, with each controller connected to two controllers in another I / O group via two PCIe links; directly connecting each server to one controller in the quad-controller storage array via an FC link; distributing I / O to the corresponding node via the server and forwarding it to the corresponding node of the volume via the inter-controller PCIe link to execute the corresponding instruction; and adjusting the server-side thread parameters to fully utilize the PCIe link bandwidth to test the actual performance of the PCIe link in use.

[0005] In some implementations, the method further includes: configuring pooled volumes on the storage end, creating eight volumes on each controller.

[0006] In some implementations, the method further includes: establishing a first mapping between the controller and the server, and establishing a second mapping between the server and the PCIe link used during I / O processing.

[0007] In some implementations, the step of forwarding the corresponding instruction to the volume-corresponding node via the inter-controller PCIe link includes: forwarding the I / O from the upper-layer I / O forwarding module of the node to the upper-layer I / O forwarding module of the volume-corresponding node via the inter-controller PCIe link; and sending the I / O from the upper-layer I / O forwarding module of the volume-corresponding node to the node cache, and receiving the read return or write return instruction from the volume-corresponding node.

[0008] In another aspect, this invention provides a system for testing the performance of PCIe links between nodes, comprising: a creation module configured to interconnect a quad-controller storage array using external PCIe links, wherein each controller is connected to two controllers in another I / O group via two PCIe links; a connection module configured to directly connect each server to one controller in the quad-controller storage array via an FC link; a distribution module configured to distribute I / O to the corresponding node via the server and forward it to the corresponding node of the volume via the inter-controller PCIe link to execute the corresponding instruction; and a testing module configured to test the actual performance of the PCIe link by adjusting the server-side thread parameters to fully utilize the PCIe link bandwidth.

[0009] In some implementations, the system also includes a configuration module configured to: configure pooled volumes for the storage end, creating eight volumes on each controller.

[0010] In some implementations, the system also includes a mapping module configured to: establish a first mapping between the controller and the server, and establish a second mapping between the server and the PCIe link used during I / O processing.

[0011] In some implementations, the dispatch module is configured to: forward IO from the upper-layer IO forwarding module of the node to the upper-layer IO forwarding module of the volume-corresponding node via the inter-controller PCIE link; and dispatch IO from the upper-layer IO forwarding module of the volume-corresponding node to the node cache, and receive read return or write return instructions from the volume-corresponding node.

[0012] In another aspect of the present invention, a computer device is provided, comprising: at least one processor; and a memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing the steps of the method described above.

[0013] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0014] This invention has the following beneficial technical effects: it fully utilizes the characteristics of storage PCIe four-controller interconnection and storage IO processing flow, and can test the performance of interconnect links between storage devices without relying on professional tools; the embodiments of this invention can test a single link to provide data support for product development, or test the overall performance value of the IO interconnection forwarding link between storage products, which can better reflect the product value and improve product competitiveness. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram illustrating an embodiment of the method for testing the performance of PCIe links between nodes provided by the present invention;

[0017] Figure 2 This is a schematic diagram of the link in an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of an embodiment of the system for testing the performance of PCIe links between nodes provided by the present invention;

[0019] Figure 4 A schematic diagram of the hardware structure of an embodiment of the computer device for testing the performance of inter-node PCIE links provided by the present invention;

[0020] Figure 5 This is a schematic diagram of an embodiment of a computer storage medium for testing the performance of an inter-node PCIe link provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0022] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0023] In a first aspect, an embodiment of a method for testing the performance of PCIE links between nodes is proposed. Figure 1This diagram illustrates an embodiment of the method for testing the performance of PCIe links between nodes provided by the present invention. Figure 1 As shown, the embodiments of the present invention include the following steps:

[0024] S1. A four-controller storage array is formed by interconnecting external PCIe links. Each controller is connected to two controllers of another IO group through two PCIe links.

[0025] S2. Connect each server directly to one of the controllers in the four-controller storage array via an FC link;

[0026] S3. The I / O is distributed from the server to the corresponding node, and then forwarded to the corresponding node of the volume via the PCIe link between controllers to execute the corresponding instructions; and

[0027] S4. By adjusting the server-side thread parameters, the front-end pressure is fully applied to the PCIe link bandwidth to test the actual performance of the PCIe link.

[0028] Figure 2 This is a schematic diagram of the link in an embodiment of the present invention, combined with... Figure 2 The embodiments of the present invention will be described.

[0029] The storage array involved in this invention is a dual-controller storage array, which can be interconnected through an external PCIe link to form a four-controller storage array. The dual-controller storage array is referred to as an IO group (IOGROUP).

[0030] A quad-controller storage array is constructed using external PCIe links. Each controller is connected to two controllers in another I / O group via two PCIe links. Alternatively, a quad-controller storage array can be constructed using external PCIe links. Each controller is connected to two controllers in another I / O group via two PCIe links, forming four PCIe interconnect links (links 1-4).

[0031] Each server is directly connected to one of the controllers in the four-controller storage array via an FC (Fibre Channel) link. This embodiment of the invention uses four servers, each directly connected to one of the controllers in the four-controller storage array via an FC link.

[0032] In some implementations, the method further includes: configuring pooled volumes on the storage side, creating eight volumes on each controller. Specifically, configuring pooled volumes on the storage side, creating eight volumes on each controller (node1-node4).

[0033] In some implementations, the method further includes: establishing a first mapping between the controller and the server, and establishing a second mapping between the server and the PCIe links used during I / O processing. Volume mapping is performed according to the following table, where the corresponding PCIe links used during I / O processing are listed in the table below:

[0034] LUN1-LUN8 Controller 1 Server 3 3 LUN9-LUN16 Controller 2 Server 4 2 LUN17-LUN24 Controller 3 Server 2 1 LUN25-LUN32 Controller 4 Server 1 4

[0035] The I / O is distributed from the server to the corresponding node, and then forwarded to the corresponding node of the volume through the PCIe link between controllers to execute the corresponding instructions.

[0036] In some implementations, the step of forwarding the corresponding instruction to the volume-corresponding node via the inter-controller PCIe link includes: forwarding the I / O from the upper-layer I / O forwarding module of the node to the upper-layer I / O forwarding module of the volume-corresponding node via the inter-controller PCIe link; and sending the I / O from the upper-layer I / O forwarding module of the volume-corresponding node to the node cache, and receiving the read return or write return instruction from the volume-corresponding node.

[0037] The Vdbench tool was used to issue I / O requests to four servers. The I / O data model was selected based on a large-block sequential read / write hit scenario, such as 1M sequential 100% read hit and 1M sequential 100% write hit. The I / O was issued from the server to the corresponding node's UFW module (upper-forward I / O forwarding module), and then forwarded to the UFW module of the corresponding node of the volume through the PCIe link between controllers. After that, it was issued to the node cache. The node returned a read return or write return command, and the command was returned to the server along the original path, thus completing one read / write operation.

[0038] For example, LUN17-LUN24 on controller 3 are mapped to server 2. When the server issues IO through the vdbench software, the IO is first issued to the UFW module of controller 2, which is directly connected to server 2 via the FC line. Because the issued IO model is a hit model, the IO is read and written back to the storage cache. Therefore, the storage forwards the IO to the UFW module of controller 3 through PCIe link 1 via the UFW module. Then, the UFW of controller 3 issues the IO to the cache to complete the read and write operation.

[0039] By adjusting server-side thread parameters to fully utilize the PCIe link bandwidth under front-end load, the actual performance of the PCIe link in real-world use was tested. Similarly, by adjusting the number of threads in the server-side vdbench tool to maximize the PCIe link bandwidth and approximate its theoretical performance, the actual performance of the PCIe link in real-world use was tested.

[0040] This invention fully utilizes the characteristics of storage PCIe quad controller interconnection and storage I / O processing flow, and can test the performance of interconnect links between storage devices without relying on professional tools. This solution can test a single link to provide data support for product development, or test the overall performance of the I / O interconnection forwarding link between storage products, which can better reflect the product value and improve product competitiveness.

[0041] It should be noted that the steps in each embodiment of the above-mentioned method for testing the performance of PCIe links between nodes can be interleaved, substituted, added, or deleted. Therefore, these reasonable permutations and combinations of the method for testing the performance of PCIe links between nodes should also fall within the protection scope of this invention, and the protection scope of this invention should not be limited to the embodiments.

[0042] Based on the above objectives, a second aspect of this invention proposes a system for testing the performance of PCIe links between nodes. For example... Figure 3 As shown, system 200 includes the following modules: a creation module, configured to interconnect a four-controller storage array using external PCIe links, with each controller connected to two controllers in another IO group via two PCIe links; a connection module, configured to directly connect each server to one controller in the four-controller storage array via an FC link; a distribution module, configured to distribute IO to the corresponding node via the server and forward it to the corresponding node of the volume via the inter-controller PCIe link to execute the corresponding instructions; and a testing module, configured to test the actual performance of the PCIe link by adjusting the server-side thread parameters to fully utilize the PCIe link bandwidth.

[0043] In some implementations, the system also includes a configuration module configured to: configure pooled volumes for the storage end, creating eight volumes on each controller.

[0044] In some implementations, the system also includes a mapping module configured to: establish a first mapping between the controller and the server, and establish a second mapping between the server and the PCIe link used during I / O processing.

[0045] In some implementations, the dispatch module is configured to: forward IO from the upper-layer IO forwarding module of the node to the upper-layer IO forwarding module of the volume-corresponding node via the inter-controller PCIE link; and dispatch IO from the upper-layer IO forwarding module of the volume-corresponding node to the node cache, and receive read return or write return instructions from the volume-corresponding node.

[0046] Based on the above objectives, a third aspect of the present invention provides a computer device, comprising: at least one processor; and a memory storing computer instructions executable by the processor to perform the following steps: S1, forming a quad-controller storage array using external PCIe links interconnected, each controller being connected to two controllers of another I / O group via two PCIe links respectively; S2, directly connecting each server to one controller in the quad-controller storage array via an FC link; S3, distributing I / O to the corresponding node through the server and forwarding it to the corresponding node of the volume via the inter-controller PCIe link to execute the corresponding instructions; and S4, testing the actual performance of the PCIe link by adjusting the server-side thread parameters to fully utilize the PCIe link bandwidth.

[0047] In some implementations, the steps further include: configuring pooled volumes on the storage end, creating eight volumes on each controller.

[0048] In some implementations, the steps further include: establishing a first mapping between the controller and the server, and establishing a second mapping between the server and the PCIe link used during I / O processing.

[0049] In some implementations, the step of forwarding the corresponding instruction to the volume-corresponding node via the inter-controller PCIe link includes: forwarding the I / O from the upper-layer I / O forwarding module of the node to the upper-layer I / O forwarding module of the volume-corresponding node via the inter-controller PCIe link; and sending the I / O from the upper-layer I / O forwarding module of the volume-corresponding node to the node cache, and receiving the read return or write return instruction from the volume-corresponding node.

[0050] like Figure 4 The diagram shown is a hardware structure schematic of an embodiment of the computer device for testing the performance of the PCIE link between nodes provided by the present invention.

[0051] For example Figure 4 Taking the device shown as an example, the device includes a processor 301 and a memory 302.

[0052] Processor 301 and memory 302 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0053] The memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the inter-node PCIe link performance testing method in this embodiment. The processor 301 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 302, thereby implementing the inter-node PCIe link performance testing method.

[0054] Memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created using methods for testing the performance of inter-node PCIe links. Furthermore, memory 302 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 302 may optionally include memory remotely located relative to processor 301, and these remote memories can be connected to the local module via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0055] The computer instructions 303 corresponding to the method for testing the performance of PCIe links between one or more nodes are stored in the memory 302. When executed by the processor 301, the method for testing the performance of PCIe links between nodes in any of the above method embodiments is executed.

[0056] Any embodiment of the computer device that performs the above-described method for testing the performance of PCIe links between nodes can achieve the same or similar effects as any of the aforementioned method embodiments.

[0057] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a method for testing the performance of an inter-node PCIe link.

[0058] like Figure 5 The diagram shown is a schematic representation of an embodiment of the computer storage medium used for inter-node PCIe link performance testing provided by the present invention. Figure 5 Taking the computer storage medium shown as an example, the computer-readable storage medium 401 stores a computer program 402 that, when executed by a processor, performs the above method.

[0059] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for the inter-node PCIe link performance testing method can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0060] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0061] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0062] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0063] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0064] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for testing the performance of an inter-node PCIe link, characterized in that, Includes the following steps: A four-controller storage array is formed by interconnecting external PCIe links, with each controller connected to two controllers in another I / O group via two PCIe links. Each server is directly connected to one of the controllers in the four-controller storage array via an FC link; The I / O is distributed from the server to the corresponding node and forwarded to the corresponding node of the volume through the PCIe link between controllers to execute the corresponding instructions. as well as By adjusting the server-side thread parameters, the front-end pressure was fully applied to the PCIe link bandwidth to test the actual performance of the PCIe link. Configure pooled volumes on the storage side, creating eight volumes on each controller; Establish the first mapping between the controller and the server, and establish the second mapping between the server and the PCIe link used during IO processing; The step of forwarding the corresponding instruction to the corresponding node of the volume via the inter-controller PCIe link includes: The I / O is forwarded from the upper-layer I / O forwarding module of the node to the upper-layer I / O forwarding module of the corresponding node of the volume via the PCIe link between controllers. as well as The I / O is sent from the upper-layer I / O forwarding module of the corresponding node of the volume to the node cache, and the read return or write return instruction of the corresponding node of the volume is received.

2. A system for testing the performance of PCIe links between nodes, characterized in that, include: Create a module and configure it to interconnect a four-controller storage array using external PCIe links. Each controller is connected to two controllers in another I / O group via two PCIe links. A connectivity module configured to directly connect each server and one of the controllers in the four-controller storage array via an FC link; The distribution module is configured to distribute I / O to the corresponding node via the server and forward it to the corresponding node of the volume through the PCIe link between controllers to execute the corresponding instructions. as well as The test module is configured to test the actual performance of the PCIe link by adjusting the server-side thread parameters to fully utilize the PCIe link bandwidth. The configuration module is used to configure pooled volumes on the storage side, creating eight volumes on each controller. The mapping module is configured to: establish the first mapping between the controller and the server, and establish the second mapping between the server and the PCIe link used during IO processing; The distribution module configuration is used for: The I / O is forwarded from the upper-layer I / O forwarding module of the node to the upper-layer I / O forwarding module of the corresponding node of the volume via the PCIe link between controllers. as well as The I / O is sent from the upper-layer I / O forwarding module of the corresponding node of the volume to the node cache, and the read return or write return instruction of the corresponding node of the volume is received.

3. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method of claim 1.

4. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 1.

Images