A stress testing system, method, and product for a Ceph cluster

Abstract

The application provides a stress test system, method and product of a Ceph cluster, and belongs to the technical field of stress test. The system comprises a distribution module configured to distribute connection configuration information of a Ceph cluster to be tested or any stress test instruction to a plurality of stress testers; a stress test module configured to generate a stress test instruction for the Ceph cluster to be tested, the stress test instruction comprising at least one of a Mon stress test instruction, a Mgr stress test instruction, a MDS stress test instruction and an OSD stress test instruction; and a monitoring module configured to monitor cluster state information of the Ceph cluster to be tested during stress test and component states of each component in the Ceph cluster to be tested. The application aims to improve the accuracy of stress test of the Ceph cluster.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of testing, in particular, to a stress test system, method and product of a Ceph cluster. BACKGROUND

[0002] In the era of rapid development of cloud computing, Ceph, as a huge distributed system, has been widely used. Ceph is a distributed unified storage platform integrating traditional block, file storage and object storage, which has high performance, reliability and scalability, and is suitable for unstructured data. Many components are included in Ceph, such as storage components OSD and control plane components. The control plane components include Mon, MDS and Mgr, etc. Generally, the number of control plane components is greater than 1 and is an odd number, such as 3 or 5 control plane components.

[0003] Among them, the OSD component is a component for storing data in the Ceph cluster; the MDS component is a metadata server in Ceph, which mainly provides metadata management services for the file system; the Mon component is mainly responsible for maintaining the overall state of the cluster; the main function of the Mgr component is to share and expand part of the function of Mon, and to reduce the burden of Mon.

[0004] In order to ensure the high availability of Ceph, it is necessary to automatically test the stress of Ceph cluster, which helps to determine the performance upper limit and possible problems of Ceph cluster components. However, the current stress test method for Ceph is mainly to test the storage component OSD, such as using FIO and Rados Bench tools to test the disk performance. However, this test method ignores the stress test of other control plane components in Ceph, resulting in that the test of Ceph is not comprehensive enough, and the stress test result of Ceph is not accurate enough. SUMMARY

[0005] Embodiments of the present application provide a stress test system, method and product of a Ceph cluster, aiming to improve the accuracy of stress test of the Ceph cluster.

[0006] In a first aspect, embodiments of the present application provide a stress test system of a Ceph cluster, the system comprising:

[0007] a distribution module configured to distribute connection configuration information of a Ceph cluster to be tested or any stress test instruction to a plurality of stress machines, the connection configuration information being used for the plurality of stress machines to establish a connection with the Ceph cluster to be tested, and the plurality of stress machines performing a stress test process on the Ceph cluster to be tested in response to any stress test instruction;

[0008] a stress testing module configured to generate stress testing instructions for the Ceph cluster to be tested, the stress testing instructions comprising at least one of a Mon stress testing instruction, a Mgr stress testing instruction, a MDS stress testing instruction, and an OSD stress testing instruction;

[0009] a monitoring module configured to monitor cluster state information of the Ceph cluster to be tested during the stress testing and component state of each component in the Ceph cluster to be tested.

[0010] Optionally, the stress testing module comprises a MDS stress testing unit configured to generate a first MDS stress testing instruction, a second MDS stress testing instruction, a third MDS stress testing instruction, and a fourth MDS stress testing instruction for testing a MDS component of the Ceph cluster to be tested.

[0011] The first MDS stress testing instruction is configured to mount a target number of directory read-write threads on the plurality of stress machines; the second MDS stress testing instruction is configured to create a first number of first files under a mounted directory on the plurality of stress machines; the third MDS stress testing instruction is configured to create a second number of second files under the mounted directory on the plurality of stress machines; and the fourth MDS stress testing instruction is configured to simulate network fault recovery and access the MDS component of the Ceph cluster to be tested on the plurality of stress machines.

[0012] The first files are files with a size less than a first threshold value; and the second files are files with a size greater than a second threshold value.

[0013] Optionally, the stress testing module comprises a Mgr stress testing unit configured to generate a Mgr stress testing instruction for testing a Mgr component of the Ceph cluster to be tested.

[0014] The Mgr stress testing instruction is configured to enable the plurality of stress machines to batch acquire cluster state information of the Ceph cluster to be tested, so as to increase the load on the Mgr component of the Ceph cluster to be tested.

[0015] Optionally, the stress testing module comprises a Mon stress testing unit configured to generate a first Mon stress testing instruction and a second Mon stress testing instruction for testing a Mon component of the Ceph cluster to be tested.

[0016] The first Mon stress testing instruction is configured to increase the size of an OSDMap in the Ceph cluster to be tested; and the second Mon stress testing instruction is configured to increase the read-write load on the Mon component of the Ceph cluster to be tested.

[0017] Optionally, the stress testing module comprises an OSD stress testing unit, which is configured to generate random read-write instructions and sequential read-write instructions for testing OSD components of the Ceph cluster to be tested.

[0018] Optionally, the monitoring module is further configured to, after obtaining the cluster state information, analyze error information of the Ceph cluster to be tested during the stress testing according to the cluster state information when the cluster state information indicates that the Ceph cluster to be tested is not in a healthy state.

[0019] Optionally, the system further comprises:

[0020] a collection and analysis module configured to collect any stress testing data of the Ceph cluster to be tested during the stress testing.

[0021] In a second aspect, an embodiment of the present application provides a stress testing method for a Ceph cluster, the method comprising:

[0022] distributing connection configuration information of the Ceph cluster to be tested or any stress testing instruction to a plurality of stress testing machines, the connection configuration information being used for the plurality of stress testing machines to establish a connection with the Ceph cluster to be tested, and the stress testing machines performing a stress testing process on the Ceph cluster to be tested in response to the any stress testing instruction;

[0023] generating stress testing instructions for the Ceph cluster to be tested, the stress testing instructions comprising at least one of a Mon stress testing instruction, a Mgr stress testing instruction, a MDS stress testing instruction, and an OSD stress testing instruction;

[0024] monitoring cluster state information of the Ceph cluster to be tested during the stress testing and component state information of each component in the Ceph cluster to be tested.

[0025] In a third aspect, an embodiment of the present application provides a computer device, comprising at least one processor and a memory, the memory storing a computer program capable of running on the processor, wherein the processor executes the computer program to perform the stress testing method for a Ceph cluster according to the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, the non-volatile readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the stress testing method for a Ceph cluster according to the embodiment.

[0027] Advantages:

[0028] The stress testing system for a Ceph cluster provided in the embodiment comprises a distribution module, a stress testing module, and a monitoring module.

[0029] The distribution module is configured to distribute connection configuration information of a Ceph cluster to be tested or any stress testing instruction to a plurality of stress machines, the connection configuration information being used for the plurality of stress machines to establish a connection with the Ceph cluster to be tested, and the plurality of stress machines performing a stress testing process on the Ceph cluster to be tested in response to any stress testing instruction.

[0030] The stress testing module is configured to generate a stress testing instruction for the Ceph cluster to be tested, the stress testing instruction including at least one of a Mon stress testing instruction, a Mgr stress testing instruction, a MDS stress testing instruction, and an OSD stress testing instruction; and the monitoring module is configured to monitor cluster state information of the Ceph cluster to be tested in a stress testing process and component states of various components in the Ceph cluster to be tested.

[0031] Compared with the prior stress testing method for Ceph, in the system, a plurality of components in a Ceph cluster are respectively subjected to targeted stress testing, in addition to OSD components, Mon components, Mgr components, and MDS components can also be tested, the lack of stress testing on control plane components in the prior stress testing process is made up, and the Ceph cluster can be comprehensively and automatically subjected to stress testing, so that the Ceph cluster to be tested can be more accurately subjected to stress testing. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative labor.

[0033] Figure 1 is a structural schematic diagram of a stress testing system of a Ceph cluster according to an embodiment of the present application;

[0034] Figure 2 is a structural schematic diagram of a stress testing system of a Ceph cluster according to an embodiment of the present application;

[0035] Figure 3 is a step flowchart of a stress testing method of a Ceph cluster according to an embodiment of the present application;

[0036] Figure 4 is an execution diagram of a stress testing method of a Ceph cluster according to an embodiment of the present application;

[0037] Figure 5 is a schematic diagram of a computer device according to an embodiment of the present application;

[0038] Figure 6 FIG. 1 is a schematic diagram of a non-volatile readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0040] Ceph: a distributed file system;

[0041] Mon: monitor, monitor;

[0042] MDS: metadata service, metadata server;

[0043] Mgr: Manager Daemon, Manager daemon;

[0044] OSD: Object Storage Daemon, object storage device;

[0045] ID: Identity document, identity;

[0046] Automated stress testing: stress testing, the process is to monitor the performance of the system under test under certain pressure by continuously applying pressure to the system under test, which can investigate the maximum load that the system can withstand under the current software and hardware environment and help find out the system bottleneck. Automatic stress testing is the process of changing manual stress testing to automation, usually using scripts and other automated methods to complete the stress testing process.

[0047] FIO: an I / O tool for stress testing and verification of hardware;

[0048] Rados Bench: a performance benchmarking tool provided by Ceph cluster, used for performance testing of the cluster; it supports three types of tests: write test and two types of read test, such as sequential read and random read.

[0049] Ceph is a distributed unified storage platform integrating traditional block, file storage and object storage, which has high performance, reliability and scalability, and is suitable for unstructured data. Ceph is a software-defined storage that can run on mainstream Linux distributions and other Unix-like systems. Ceph includes storage components and control plane components, where the storage components refer to OSD, and the control plane components include Mon components, MDS components and Mgr components, etc.

[0050] In the era of rapid development of cloud computing, Ceph is widely used. As a large distributed system, Ceph has many components, such as storage components OSD and control plane components, including Mon, MDS and Mgr, etc. In order to ensure high availability, the number of control plane components is generally greater than 1 and is an odd number, such as 3 or 5 control plane components.

[0051] In order to ensure the high availability of Ceph, it is necessary to automatically stress test the Ceph cluster, which helps to determine the performance upper limit of the Ceph cluster components and possible problems.

[0052] At present, the stress test method for Ceph is mainly to test the storage component OSD, such as using FIO and Rados Bench tools to test the disk performance, but this test method ignores the stress test of other control plane components in Ceph, resulting in incomplete test of Ceph.

[0053] Therefore, the embodiment of the present application provides a stress test system for Ceph cluster, which can not only test OSD components, but also test Mon components, Mgr components and MDS components, making up for the lack of control plane component stress test in the current stress test process, and can comprehensively automatically stress test the Ceph cluster and more accurately stress test the Ceph cluster to be tested.

[0054] Referring to Figure 1 , a structure schematic diagram of a stress test system for Ceph cluster provided by an embodiment of the present application is shown, the system comprises a distribution module, a stress test module and a monitoring module.

[0055] The distribution module is used to distribute the connection configuration information of the Ceph cluster to be tested or any stress test instruction to a plurality of stress machines.

[0056] The connection configuration information is used for the plurality of stress machines to establish a connection with the Ceph cluster to be tested, and the connection configuration information includes keyring and conf configuration information, etc. The plurality of stress machines perform a stress test process on the Ceph cluster to be tested in response to any stress test instruction.

[0057] The stress testing module is configured to generate stress testing instructions for the Ceph cluster to be tested, wherein the stress testing instructions comprise at least one of Mon stress testing instructions, Mgr stress testing instructions, MDS stress testing instructions, and OSD stress testing instructions.

[0058] The monitoring module is configured to monitor cluster state information of the Ceph cluster to be tested during stress testing and component states of each component in the Ceph cluster to be tested.

[0059] In an embodiment, the stress testing module comprises an MDS stress testing unit configured to generate stress testing instructions for stress testing of an MDS component of the Ceph cluster to be tested.

[0060] Specifically, the MDS stress testing unit is configured to generate first MDS stress testing instructions, second MDS stress testing instructions, third MDS stress testing instructions, and fourth MDS stress testing instructions for testing of the MDS component of the Ceph cluster to be tested.

[0061] The first MDS stress testing instructions are configured to be used by the plurality of stress testing machines to mount a target number of directory read-write threads.

[0062] The second MDS stress testing instructions are configured to be used by the plurality of stress testing machines to create a first number of first files under the mounted directory.

[0063] The third MDS stress testing instructions are configured to be used by the plurality of stress testing machines to create a second number of second files under the mounted directory.

[0064] The fourth MDS stress testing instructions are configured to be used by the plurality of stress testing machines to simulate network fault recovery and access the MDS component of the Ceph cluster to be tested; the first files are files with a size less than a first threshold value; and the second files are files with a size greater than a second threshold value; that is, the first files are small files, and the second files are large files, and the first threshold value and the second threshold value can be set according to actual application requirements.

[0065] In actual implementation, the MDS stress testing unit can be configured to generate other MDS stress testing instructions according to actual application requirements, thereby extending stress testing of the MDS component of the Ceph cluster in different scenarios.

[0066] In an embodiment, the stress testing module comprises an Mgr stress testing unit configured to generate stress testing instructions for stress testing of an Mgr component of the Ceph cluster to be tested.

[0067] The Mgr stress testing unit is configured to generate Mgr stress testing instructions for testing of the Mgr component of the Ceph cluster to be tested.

[0068] The Mgr stress test instruction is used for the plurality of stress testers to batch acquire cluster state information of the Ceph cluster to be tested, so as to increase the load of the Mgr component of the Ceph cluster to be tested.

[0069] In actual implementation, the Mgr stress test unit can also generate other Mgr stress test instructions according to actual application requirements, so as to expand stress testing of the Mgr component of the Ceph cluster in different scenarios.

[0070] In a feasible implementation, the stress test module includes a Mon stress test unit, which is configured to generate stress test instructions for stress testing of a Mon component of a Ceph cluster to be tested.

[0071] The Mon stress test unit is configured to generate a first Mon stress test instruction and a second Mon stress test instruction for testing of the Mon component of the Ceph cluster to be tested.

[0072] The first Mon stress test instruction is used for increasing the size of an OSDMap in the Ceph cluster to be tested.

[0073] The second Mon stress test instruction is used for increasing the read-write load of the Mon component of the Ceph cluster to be tested.

[0074] In actual implementation, the Mon stress test unit can also generate other Mon stress test instructions according to actual application requirements, so as to expand stress testing of the Mon component of the Ceph cluster in different scenarios.

[0075] In a feasible implementation, the stress test module includes an OSD stress test unit, which is configured to generate stress test instructions for stress testing of an OSD component of a Ceph cluster to be tested.

[0076] The OSD stress test unit is configured to generate a random read-write instruction and a sequential read-write instruction for testing of the OSD component of the Ceph cluster to be tested.

[0077] In actual implementation, the OSD stress test unit can also generate other OSD stress test instructions according to actual application requirements, so as to expand stress testing of the OSD component of the Ceph cluster in different scenarios.

[0078] In a feasible implementation, the monitoring module is further configured to, after acquiring the cluster state information, when the cluster state information indicates that the Ceph cluster to be tested is not in a healthy state, analyze error information of the Ceph cluster to be tested in the stress testing process according to the cluster state information.

[0079] The system provided by the embodiment can test the Ceph cluster in a targeted manner, and can test the Mon component, the Mgr component and the MDS component in addition to the OSD component, so that the performance of the Ceph cluster can be tested more accurately.

[0080] With reference to Figure 2 , a structure diagram of a stress test system of a Ceph cluster is shown, the system includes a distribution module, a stress test module, a monitoring module and a collection and analysis module, and is used for collecting any stress test data of the Ceph cluster to be tested in a stress test process.

[0081] The distribution module is used for distributing the connection configuration information of the Ceph cluster to be tested or any stress test instruction to a plurality of stress machines. The stress test module includes a stress test unit for generating a stress test instruction for different components of the Ceph cluster to be tested. The stress test module specifically includes an MDS stress test unit for generating a stress test instruction for testing the MDS component of the Ceph cluster to be tested, an Mgr stress test unit for generating a stress test instruction for testing the Mgr component of the Ceph cluster to be tested, a stress test unit for testing the Mon component of the Ceph cluster to be tested, and an OSD stress test unit for generating a stress test instruction for testing the OSD component of the Ceph cluster to be tested.

[0082] The system provided by the embodiment can test the Ceph cluster in a targeted manner, and can test the Mon component, the Mgr component and the MDS component in addition to the OSD component, so that the performance of the Ceph cluster can be tested more accurately.

[0083] With reference to Figure 3 , a step flow chart of a stress test method of a Ceph cluster is shown. In a feasible implementation manner, the method specifically can include the following steps:

[0084] S101: Distribute the connection configuration information of the Ceph cluster to be tested or any stress test instruction to a plurality of stress machines.

[0085] The connection configuration information is used for the plurality of stress machines to establish a connection with the Ceph cluster to be tested, and the stress machines perform a stress test process on the Ceph cluster to be tested in response to any stress test instruction.

[0086] S102: generating a stress test instruction for the Ceph cluster to be tested; the stress test instruction includes at least one of a Mon stress test instruction, a Mgr stress test instruction, a MDS stress test instruction, and an OSD stress test instruction.

[0087] Specifically, the stress test module in the system provided by the embodiment includes a MDS stress test unit, a Mgr stress test unit, a Mon stress test unit, and an OSD stress test unit.

[0088] The MDS stress test unit generates a first MDS stress test instruction, a second MDS stress test instruction, a third MDS stress test instruction, and a fourth MDS stress test instruction for testing the MDS component of the Ceph cluster to be tested.

[0089] The first MDS stress test instruction can make the plurality of stress machines mount a target number of directory read-write threads.

[0090] The second MDS stress test instruction can make the plurality of stress machines create a first calibration number of first files under the mounted directory.

[0091] The third MDS stress test instruction can make the plurality of stress machines create a second calibration number of second files under the mounted directory.

[0092] The fourth MDS stress test instruction can make the plurality of stress machines simulate network fault recovery to access the MDS component of the Ceph cluster to be tested; the first file is a file with a size less than a first threshold value; and the second file is a file with a size greater than a second threshold value.

[0093] The Mgr stress test unit generates a Mgr stress test instruction for testing the Mgr component of the Ceph cluster to be tested.

[0094] The Mgr stress test instruction can make the plurality of stress machines batch acquire cluster state information of the Ceph cluster to be tested to increase the load on the Mgr component of the Ceph cluster to be tested.

[0095] The Mon stress test unit can generate a first Mon stress test instruction and a second Mon stress test instruction for testing the Mon component of the Ceph cluster to be tested.

[0096] The first Mon stress test instruction can increase the size of an OSDMap in the Ceph cluster to be tested.

[0097] The second Mon stress testing instruction can increase the read-write load of the Mon component of the Ceph cluster to be tested.

[0098] The OSD stress testing unit can generate random read-write instructions and sequential read-write instructions for testing the OSD component of the Ceph cluster to be tested.

[0099] S103: Monitor the cluster state information of the Ceph cluster to be tested and the component state of each component in the Ceph cluster to be tested during stress testing.

[0100] Specifically, the cluster state information and the component state of each component in the Ceph cluster to be tested are acquired in real time, it is determined whether the Ceph cluster to be tested is in a healthy state, if not, it is further determined error information of the Ceph cluster to be tested, and the error information is stored.

[0101] Referring to Figure 4 , an execution diagram of a stress testing method of a Ceph cluster provided by an embodiment of the present application is shown, in a feasible implementation manner, the execution process of the method can specifically include the following steps:

[0102] S1: Obtain the connection configuration information of the Ceph cluster to be tested.

[0103] The connection configuration information of the Ceph cluster to be tested can include configuration information for connecting the Ceph cluster, for example, the connection configuration information can be the keyring and conf configuration information of the Ceph cluster to be tested.

[0104] In actual implementation, the connection configuration information can be read in the Ceph cluster to be tested, and then the connection configuration information of the Ceph cluster to be tested is sent or uploaded to the distribution module of the stress testing system of the Ceph cluster, or other ways can be used to make the distribution module receive the connection configuration information.

[0105] S2: The distribution module distributes the connection configuration information of the Ceph cluster to be tested to each stressor, and any stressor connects the Ceph cluster to be tested according to the connection configuration information.

[0106] Referring to Figure 5 , a stress testing diagram of a Ceph cluster stress testing system provided by an embodiment of the present application is shown, the Ceph cluster stress testing system can configure one stressor node, and the stressor node can configure multiple stressors, such as stressors 1-n; the multiple stressors can be combined to complete the stress testing process of any component, and provide more abundant stress instances to increase the stress of the Ceph cluster.

[0107] When the distribution module distributes the connection configuration information of the Ceph cluster to be tested to each stressor, each stressor can connect the Ceph cluster to be tested according to the connection configuration information.

[0108] S3: The stress testing module generates stress testing instructions for the Ceph cluster to be tested.

[0109] In this embodiment, the stress testing module includes an MDS stress testing unit for the MDS component, an Mgr stress testing unit for the Mgr component, a Mon stress testing unit for the Mon component, and an OSD stress testing unit for the OSD component.

[0110] In the actual application of the stress testing process, not only can the stress testing module based on each component be used to jointly stress test the Ceph cluster to be tested to obtain a more comprehensive stress testing result of the Ceph cluster to be tested.

[0111] The stress testing process can also be performed in response to a stress testing setting operation, in which the component to be tested is selected, and the stress testing unit corresponding to the determined component to be tested is used for targeted stress testing.

[0112] The stress testing process can also be performed in response to a stress testing environment configuration, in which the stress testing environment of any component to be tested is determined, and the stress testing unit corresponding to the component to be tested generates stress testing instructions suitable for the stress testing environment, that is, in actual application, the stress testing module and the stress testing instructions of any stress testing module can be selected to perform the stress testing process.

[0113] The MDS component is a metadata server in Ceph, which is used to provide metadata management services of a file system and mainly processes requests related to file / directory metadata, such as creating, deleting, and querying files.

[0114] Specifically, the process of generating instructions by the MDS stress testing unit includes:

[0115] The first MDS stress testing instruction is generated for the multiple stressors to mount a target number of directory read-write threads, and after responding to the first MDS stress testing instruction, the multiple stressors perform a large number of client mounts and read-writes on the Ceph cluster to be tested, that is, multiple directories are simultaneously mounted on the stressors for read-write operations.

[0116] The second MDS stress testing instruction is generated for the multiple stressors to create a first calibration number of first files under the mounted directories, and the first files are files with a size smaller than a first threshold, that is, small files. After responding to the second MDS stress testing instruction, the multiple stressors perform a large number of small file operations, including creating a large number of small files under the mounted directories and performing decompression operations.

[0117] The third MDS stress test instruction is generated to instruct the plurality of stressors to create a second number of second files under a mounted directory, the second files being large files with a size greater than a second threshold. After responding to the third MDS stress test instruction, the plurality of stressors perform operations on the large files, including creating and deleting the large files in the mounted directory.

[0118] The fourth MDS stress test instruction is generated to instruct the plurality of stressors to simulate network fault recovery to access the MDS component of the Ceph cluster to be tested. For example, after responding to the fourth MDS stress test instruction, the plurality of stressors can use the black list adding method or the firewall method to expel the client to simulate the network fault.

[0119] In actual implementation, the MDS stress test unit can also generate other MDS stress test instructions according to actual application requirements to expand the stress test of the MDS component of the Ceph cluster in different scenarios and enrich the stress test scenarios of the MDS component of the Ceph cluster.

[0120] The Mon component is mainly responsible for maintaining the overall state of the cluster. When the OSD scale is large, more OSDs will request the full or incremental OSDMap from the Mon. At the same time, when the OSDMap changes, the connected client will also request the Mon to obtain the new OSDMap. Therefore, the large OSDMap and the frequently changing OSDMap can increase the load of the Mon.

[0121] The information of the OSDMap includes the Ceph cluster ID, the latest version number since creation and the last modification time, the cluster flag, the storage pool information, and the information of the OSD. The storage pool information includes the storage pool name, ID, type, replica level, snapshot, and blacklist information. The information of the OSD includes the number, state, weight, and OSD node information.

[0122] Specifically, the process of generating the Mon stress test instruction by the Mon stress test unit includes:

[0123] The first Mon stress test instruction is generated to increase the size of the OSDMap in the Ceph cluster to be tested. After responding to the first Mon stress test instruction, the plurality of stressors can first increase the size of the OSDMap to a preset value by adding the blacklist when the OSD resource is insufficient, so as to quickly simulate the OSDMap in the case of a large number of OSDs.

[0124] The second Mon stress test instruction is used to increase the read-write load of the Mon component of the Ceph cluster to be tested. After responding to the second Mon stress test instruction, the plurality of stress machines can create an rbd volume during the stress test and frequently create and delete snapshots to achieve frequent changes of the OSDMap, that is, to increase the write load of the Mon. During the snapshot creation and deletion, the FIO can be used to perform read-write operations on the rbd volume. At this time, not only the Mon is stressed, but also the OSD is stressed. Then, a large number of clients are simulated to obtain the OSDMap to increase the read load of the Mon.

[0125] In actual implementation, the Mon stress test unit can be configured to generate other Mon stress test instructions according to actual application requirements, to expand the stress test of the Mon component of the Ceph cluster in different scenarios and enrich the stress test scenarios of the Mon component of the Ceph cluster.

[0126] The main function of the Mgr component is to share and expand part of the functions of the Mon, to reduce the burden of the Mon, and to monitor and obtain cluster state information. Therefore, when the Mon component is stressed, the Mgr is indirectly subjected to a certain load.

[0127] In addition, the Mgr stress test unit can generate a Mgr stress test instruction, which is used for the plurality of stress machines to batch obtain the cluster state information of the Ceph cluster to be tested, to increase the load of the Mgr component of the Ceph cluster to be tested.

[0128] In actual implementation, the Mgr stress test unit can be configured to generate other Mgr stress test instructions according to actual application requirements, to expand the stress test of the Mgr component of the Ceph cluster in different scenarios and enrich the stress test scenarios of the Mgr component of the Ceph cluster.

[0129] The OSD component is a component for storing data in the Ceph cluster, and the stress test thereof is mainly implemented by FIO, including random read-write and sequential read-write. The OSD stress test unit is necessary during the stress test, and the stress test of other control plane components is accompanied by FIO read-write.

[0130] In the system provided in this embodiment, the OSD stress test unit can also generate random read-write instructions and sequential read-write instructions for testing the OSD component of the Ceph cluster to be tested.

[0131] In actual implementation, the OSD stress test unit can be configured to generate other OSD stress test instructions according to actual application requirements, to expand the stress test of the OSD component of the Ceph cluster in different scenarios and enrich the stress test scenarios of the OSD component of the Ceph cluster.

[0132] S4: The distribution module distributes each stress testing instruction generated by the stress testing module to a plurality of pressure machines.

[0133] After the modules corresponding to different components in the stress testing module generate stress testing instructions, the stress testing instructions are sent to the distribution module, and the distribution module distributes each stress testing instruction to a plurality of pressure machines that have been connected to the Ceph cluster to be tested. The pressure machines execute a stress testing process on the Ceph cluster to be tested in response to any stress testing instruction.

[0134] S5: In response to the stress testing instruction, the plurality of pressure machines sends a stress testing instance corresponding to the stress testing instruction to the Ceph cluster to be tested.

[0135] For example, for the first MDS stress testing instruction, the plurality of pressure machines mounts a large number of clients and reads and writes on the Ceph cluster to be tested in response to the first MDS stress testing instruction, i.e., simultaneously mounts a plurality of directories on the pressure machines for reading and writing.

[0136] S6: The monitoring module is connected to the Ceph cluster to be tested and monitors cluster state information of the Ceph cluster to be tested during the stress testing process and component states of each component in the Ceph cluster to be tested.

[0137] Specifically, the monitoring module acquires the cluster state information of the Ceph cluster to be tested in real time, judges whether it is in a healthy state, and the Ceph cluster is in a healthy state HEALTH_OK under normal circumstances. When the component state of any component is abnormal during the stress testing process, the cluster state information of the Ceph cluster becomes HEALTH_WARN or HEALTH_ERR.

[0138] When the Ceph cluster to be tested is not in the healthy state HEAKTH_OK, the error information of the cluster state information is extracted. If any component is down, it can be considered that the component has a problem or an abnormality.

[0139] In addition, the control plane components are generally deployed in a high-availability manner, so the monitoring module also needs to monitor the election state of the Mon component, the MDS component, and the Mgr component. When these components have a master switching behavior, it can be considered that the corresponding component has a risk.

[0140] For example, the plurality of Mon components adopt a master-slave mode. If the master Mon component crashes due to excessive pressure during the stress testing process, the plurality of Mon components will select a new master Mon component. When the election state between the plurality of Mon components changes, it indicates that the previous master Mon component has an abnormality.

[0141] The MDS component and the Mgr component both adopt a master-backup mode, if the master MDS component or the master Mgr component collapses due to excessive pressure during the stress test, the standby MDS component or the standby Mgr component will serve as the working component, that is, the master switching behavior occurs, and it can be considered that the corresponding component has a risk.

[0142] S7: The collection and analysis module collects any stress test data of the Ceph cluster to be tested during the stress test.

[0143] The embodiment provides a stress test system and a stress test method of a Ceph cluster, and the system mainly comprises a distribution module, a stress test module, a collection and analysis module and a monitoring module. The stress test method of the Ceph cluster based on the stress test system of the Ceph cluster first acquires connection configuration information of the Ceph cluster, and then distributes the connection configuration information to each stressor by the distribution module so that each stressor is connected with the Ceph cluster.

[0144] Then the stress test module generates specific stress test instructions, and the instructions are distributed to each stressor by the distribution module, and then the stressor sends the stress test instructions to the Ceph cluster to start the stress test.

[0145] During the stress test, the Ceph cluster is continuously monitored by the monitoring module, and the collection and analysis module collects the data generated by the stress test.

[0146] The system and method provided in the embodiment have at least the following beneficial effects:

[0147] 1. The automatic stress test process of the Ceph cluster is realized, the trouble and tediousness of manual stress test by manpower are avoided, time and labor are saved, and more comprehensive automatic stress test of the Ceph cluster can be realized;

[0148] 2. The stress test instructions generated by the stress test module are directed to multiple components of the Ceph cluster, compared with the existing Ceph stress test method, not only the storage component OSD is stress tested, but also the control plane component Mon component, Mgr component and MDS component are effectively stress tested, the lack of stress test on the control plane component in the current stress test process is made up, and the Ceph cluster to be tested is more accurately stress tested;

[0149] 3、can be extended according to the actual application needs of the stress test module stress test instruction, including extending the stress test instruction of the MDS component of the Ceph cluster in different scenarios, extending the stress test instruction of the Mon component of the Ceph cluster in different scenarios, extending the stress test instruction of the Mgr component of the Ceph cluster in different scenarios, and extending the stress test instruction of the OSD component of the Ceph cluster in different scenarios, so that the system can perform more comprehensive stress testing on the Ceph cluster based on more abundant stress test instructions, and further perform more accurate stress testing on the Ceph cluster to be tested.

[0150] Referring to Figure 5 , a schematic diagram of a computer device provided by an embodiment of the application is shown, which includes at least one processor 501 and a memory 502, the memory 502 stores a computer program capable of running on the processor 501, wherein the processor 501 executes the computer program to execute the stress testing method of the Ceph cluster provided by the embodiment.

[0151] When the processor 501 executes the computer program to execute the stress testing method of the Ceph cluster provided by the embodiment, it specifically includes executing: distributing the connection configuration information of the Ceph cluster to be tested or any stress test instruction to a plurality of stress machines, the connection configuration information is used for the plurality of stress machines to establish a connection with the Ceph cluster to be tested, and the stress machine executes a stress test process on the Ceph cluster to be tested in response to any stress test instruction; generating a stress test instruction for the Ceph cluster to be tested, the stress test instruction includes at least one of a Mon stress test instruction, a Mgr stress test instruction, a MDS stress test instruction and an OSD stress test instruction; monitoring the cluster state information of the Ceph cluster to be tested in the stress test process and the component state of each component in the Ceph cluster to be tested.

[0152] Referring to Figure 6 , a schematic diagram of a non-volatile readable storage medium provided by an embodiment of the application is shown, the non-volatile readable storage medium 600 stores a computer program 601, wherein the computer program 601 is executed by a processor to execute the stress testing method of the Ceph cluster provided by the embodiment.

[0153] The computer program 601 is executed by the processor to perform the stress testing method of the Ceph cluster, and specifically comprises the following steps: distributing connection configuration information of a Ceph cluster to be tested or any stress testing instruction to a plurality of stress machines, the connection configuration information being used for the plurality of stress machines to establish a connection with the Ceph cluster to be tested, and the stress machines performing a stress testing process on the Ceph cluster to be tested in response to any stress testing instruction; generating a stress testing instruction for the Ceph cluster to be tested, the stress testing instruction comprising at least one of a Mon stress testing instruction, a Mgr stress testing instruction, a MDS stress testing instruction, and an OSD stress testing instruction; and monitoring cluster state information of the Ceph cluster to be tested in the stress testing process and component states of various components in the Ceph cluster to be tested.

[0154] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0155] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device or computer program product. Therefore, the embodiments of the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.

[0156] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The device that implements the function specified in one flow or multiple flows and / or blocks. Figure 1 The device that implements the function specified in one flow or multiple flows and / or blocks.

[0157] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including instruction devices, which implement the flowcharts and / or block diagrams. Figure 1 The device that implements the function specified in one flow or multiple flows and / or blocks.Figure 1 the function specified in the one or more blocks.

[0158] These computer program instructions can also be loaded into computer or other programmable data processing terminal devices, so that a series of operation steps are performed on the computer or other programmable terminal devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable terminal devices provide processes for implementing the flowcharts Figure 1 the flowcharts or multiple flowcharts and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0159] Although the preferred embodiments of the present application have been described, those skilled in the art who understand the basic inventive concept after getting to know the present application can make additional changes and modifications to the embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0160] Finally, it should be noted that, in the present text, the relational terms such as first and second and the like are used merely to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between or among the entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or terminal device including a list of elements does not include only those elements but also other elements not expressly listed or other elements inherent in such process, method, article, or terminal device. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or terminal device including the element.

[0161] The principles and implementation manners of the present application are described herein by applying specific examples; the above descriptions of the embodiments are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application scopes will have changes; in conclusion, the content of the present description should not be understood as limiting the present application.

Claims

1. A stress testing system for a Ceph cluster, the system comprising: The system comprises: a distribution module configured to distribute connection configuration information of a Ceph cluster to be tested or any stress testing instruction to a plurality of stress testers, the connection configuration information being used for the plurality of stress testers to establish a connection with the Ceph cluster to be tested, and the plurality of stress testers performing a stress testing process on the Ceph cluster to be tested in response to any stress testing instruction; a stress testing module configured to generate a stress testing instruction for the Ceph cluster to be tested, the stress testing instruction comprising at least one of a Mon stress testing instruction, a Mgr stress testing instruction, a MDS stress testing instruction, and an OSD stress testing instruction; a monitoring module configured to monitor cluster state information of the Ceph cluster to be tested in a stress testing process and component states of various components in the Ceph cluster to be tested; the stress testing module comprises a MDS stress testing unit configured to generate a first MDS stress testing instruction, a second MDS stress testing instruction, a third MDS stress testing instruction, and a fourth MDS stress testing instruction for testing a MDS component of the Ceph cluster to be tested; wherein the first MDS stress testing instruction is used for the plurality of stress testers to mount a target number of directory read-write threads; the second MDS stress testing instruction is used for the plurality of stress testers to create a first calibration number of first files under the mounted directory; the third MDS stress testing instruction is used for the plurality of stress testers to create a second calibration number of second files under the mounted directory; and the fourth MDS stress testing instruction is used for the plurality of stress testers to simulate network fault recovery and access the MDS component of the Ceph cluster to be tested; the first files are files with a size less than a first threshold value; and the second files are files with a size greater than a second threshold value.

2. The system of claim 1, wherein, the stress testing module comprises a Mgr stress testing unit configured to generate a Mgr stress testing instruction for testing a Mgr component of the Ceph cluster to be tested; the Mgr stress testing instruction is used for the plurality of stress testers to batch acquire cluster state information of the Ceph cluster to be tested, so as to increase a load on the Mgr component of the Ceph cluster to be tested.

3. The system of claim 2, wherein, the stress testing module comprises a Mon stress testing unit configured to generate a first Mon stress testing instruction and a second Mon stress testing instruction for testing a Mon component of the Ceph cluster to be tested; wherein the first Mon stress testing instruction is used for increasing a size of an OSDMap in the Ceph cluster to be tested; and the second Mon stress testing instruction is used for increasing a read-write load of the Mon component of the Ceph cluster to be tested.

4. The system of claim 3, wherein, the stress testing module comprises an OSD stress testing unit configured to generate a random read-write instruction and a sequential read-write instruction for testing an OSD component of the Ceph cluster to be tested.

5. The system of claim 1, wherein, The monitoring module is further configured to, after acquiring the cluster state information, analyze error information of the Ceph cluster to be tested in the stress testing process according to the cluster state information when the cluster state information indicates that the Ceph cluster to be tested is not in a healthy state.

6. The system of claim 1, wherein, The system further comprises: The collection analysis module is configured to collect any stress testing data of the Ceph cluster to be tested during the stress testing process. 7.A stress testing method of a Ceph cluster, characterized in that, The method comprises: connection configuration information of the Ceph cluster to be tested or any stress testing instruction is distributed to the plurality of stress testing machines, the connection configuration information being used for the plurality of stress testing machines to establish a connection with the Ceph cluster to be tested, and the stress testing machines perform a stress testing process on the Ceph cluster to be tested in response to any stress testing instruction; a stress testing instruction for the Ceph cluster to be tested is generated, the stress testing instruction comprising at least one of a Mon stress testing instruction, a Mgr stress testing instruction, a MDS stress testing instruction and an OSD stress testing instruction, wherein the MDS stress testing instruction comprises a first MDS stress testing instruction, a second MDS stress testing instruction, a third MDS stress testing instruction and a fourth MDS stress testing instruction generated by a MDS stress testing unit of the stress testing module to test a MDS component of the Ceph cluster to be tested, the first MDS stress testing instruction being used for the plurality of stress testing machines to mount a target number of directory read-write threads, the second MDS stress testing instruction being used for the plurality of stress testing machines to create a first calibration number of first files under the mounted directory, the third MDS stress testing instruction being used for the plurality of stress testing machines to create a second calibration number of second files under the mounted directory, and the fourth MDS stress testing instruction being used for the plurality of stress testing machines to simulate network fault recovery to access the MDS component of the Ceph cluster to be tested, the first files being files with a size less than a first threshold value, and the second files being files with a size greater than a second threshold value; cluster state information of the Ceph cluster to be tested during the stress testing process and component states of each component in the Ceph cluster to be tested are monitored.

8. A computer device, comprising: The method comprises: at least one processor and a memory, the memory storing a computer program capable of running on the processor, wherein the processor executes the computer program to execute the stress testing method of the Ceph cluster according to claim 7.

9. A non-volatile readable storage medium, characterized by The non-volatile readable storage medium stores a computer program, wherein the computer program is executed by a processor to execute the stress testing method of the Ceph cluster according to claim 7.

Images