Pressure test system, method and product of Ceph cluster

Abstract

The invention provides a pressure test system, method and product of a Ceph cluster, and belongs to the technical field of pressure test. The system comprises a distribution module which is used for distributing connection configuration information of a Ceph cluster to be tested or any pressure testing instruction to a plurality of press machines; the voltage measurement module is used for generating a voltage measurement instruction for a to-be-tested Ceph cluster, and the voltage measurement instruction comprises at least one of a Mon voltage measurement instruction, an Mgr voltage measurement instruction, an MDS voltage measurement instruction and an OSD voltage measurement instruction; the monitoring module is used for monitoring cluster state information of the to-be-tested Ceph cluster in the pressure testing process and the component state of each component in the to-be-tested Ceph cluster. The invention aims to improve the accuracy of the pressure test of the Ceph cluster.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of testing, and in particular, to a Ceph cluster stress testing system, method, and product. Background Art

[0002] In the era of booming cloud computing, Ceph has been widely used as a huge distributed system. Ceph is a distributed unified storage platform that integrates traditional block, file storage and object storage. It has high performance, reliability and scalability and is suitable for unstructured data. Ceph includes many components, such as storage components OSD and control plane components. Among them, control plane components include Mon, MDS and Mgr, etc. Usually, the number of control plane components is generally greater than 1 and is an odd number, such as 3 or 5 control plane components.

[0003] Among them, the OSD component is the component used to store data in the Ceph cluster; the MDS component is the metadata server in Ceph, which mainly provides metadata management services for the file system; the Mon component is mainly responsible for maintaining the overall status of the cluster; the main function of the Mgr component is to share and expand some of the functions of Mon and reduce the burden on Mon.

[0004] In order to ensure the high availability of Ceph, it is very necessary to perform automated stress testing on the Ceph cluster, which helps to determine the performance ceiling and possible problems of Ceph cluster components. However, the current stress testing methods for Ceph are mostly tests on the storage component OSD, such as using tools such as FIO and Rados Bench to test disk performance. However, this testing method ignores the stress testing of other control plane components in Ceph, resulting in incomplete testing of Ceph and inaccurate stress testing results for Ceph. Summary of the invention

[0005] The embodiments of the present application provide a Ceph cluster stress testing system, method, and product, aiming to improve the accuracy of Ceph cluster stress testing.

[0006] In a first aspect, an embodiment of the present application provides a Ceph cluster stress testing system, the system comprising: A distribution module, used to distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple presses, wherein the connection configuration information is used for the multiple presses to establish a connection with the Ceph cluster to be tested, and the multiple presses perform a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction; A stress testing module, used to generate a stress testing instruction for the Ceph cluster to be tested, wherein the stress testing instruction includes at least one of a Mon stress testing instruction, an Mgr stress testing instruction, an MDS stress testing instruction, and an OSD stress testing instruction; The monitoring module is used to monitor the cluster status information of the Ceph cluster to be tested during the stress test and the component status of each component in the Ceph cluster to be tested.

[0007] Optionally, the stress testing module includes an MDS stress testing unit, and the MDS stress testing unit is used to generate a first MDS stress testing instruction, a second MDS stress testing instruction, a third MDS stress testing instruction, and a fourth MDS stress testing instruction for testing the MDS component of the Ceph cluster to be tested; Among them, the first MDS stress test instruction is used for the multiple presses to mount the target number of directory read and write threads; the second MDS stress test instruction is used for the multiple presses to create a first calibrated number of first files under the mounted directory; the third MDS stress test instruction is used for the multiple presses to create a second calibrated number of second files under the mounted directory; the fourth MDS stress test instruction is used for the multiple presses to access the MDS components of the Ceph cluster to be tested after simulating network failure recovery; The first file is a file whose size is smaller than a first threshold; the second file is a file whose size is larger than a second threshold.

[0008] Optionally, the stress testing module includes an Mgr stress testing unit, and the Mgr stress testing unit is used to generate an Mgr stress testing instruction for testing the Mgr component of the Ceph cluster to be tested; The Mgr stress test instruction is used by the multiple stress machines to batch obtain cluster status information of the Ceph cluster to be tested, so as to increase the load on the Mgr component of the Ceph cluster to be tested.

[0009] Optionally, the stress testing module includes a Mon stress testing unit, and the Mon stress testing unit is used to generate a first Mon stress testing instruction and a second Mon stress testing instruction for testing the Mon component of the Ceph cluster to be tested; Among them, the first Mon stress testing instruction is used to increase the size of OSDMap in the Ceph cluster to be tested; the second Mon stress testing instruction is used to increase the read and write load of the Mon component of the Ceph cluster to be tested.

[0010] Optionally, the stress testing module includes an OSD stress testing unit, and the OSD stress testing unit is used to generate random read and write instructions and sequential read and write instructions for testing the OSD components of the Ceph cluster to be tested.

[0011] Optionally, the monitoring module is further used to, after acquiring the cluster status information, parse error information of the Ceph cluster to be tested during the stress testing process according to the cluster status information when the cluster status information indicates that the Ceph cluster to be tested is not in a healthy state.

[0012] Optionally, the system further comprises: The collection and analysis module is used to collect any stress testing data of the Ceph cluster to be tested during the stress testing process.

[0013] In a second aspect, an embodiment of the present application provides a Ceph cluster stress testing method, the method comprising: Distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple presses, wherein the connection configuration information is used for the multiple presses to establish a connection with the Ceph cluster to be tested, and the presses perform a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction; Generate a stress test instruction for the Ceph cluster to be tested, wherein the stress test instruction includes at least one of a Mon stress test instruction, an Mgr stress test instruction, an MDS stress test instruction, and an OSD stress test instruction; Monitor the cluster status information of the Ceph cluster to be tested during the stress test and the component status of each component in the Ceph cluster to be tested.

[0014] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the stress testing method of the Ceph cluster described in the first aspect of the embodiment is performed.

[0015] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the stress testing method of the Ceph cluster described in the embodiment is performed.

[0016] Beneficial effects: The Ceph cluster stress testing system provided in this embodiment includes a distribution module, a stress testing module and a monitoring module.

[0017] The distribution module is used to distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple stress machines. The connection configuration information is used for the multiple stress machines to establish a connection with the Ceph cluster to be tested. After the multiple stress machines respond to any stress testing instruction, they perform a stress testing process on the Ceph cluster to be tested.

[0018] The stress testing module is used to generate stress testing instructions for the Ceph cluster to be tested, and the stress testing instructions include at least one of the Mon stress testing instructions, the Mgr stress testing instructions, the MDS stress testing instructions and the OSD stress testing instructions; the monitoring module is used to monitor the cluster status information of the Ceph cluster to be tested during the stress testing process and the component status of each component in the Ceph cluster to be tested.

[0019] Compared with the existing stress testing methods for Ceph, this system performs targeted stress testing on multiple components in the Ceph cluster when stress testing the Ceph cluster. In addition to testing the OSD component, it can also test the Mon component, Mgr component and MDS component, which makes up for the lack of stress testing on the control plane components in the current stress testing process. It can fully automate the stress testing of the Ceph cluster and perform more accurate stress testing on the Ceph cluster to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 It is a structural diagram of a Ceph cluster stress testing system proposed in an embodiment of the present application; Figure 2 It is a structural diagram of a Ceph cluster stress testing system proposed in an embodiment of the present application; Figure 3 It is a flowchart of the steps of a Ceph cluster stress testing method provided in one embodiment of the present application; Figure 4 It is an execution diagram of a Ceph cluster stress testing method proposed in an embodiment of the present application; Figure 5 is a schematic diagram of a computer device proposed in one embodiment of the present application; Figure 6 It is a schematic diagram of a non-volatile readable storage medium provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] Ceph - distributed file system. Mon: monitor, monitor; MDS: metadata service, metadata server; Mgr: Manager Daemon, Manager daemon process; OSD: Object Storage Daemon, object storage device; ID: Identity document, identification; Automated stress testing: Stress testing is a process that continuously applies pressure to the system to monitor its performance under certain pressure. It can examine the maximum load that the system can withstand under the current software and hardware environment and help identify system bottlenecks. Automated stress testing is the process of automating manual stress testing. It is usually completed using automated methods such as scripts. FIO: is an I / O tool used to stress test and verify hardware; Rados Bench: A performance benchmark tool that comes with the Ceph cluster and is used to perform performance testing on the cluster; it supports three types of tests: write tests and two types of read tests, such as sequential reads and random reads.

[0024] Ceph is a distributed unified storage platform that integrates traditional block, file storage and object storage. It has high performance, reliability and scalability and is suitable for unstructured data. Ceph is a software-defined storage that can run on mainstream Linux distributions and other Uinx-like distributions. Ceph includes storage components and control plane components, where the storage component refers to OSD, and the control plane components include Mon components, MDS components and Mgr components.

[0025] In the era of booming cloud computing, Ceph is widely used. As a huge distributed system, Ceph has many components, such as storage components OSD and control plane components. Control plane components include Mon, MDS and Mgr. In order to ensure high availability, the number of control plane components is generally greater than 1 and is an odd number, such as 3 or 5 control plane components.

[0026] In order to ensure the high availability of Ceph, it is necessary to perform automated stress testing on the Ceph cluster, which helps to determine the performance limits and possible problems of Ceph cluster components.

[0027] Currently, most stress testing methods for Ceph focus on testing the storage component OSD, such as using tools such as FIO and Rados Bench to test disk performance. However, this testing method ignores the stress testing of other control plane components in Ceph, resulting in incomplete testing of Ceph.

[0028] Therefore, the embodiment of the present application provides a stress testing system for a Ceph cluster, which can not only test the OSD component, but also test the Mon component, Mgr component and MDS component, making up for the lack of stress testing of control plane components in the current stress testing process, and can comprehensively perform automated stress testing on the Ceph cluster, and perform more accurate stress testing on the Ceph cluster to be tested.

[0029] Reference Figure 1 , showing a schematic diagram of the structure of a Ceph cluster stress testing system provided in an embodiment of the present application, the system includes a distribution module, a stress testing module and a monitoring module.

[0030] The distribution module is used to distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple stress machines.

[0031] The connection configuration information is used for the multiple stressors to establish a connection with the Ceph cluster to be tested, and the connection configuration information includes keyring and conf configuration information, etc.; the multiple stressors perform a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction.

[0032] The stress testing module is used to generate stress testing instructions for the Ceph cluster to be tested, and the stress testing instructions include at least one of Mon stress testing instructions, Mgr stress testing instructions, MDS stress testing instructions and OSD stress testing instructions.

[0033] The monitoring module is used to monitor the cluster status information of the Ceph cluster to be tested during the stress test and the component status of each component in the Ceph cluster to be tested.

[0034] In a feasible implementation manner, the stress testing module includes an MDS stress testing unit, which is used to generate a stress testing instruction for stress testing the MDS component of the Ceph cluster to be tested.

[0035] Specifically, the MDS stress testing unit is used to generate a first MDS stress testing instruction, a second MDS stress testing instruction, a third MDS stress testing instruction and a fourth MDS stress testing instruction for testing the MDS component of the Ceph cluster to be tested.

[0036] The first MDS stress test instruction is used for the directory reading and writing threads of the target number of the multiple presses to mount.

[0037] The second MDS pressure test instruction is used for the multiple presses to create a first calibrated number of first files in the mounted directory.

[0038] The third MDS pressure test instruction is used by the multiple presses to create a second calibrated number of second files in the mounted directory.

[0039] The fourth MDS stress test instruction is used for the multiple stress machines to access the MDS component of the Ceph cluster to be tested after simulating network failure recovery; the first file is a file whose size is smaller than a first threshold; the second file is a file whose size is larger than a second threshold; that is, the first file is a small file, and the second file is a large file, and the first threshold and the second threshold can be set in a range according to the needs of actual applications.

[0040] In the actual implementation process, the MDS stress testing unit can also be configured to generate other MDS stress testing instructions according to the needs of actual applications, thereby expanding the stress testing of the MDS components of the Ceph cluster in different scenarios.

[0041] In a feasible implementation manner, the stress testing module includes an Mgr stress testing unit, which is used to generate a stress testing instruction for stress testing the Mgr component of the Ceph cluster to be tested.

[0042] The Mgr stress testing unit is used to generate an Mgr stress testing instruction for testing the Mgr component of the Ceph cluster to be tested.

[0043] The Mgr stress test instruction is used by the multiple stress machines to batch obtain cluster status information of the Ceph cluster to be tested, so as to increase the load on the Mgr component of the Ceph cluster to be tested.

[0044] In the actual implementation process, the Mgr stress testing unit can also be configured to generate other Mgr stress testing instructions according to the needs of actual applications, thereby expanding the stress testing of the Mgr components of the Ceph cluster in different scenarios.

[0045] In a feasible implementation manner, the stress testing module includes a Mon stress testing unit, which is used to generate a stress testing instruction for stress testing the Mon component of the Ceph cluster to be tested.

[0046] The Mon stress testing unit is used to generate a first Mon stress testing instruction and a second Mon stress testing instruction for testing the Mon component of the Ceph cluster to be tested.

[0047] The first Mon stress test instruction is used to increase the size of OSDMap in the Ceph cluster to be tested.

[0048] The second Mon stress testing instruction is used to increase the read and write load of the Mon component of the Ceph cluster to be tested.

[0049] In the actual implementation process, the Mon stress testing unit can also be configured to generate other Mon stress testing instructions according to the needs of actual applications, thereby expanding the stress testing of the Mon component of the Ceph cluster in different scenarios.

[0050] In a feasible implementation, the stress testing module includes an OSD stress testing unit, which is used to generate an instruction for stress testing the OSD component of the Ceph cluster to be tested.

[0051] The OSD stress testing unit is used to generate random read and write instructions and sequential read and write instructions for testing the OSD components of the Ceph cluster to be tested.

[0052] In the actual implementation process, the OSD stress testing unit can also be configured to generate other OSD stress testing instructions according to the needs of actual applications, thereby expanding the stress testing of OSD components of the Ceph cluster in different scenarios.

[0053] In a feasible implementation manner, the monitoring module is further used to, after acquiring the cluster status information, parse error information of the Ceph cluster to be tested during the stress testing process according to the cluster status information when the cluster status information indicates that the Ceph cluster to be tested is not in a healthy state.

[0054] When the system provided in this embodiment performs stress testing on the Ceph cluster, multiple components in the Ceph cluster are subjected to targeted stress testing respectively. In addition to testing the OSD component, the Mon component, Mgr component, and MDS component can also be tested, which makes up for the lack of stress testing on the control plane components in the current stress testing process, and can fully automate the stress testing of the Ceph cluster, so as to perform more accurate stress testing on the Ceph cluster to be tested.

[0055] Reference Figure 2 , showing a structural schematic diagram of a Ceph cluster stress testing system provided in an embodiment of the present application, the system includes a distribution module, a stress testing module, a monitoring module and a collection and analysis module, which are used to collect any stress testing data of the Ceph cluster to be tested during the stress testing process.

[0056] Among them, the distribution module is used to distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple stress machines; the stress testing module includes a stress testing unit that generates stress testing instructions for different components of the Ceph cluster to be tested, and the stress testing module can specifically include: an MDS stress testing unit for generating stress testing instructions for stress testing the MDS component of the Ceph cluster to be tested, an Mgr stress testing unit for generating stress testing instructions for stress testing the Mgr component of the Ceph cluster to be tested, a stress testing unit for generating stress testing instructions for stress testing the Mon component of the Ceph cluster to be tested, and an OSD stress testing unit for generating instructions for stress testing the OSD component of the Ceph cluster to be tested.

[0057] When the system provided in this embodiment performs stress testing on the Ceph cluster, in addition to testing the OSD component, it can also test the Mon component, the Mgr component, and the MDS component, which makes up for the lack of stress testing on the control plane components in the current stress testing process, and can fully automate the stress testing of the Ceph cluster, and perform more accurate stress testing on the Ceph cluster to be tested. In addition, any stress testing data of the Ceph cluster to be tested in the stress testing process can be summarized and stored through the collection and analysis module. When storing, it can be classified and stored according to the cluster status information of the Ceph cluster, the component status of each component, etc., to facilitate the subsequent call of stress testing data to analyze the performance of the Ceph cluster.

[0058] Reference Figure 3 , shows a flowchart of the steps of a Ceph cluster stress testing method in an embodiment of the present application. In a feasible implementation, the method may specifically include the following steps: S101: Distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple stress machines.

[0059] The connection configuration information is used for the multiple stressors to establish a connection with the Ceph cluster to be tested, and the stressors perform a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction.

[0060] S102: Generate a stress test instruction for the Ceph cluster to be tested; the stress test instruction includes at least one of a Mon stress test instruction, an Mgr stress test instruction, an MDS stress test instruction, and an OSD stress test instruction.

[0061] Specifically, the stress testing module in the system provided in this embodiment includes an MDS stress testing unit, an Mgr stress testing unit, a Mon stress testing unit, and an OSD stress testing unit.

[0062] The MDS stress testing unit generates a first MDS stress testing instruction, a second MDS stress testing instruction, a third MDS stress testing instruction, and a fourth MDS stress testing instruction for testing the MDS component of the Ceph cluster to be tested.

[0063] The first MDS stress testing instruction may enable the multiple stress machines to mount a target number of directory read and write threads.

[0064] The second MDS stress test instruction can enable the multiple presses to create a first calibrated number of first files in the mounted directory.

[0065] The third MDS stress test instruction may enable the multiple presses to create a second calibrated number of second files under the mounted directory.

[0066] The fourth MDS stress test instruction allows the multiple stress machines to access the MDS component of the Ceph cluster to be tested after simulating network failure recovery; the first file is a file whose size is smaller than a first threshold; and the second file is a file whose size is larger than a second threshold.

[0067] The Mgr stress testing unit generates an Mgr stress testing instruction for testing the Mgr component of the Ceph cluster to be tested.

[0068] The Mgr stress test instruction may enable the multiple stress machines to batch obtain cluster status information of the Ceph cluster to be tested, so as to increase the load on the Mgr component of the Ceph cluster to be tested.

[0069] The Mon stress testing unit may generate a first Mon stress testing instruction and a second Mon stress testing instruction for testing the Mon component of the Ceph cluster to be tested.

[0070] The first Mon stress testing instruction may increase the size of the OSDMap in the Ceph cluster to be tested.

[0071] The second Mon stress testing instruction may increase the read and write load of the Mon component of the Ceph cluster to be tested.

[0072] The OSD stress testing unit may generate random read and write instructions and sequential read and write instructions for testing the OSD components of the Ceph cluster to be tested.

[0073] S103: Monitor the cluster status information of the Ceph cluster to be tested during the stress test and the component status of each component in the Ceph cluster to be tested.

[0074] Specifically, the cluster status information and the component status of each component in the Ceph cluster to be tested are obtained in real time to determine whether the Ceph cluster to be tested is in a healthy state. If it is not in a healthy state, the error information of the Ceph cluster to be tested is further determined and stored.

[0075] Reference Figure 4 , shows an execution diagram of a Ceph cluster stress testing method provided by an embodiment of the present application. In a feasible implementation, the execution process of the method may specifically include the following steps: S1: Get the connection configuration information of the Ceph cluster to be tested.

[0076] The connection configuration information of the Ceph cluster to be tested may include configuration information for connecting to the Ceph cluster. For example, the connection configuration information may be keyring and conf configuration information of the Ceph cluster to be tested.

[0077] In the actual implementation process, the connection configuration information can be read in the Ceph cluster to be tested, and then the connection configuration information of the Ceph cluster to be tested is sent or uploaded to the distribution module of the stress testing system of the Ceph cluster. Other methods can also be used to allow the distribution module to receive the connection configuration information.

[0078] S2: The distribution module distributes the connection configuration information of the Ceph cluster to be tested to each press, and any press connects to the Ceph cluster to be tested according to the connection configuration information.

[0079] Reference Figure 5 , showing a stress testing schematic diagram of the stress testing system of the Ceph cluster provided in an embodiment of the present application. The stress testing system of the Ceph cluster can be configured with a stress machine node, and multiple stress machines are configured in the stress machine node, such as stress machines 1-n; multiple stress machines can be combined to complete the stress testing process of any component, providing more abundant stress instances to increase the stress testing pressure on the Ceph cluster.

[0080] When the distribution module distributes the connection configuration information of the Ceph cluster to be tested to each press, each press can connect to the Ceph cluster to be tested according to the connection configuration information.

[0081] S3: The stress testing module generates a stress testing instruction for the Ceph cluster to be tested.

[0082] In this embodiment, the stress testing module includes an MDS stress testing unit for the MDS component, an Mgr stress testing unit for the Mgr component, a Mon stress testing unit for the Mon component, and an OSD stress testing unit for the OSD component.

[0083] In the stress testing process of actual applications, not only can the Ceph cluster to be tested be jointly stress tested based on the stress testing modules of each component, but also a more comprehensive stress testing result of the Ceph cluster to be tested can be obtained.

[0084] Alternatively, in response to the stress test setting operation, a component to be tested in this stress test process may be selected, and targeted stress testing may be performed according to the stress test unit corresponding to the determined component to be tested.

[0085] The stress testing environment of any component to be tested can also be determined in response to the stress testing environment configuration, and the stress testing unit corresponding to the component to be tested generates stress testing instructions adapted to the stress testing environment. That is, in actual applications, a stress testing module and a stress testing instruction of any stress testing module can be selected to execute the current stress testing process.

[0086] The MDS component is the metadata server in Ceph, which is used to provide metadata management services for the file system. It mainly handles requests involving file / directory metadata, such as creating, deleting, and querying files.

[0087] Specifically, the process of the MDS stress testing unit generating instructions includes: Generate a first MDS stress test instruction for the multiple presses to mount a target number of directory read and write threads. After responding to the first MDS stress test instruction, the multiple presses perform a large number of client mounts and read and write on the Ceph cluster to be tested, that is, mount multiple directories on the press at the same time for reading and writing.

[0088] Generate a second MDS stress testing instruction, which is used for the multiple presses to create a first calibrated number of first files in the mounted directory. The first files are files whose sizes are smaller than a first threshold, i.e., small files. After responding to the second MDS stress testing instruction, the multiple presses perform a large number of small file operations, including creating a large number of small files in the mounted directory and performing operations such as decompression.

[0089] Generate a third MDS stress testing instruction, which is used for the multiple presses to create a second calibrated number of second files in the mounted directory. The second files are files whose sizes are greater than a second threshold, that is, large files. After responding to the third MDS stress testing instruction, the multiple presses perform operations on the large files, including operations such as creating and deleting large files in the mounted directory.

[0090] Generate a fourth MDS stress test instruction for the multiple stress machines to access the MDS component of the Ceph cluster to be tested after simulating network failure recovery. For example, after responding to the fourth MDS stress test instruction, the multiple stress machines can use a blacklist or firewall method to expel the client to simulate a network failure.

[0091] In the actual implementation process, the MDS stress testing unit can also be configured to generate other MDS stress testing instructions according to the needs of actual applications, expand the stress testing of the MDS components of the Ceph cluster in different scenarios, and enrich the stress testing scenarios of the MDS components of the Ceph cluster.

[0092] The Mon component is mainly responsible for maintaining the overall status of the cluster. When the OSD scale is large, more OSDs will ask Mon for full or incremental OSDMaps. At the same time, when the OSDMap changes, the connected clients will also send requests to Mon to obtain the new OSDMap. Therefore, larger OSDMaps and frequently changing OSDMaps can increase the load on Mon.

[0093] The OSDMap information includes the Ceph cluster ID, the latest version number since creation and the last modification time, the cluster flag tag, the storage pool information, and the OSD information; the storage pool information includes the storage pool name, ID, type, replica level, snapshot, and blacklist information, and the OSD information such as the number, status, weight, and OSD node information.

[0094] Specifically, the process of generating instructions by the Mon stress testing unit includes: Generate a first Mon stress test instruction for increasing the size of OSDMap in the Ceph cluster to be tested. After responding to the first Mon stress test instruction, multiple stress machines can first increase the size of OSDMap to a preset value by adding a blacklist when OSD resources are insufficient, thereby quickly simulating the OSDMap under a large number of OSDs.

[0095] A second Mon stress test instruction is generated to increase the read and write load of the Mon component of the Ceph cluster to be tested. After responding to the second Mon stress test instruction, multiple stress machines can create rbd volumes during the stress test process and frequently create and delete snapshots to achieve frequent changes in OSDMap, that is, increase the write load of Mon. During the creation and deletion of snapshots, FIO can also be used to perform read and write operations on the rbd volume. At this time, not only Mon but also OSD is stressed, and then a large number of clients are simulated to obtain OSDMap to increase the read load of Mon.

[0096] In the actual implementation process, the Mon stress testing unit can also be configured to generate other Mon stress testing instructions according to the needs of actual applications, expand the stress testing of the Mon component of the Ceph cluster in different scenarios, and enrich the stress testing scenarios of the Mon component of the Ceph cluster.

[0097] The main function of the Mgr component is to share and expand some of the functions of Mon, reduce the burden on Mon, and monitor and obtain cluster status information. Therefore, when stress testing the Mon component, it will indirectly cause a certain load on the Mgr.

[0098] In addition, the Mgr stress testing unit may also generate an Mgr stress testing instruction for the multiple stress machines to batch obtain cluster status information of the Ceph cluster to be tested, so as to increase the load on the Mgr component of the Ceph cluster to be tested.

[0099] In the actual implementation process, the Mgr stress testing unit can also be configured to generate other Mgr stress testing instructions according to the needs of actual applications, expand the stress testing of the Mgr components of the Ceph cluster in different scenarios, and enrich the stress testing scenarios of the Mgr components of the Ceph cluster.

[0100] The OSD component is a component used to store data in the Ceph cluster. Its stress testing is mainly implemented through FIO, including random read and write and sequential read and write. The OSD stress testing unit is necessary in the stress testing process, and FIO read and write must be accompanied when stress testing other control plane components.

[0101] In the system provided in this embodiment, the OSD stress testing unit may also generate random read and write instructions and sequential read and write instructions for testing the OSD components of the Ceph cluster to be tested.

[0102] In the actual implementation process, the OSD stress testing unit can also be configured to generate other OSD stress testing instructions according to the needs of actual applications, expand the stress testing of OSD components of the Ceph cluster in different scenarios, and enrich the stress testing scenarios of OSD components of the Ceph cluster.

[0103] S4: The distribution module distributes each stress testing instruction generated by the stress testing module to multiple presses.

[0104] After generating stress testing instructions, the modules corresponding to different components in the stress testing module send them to the distribution module, and the distribution module distributes each stress testing instruction to multiple stress machines that have been connected to the Ceph cluster to be tested. After the stress machine responds to any stress testing instruction, it executes the stress testing process on the Ceph cluster to be tested.

[0105] S5: After the multiple stress machines respond to the stress testing instruction, they send the stress testing instance corresponding to the stress testing instruction to the Ceph cluster to be tested.

[0106] For example, for the first MDS stress test instruction, after responding to the first MDS stress test instruction, multiple stress machines perform a large number of client mounts and read and write on the Ceph cluster to be tested, that is, multiple directories are simultaneously mounted on the stress machines for reading and writing.

[0107] S6: The monitoring module is connected to the Ceph cluster to be tested, and monitors the cluster status information of the Ceph cluster to be tested during the stress test and the component status of each component in the Ceph cluster to be tested.

[0108] Specifically, the monitoring module obtains the cluster status information of the Ceph cluster to be tested in real time to determine whether it is in a healthy state. Under normal circumstances, the Ceph cluster is in a healthy state of HEALTH_OK. When any component is abnormal during the stress test, the cluster status information of the Ceph cluster will become HEALTH_WARN or HEALTH_ERR.

[0109] When the Ceph cluster to be tested is not in the healthy state HEAKTH_OK, extract the error information of the cluster status information. If any component is down, it can be considered that there is a problem or abnormality in the component.

[0110] In addition, control plane components are generally deployed with high availability, so the monitoring module also needs to monitor the election status of the Mon component, MDS component, and Mgr component. When these components fail to switch to the primary, it can be considered that the corresponding components are at risk.

[0111] For example, multiple Mon components use a master-slave mode. If the master Mon component crashes due to excessive pressure during the stress test, multiple Mon components will reselect the master Mon component. When the election status between multiple Mon components changes, it indicates that the previous master Mon component is abnormal.

[0112] Both the MDS component and the Mgr component adopt the master-slave mode. If during the stress test, the main MDS component or the main Mgr component crashes due to excessive pressure, the backup MDS component or the backup Mgr component will serve as the working component, that is, the master switching behavior occurs, and it can be considered that the corresponding component is at risk.

[0113] S7: The collection and analysis module collects any stress testing data of the Ceph cluster to be tested during the stress testing process.

[0114] This embodiment proposes a Ceph cluster stress testing system and a Ceph cluster stress testing method, the system mainly includes a distribution module, a stress testing module, a collection and analysis module and a monitoring module. The Ceph cluster stress testing method based on the Ceph cluster stress testing system first obtains the Ceph cluster connection configuration information, and then the distribution module distributes the connection configuration information to each press to connect each press to the Ceph cluster.

[0115] The stress testing module then generates specific stress testing instructions and the distribution module distributes the instructions to each stress machine. The stress machine then sends the stress testing instructions to the Ceph cluster to start stress testing.

[0116] During the stress test, the monitoring module continuously monitors the Ceph cluster, while the collection and analysis module collects the data generated by the stress test.

[0117] The system and method proposed in this embodiment have at least the following beneficial effects: 1. Implement the automated stress testing process for Ceph clusters, avoiding the hassle and tediousness of manual stress testing, which not only saves time and effort, but also enables more comprehensive automated stress testing for Ceph clusters; 2. The stress testing instructions generated by the stress testing module target multiple components of the Ceph cluster. Compared with the existing Ceph stress testing methods, it not only performs stress testing on the storage component OSD, but also performs effective stress testing on the control plane components Mon component, Mgr component, and MDS component, making up for the lack of stress testing on the control plane components in the current stress testing process, and performing more accurate stress testing on the Ceph cluster to be tested; 3. The stress testing instructions of the stress testing module can be expanded according to the needs of actual applications, including expanding the stress testing instructions for the MDS component of the Ceph cluster in different scenarios, expanding the stress testing instructions for the Mon component of the Ceph cluster in different scenarios, expanding the stress testing instructions for the Mgr component of the Ceph cluster in different scenarios, and expanding the stress testing instructions for the OSD component of the Ceph cluster in different scenarios. This allows the system to perform a more comprehensive stress test on the Ceph cluster based on more abundant stress testing instructions, and further perform a more accurate stress test on the Ceph cluster to be tested.

[0118] Reference Figure 5 , shows a schematic diagram of a computer device provided in an embodiment of the present application, including: at least one processor 501, and a memory 502, wherein the memory 502 stores a computer program that can be run on the processor 501, wherein the processor 501 executes the Ceph cluster stress testing method described in the embodiment when executing the computer program.

[0119] When the processor 501 executes the computer program to execute the stress testing method of the Ceph cluster described in the embodiment, it specifically includes the following steps: distributing the connection configuration information or any stress testing instruction of the Ceph cluster to be tested to multiple stress machines, the connection configuration information is used for the multiple stress machines to establish a connection with the Ceph cluster to be tested, and the stress machine performs a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction; generating a stress testing instruction for the Ceph cluster to be tested, the stress testing instruction including at least one of a Mon stress testing instruction, an Mgr stress testing instruction, an MDS stress testing instruction, and an OSD stress testing instruction; monitoring the cluster status information of the Ceph cluster to be tested during the stress testing process and the component status of each component in the Ceph cluster to be tested.

[0120] Reference Figure 6 , showing a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 600 stores a computer program 601, wherein the computer program 601, when executed by a processor, performs the Ceph cluster stress testing method described in the embodiment.

[0121] When the computer program 601 is executed by the processor to perform the stress testing method of the Ceph cluster described in the embodiment, it specifically includes the following steps: distributing the connection configuration information or any stress testing instruction of the Ceph cluster to be tested to multiple stress machines, the connection configuration information is used for the multiple stress machines to establish a connection with the Ceph cluster to be tested, and the stress machine performs a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction; generating a stress testing instruction for the Ceph cluster to be tested, the stress testing instruction including at least one of a Mon stress testing instruction, an Mgr stress testing instruction, an MDS stress testing instruction, and an OSD stress testing instruction; monitoring the cluster status information of the Ceph cluster to be tested during the stress testing process and the component status of each component in the Ceph cluster to be tested.

[0122] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0123] It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0124] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0128] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0129] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A Ceph cluster stress testing system, characterized in that: The system comprises: A distribution module, used to distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple presses, wherein the connection configuration information is used for the multiple presses to establish a connection with the Ceph cluster to be tested, and the multiple presses perform a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction; A stress testing module, used to generate a stress testing instruction for the Ceph cluster to be tested, wherein the stress testing instruction includes at least one of a Mon stress testing instruction, an Mgr stress testing instruction, an MDS stress testing instruction, and an OSD stress testing instruction; The monitoring module is used to monitor the cluster status information of the Ceph cluster to be tested during the stress test and the component status of each component in the Ceph cluster to be tested.

2. The system according to claim 1, characterized in that The stress testing module includes an MDS stress testing unit, and the MDS stress testing unit is used to generate a first MDS stress testing instruction, a second MDS stress testing instruction, a third MDS stress testing instruction, and a fourth MDS stress testing instruction for testing the MDS component of the Ceph cluster to be tested; Among them, the first MDS stress test instruction is used for the multiple presses to mount the target number of directory read and write threads; the second MDS stress test instruction is used for the multiple presses to create a first calibrated number of first files under the mounted directory; the third MDS stress test instruction is used for the multiple presses to create a second calibrated number of second files under the mounted directory; the fourth MDS stress test instruction is used for the multiple presses to access the MDS components of the Ceph cluster to be tested after simulating network failure recovery; The first file is a file whose size is smaller than a first threshold; the second file is a file whose size is larger than a second threshold.

3. The system according to claim 2, characterized in that The stress testing module includes an Mgr stress testing unit, and the Mgr stress testing unit is used to generate an Mgr stress testing instruction for testing the Mgr component of the Ceph cluster to be tested; The Mgr stress test instruction is used by the multiple stress machines to batch obtain cluster status information of the Ceph cluster to be tested, so as to increase the load on the Mgr component of the Ceph cluster to be tested.

4. The system according to claim 3, characterized in that The stress testing module includes a Mon stress testing unit, and the Mon stress testing unit is used to generate a first Mon stress testing instruction and a second Mon stress testing instruction for testing the Mon component of the Ceph cluster to be tested; Among them, the first Mon stress testing instruction is used to increase the size of OSDMap in the Ceph cluster to be tested; the second Mon stress testing instruction is used to increase the read and write load of the Mon component of the Ceph cluster to be tested.

5. The system according to claim 4, characterized in that The stress testing module includes an OSD stress testing unit, and the OSD stress testing unit is used to generate random read and write instructions and sequential read and write instructions for testing the OSD components of the Ceph cluster to be tested.

6. The system according to claim 1, characterized in that The monitoring module is also used to, after acquiring the cluster status information, parse error information of the Ceph cluster to be tested during the stress testing process according to the cluster status information when the cluster status information indicates that the Ceph cluster to be tested is not in a healthy state.

7. The system according to claim 1, characterized in that The system further comprises: The collection and analysis module is used to collect any stress testing data of the Ceph cluster to be tested during the stress testing process.

8. A Ceph cluster stress testing method, characterized in that: The method comprises: Distribute the connection configuration information of the Ceph cluster to be tested or any stress testing instruction to multiple presses, wherein the connection configuration information is used for the multiple presses to establish a connection with the Ceph cluster to be tested, and the presses perform a stress testing process on the Ceph cluster to be tested after responding to any stress testing instruction; Generate a stress test instruction for the Ceph cluster to be tested, wherein the stress test instruction includes at least one of a Mon stress test instruction, an Mgr stress test instruction, an MDS stress test instruction, and an OSD stress test instruction; Monitor the cluster status information of the Ceph cluster to be tested during the stress test and the component status of each component in the Ceph cluster to be tested.

9. A computer device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the processor executes the Ceph cluster stress testing method according to claim 8 when executing the computer program.

10. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the Ceph cluster stress testing method according to claim 8.

Images