Stability test method, device and equipment for distributed storage system and storage medium

Abstract

The invention discloses a stability test method, device and equipment of a distributed storage system and a storage medium, relates to the technical field of cloud computing, is applied to a multi-pool test engine, and comprises the following steps: acquiring an external test request for the distributed storage system and initializing a current multi-pool environment based on the external test request, obtaining a target multi-pool environment; reading a preset multi-pool coding table in the target multi-pool environment to generate a weighted test case set based on the preset multi-pool coding table; the preset multi-pool coding table is a new coding structure obtained by modifying a historical test coding table based on a pool relationship, data distribution and cross-pool operation in the distributed storage system; and creating a corresponding target virtual machine based on the weighted test case set, and starting multiple threads by using the weighted test case set and the target virtual machine to carry out a cross-pool stability test on the target resource pool so as to obtain a corresponding system stability test result. Therefore, the test requirements of the multi-pool architecture of the modern distributed storage system can be met.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method, apparatus, device, and storage medium for stability testing of a distributed storage system. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, distributed storage systems have become a core infrastructure of modern data centers. Distributed storage systems offer advantages such as high availability, high scalability, and high fault tolerance by distributing data across multiple nodes. However, the complexity of distributed storage systems also brings stability challenges, especially in multi-pool architectures where interactions between different storage pools can trigger various potential problems. Currently, commonly used distributed storage testing tools primarily focus on performance testing, lacking systematic testing capabilities for distributed storage stability, particularly for failure scenarios in multi-pool environments. DR-Test (Data-Driven Testing), as a stability testing tool specifically for distributed storage, offers automatic test case generation and automated execution, but its current implementation only addresses single-pool scenarios and cannot meet the testing needs of modern multi-pool distributed storage system architectures.

[0003] It is evident that how to conduct system stability testing for the multi-pool architecture of modern distributed storage systems is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device, and storage medium for stability testing of distributed storage systems, capable of performing system stability testing on modern distributed storage systems with multi-pool architectures. The specific solution is as follows:

[0005] In a first aspect, this invention discloses a stability testing method for a distributed storage system, applied to a multi-pool testing engine, comprising:

[0006] Obtain external test requests for the distributed storage system, and initialize the current multi-pool environment based on the external test requests to obtain the target multi-pool environment;

[0007] In the target multi-pool environment, a preset multi-pool encoding table is read to generate a weighted test case set based on the preset multi-pool encoding table. The preset multi-pool encoding table is a new encoding structure obtained by modifying the historical test encoding table based on the multi-pool related dimensions. The multi-pool related dimensions include the pool relationship, data distribution and cross-pool operation in the distributed storage system.

[0008] Based on the weighted test case set, a corresponding target virtual machine is created, and multi-threaded testing is started using the weighted test case set and the target virtual machine to perform cross-pool stability testing on the target resource pool, so as to obtain the corresponding system stability test results.

[0009] Optionally, obtain external test requests for the distributed storage system, and initialize the current multi-pool environment based on the external test requests to obtain the target multi-pool environment, including:

[0010] Obtain external test requests for the distributed storage system;

[0011] The corresponding pool topology configuration file is loaded based on the external test request, and a corresponding Secure Shell protocol mutual trust channel is established to initialize the current multi-pool environment based on the pool topology configuration file and the Secure Shell protocol mutual trust channel to obtain the target multi-pool environment.

[0012] Optionally, a preset multi-pool encoding table is read to generate a weighted test case set based on the preset multi-pool encoding table, including:

[0013] The preset test case generator reads the preset multi-pool encoding table and generates a weighted test case set based on the external test request and the pool relationships, data distribution and cross-pool operations in the preset multi-pool encoding table.

[0014] Optionally, a corresponding target virtual machine is created based on the weighted test case set, and multi-threaded testing is launched using the weighted test case set and the target virtual machine to perform cross-pool stability testing on the target resource pool, including:

[0015] Based on the weighted test case set and the test case requirements corresponding to external test requests, the data resources of the distributed storage system are allocated, and a corresponding resource allocation table is constructed based on the allocated data resources.

[0016] Create the corresponding target virtual machine based on the resource allocation table, and determine the target resource pool;

[0017] Based on the resource allocation table and the target virtual machine, multiple threads will be launched to perform cross-pool stability testing on the target resource pool.

[0018] Optionally, use a weighted test case set and the target virtual machine to start multiple threads to perform cross-pool stability testing on the target resource pool, including:

[0019] Start a preset fault thread using the target virtual machine, and use fault cases from the weighted test case set to perform cross-pool stability testing on the target resource pool;

[0020] Meanwhile, a preset monitoring thread is started using the target virtual machine to monitor the status of the target resource pool during cross-pool stability testing, and a preset main thread is started to record the status of the target resource pool to generate a resource pool status table.

[0021] Optionally, cross-pool stability testing of the target resource pool is performed using faulty test cases from a weighted test case set; simultaneously, a preset monitoring thread is started using the target virtual machine to monitor the status of the target resource pool during the cross-pool stability test, including:

[0022] The target resource pool is determined based on the fault types corresponding to the fault cases in the weighted test case set. The fault cases include isolated node network cases with a single node in a single pool, batch service stop cases with multiple nodes in a single pool, simulated network partitioning cases with multi-pool collaboration, and cases of cutting off inter-pool links in cross-pool networks.

[0023] Input the failure test cases into the target resource pool to enable cross-pool stability testing;

[0024] Record the initial state of the target resource pool, and simultaneously start a preset monitoring thread based on the target virtual machine to monitor the target resource pool in order to obtain the current state;

[0025] Fault information of the target resource pool is determined based on the data consistency between the initial state and the current state;

[0026] If the target resource pool is in a fault state, the fault information corresponding to the fault state is marked, and the target resource pool is restored based on the adaptive recovery strategy.

[0027] If the target resource pool completes fault recovery, the corresponding resource lock is released and a corresponding fault recovery report is generated;

[0028] If the target resource pool has not completed fault recovery, the adaptive recovery strategy will be adjusted based on the current fault recovery progress until the target resource pool completes fault recovery.

[0029] Optionally, a preset monitoring thread is started based on the target virtual machine to monitor the target resource pool in order to obtain the current status, including:

[0030] Start a preset monitoring thread using the target virtual machine, and based on the preset detection frequency, sequentially obtain node inventory status information, data replica count information, and cross-pool consistency verification information of the target resource pool;

[0031] The current state of the target resource pool is determined based on node inventory status information, data replica count information, and cross-pool consistency verification information.

[0032] Secondly, this application discloses a stability testing device for a distributed storage system, applied to a multi-pool testing engine, comprising:

[0033] The environment initialization module is used to obtain external test requests for the distributed storage system and initialize the current multi-pool environment based on the external test requests to obtain the target multi-pool environment.

[0034] The test case set generation module is used to read the preset multi-pool encoding table in the target multi-pool environment, and generate a weighted test case set based on the preset multi-pool encoding table. The preset multi-pool encoding table is a new encoding structure obtained by modifying the historical test encoding table based on the multi-pool related dimensions. The multi-pool related dimensions include the pool relationship, data distribution and cross-pool operation in the distributed storage system.

[0035] The resource pool testing module is used to create a corresponding target virtual machine based on a weighted test case set, and to start multi-threaded testing of the target resource pool using the weighted test case set and the target virtual machine to obtain the corresponding system stability test results.

[0036] Thirdly, this application discloses an electronic device, including:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute computer programs to implement the aforementioned stability testing methods for distributed storage systems.

[0039] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned stability testing method for a distributed storage system.

[0040] As can be seen, in this invention, an external test request for the distributed storage system is obtained, and the current multi-pool environment is initialized based on the external test request to obtain the target multi-pool environment. In the target multi-pool environment, a preset multi-pool encoding table is read to generate a weighted test case set based on the preset multi-pool encoding table. The preset multi-pool encoding table is a new encoding structure obtained by modifying the historical test encoding table based on multi-pool related dimensions. The multi-pool related dimensions include pool relationships, data distribution, and cross-pool operations in the distributed storage system. A corresponding target virtual machine is created based on the weighted test case set, and multi-threading is started using the weighted test case set and the target virtual machine to perform cross-pool stability testing on the target resource pool to obtain the corresponding system stability test results.

[0041] As can be seen from the above technical solution, this application initializes the multi-pool environment according to the test request to obtain a target multi-pool environment suitable for resource pool testing. In the target multi-pool environment, a new multi-pool encoding table modified based on multi-pool related dimensions is used to generate a corresponding weighted test case set. Based on the generated weighted test case set, multiple threads are started to perform cross-pool stability testing on the target resource pool, obtaining the final test results. By using an encoding table containing multi-pool related parameters for test case design, the generated test cases can cover multi-pool scenarios, enabling testing of distributed systems in multi-pool scenarios. Attached Figure Description

[0042] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a stability testing method for a distributed storage system disclosed in this invention;

[0044] Figure 2 This is a flowchart of a specific stability testing method for a distributed storage system disclosed in this invention;

[0045] Figure 3 This is a flowchart of a specific resource pool fault testing method for a distributed storage system disclosed in this invention;

[0046] Figure 4 This is a flowchart of a specific stability testing method for a distributed storage system disclosed in this invention;

[0047] Figure 5 This is a schematic diagram of the structure of a stability testing device for a distributed storage system disclosed in this invention.

[0048] Figure 6 This is a structural diagram of an electronic device disclosed in this invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0050] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0051] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Currently, the DR-Test single-pool testing framework for resource pool stability testing has limitations, including the inability to simulate cross-pool failure scenarios, the lack of resource allocation and state management mechanisms in multi-pool environments, the lack of support for multi-pool-specific data migration and load balancing tests, and the absence of multi-pool-related parameters in existing coding tables. Therefore, this invention will specifically introduce a stability testing method for distributed storage systems, capable of performing system stability testing on multi-pool architectures of distributed storage systems.

[0053] See Figure 1 As shown in the figure, this application discloses a stability testing method for a distributed storage system, applied to a multi-pool testing engine, including:

[0054] Step S11: Obtain external test requests for the distributed storage system, and initialize the current multi-pool environment based on the external test requests to obtain the target multi-pool environment.

[0055] In this embodiment, obtaining an external test request for the distributed storage system and initializing the current multi-pool environment based on the external test request to obtain the target multi-pool environment includes: obtaining an external test request for the distributed storage system; loading the corresponding pool topology configuration file based on the external test request and establishing a corresponding Secure Shell protocol mutual trust channel to initialize the current multi-pool environment based on the pool topology configuration file and the Secure Shell protocol mutual trust channel to obtain the target multi-pool environment. First, the multi-pool test engine, as the central system, receives the external test request for the distributed storage system and, according to the test requirements in the external test request, loads the pool topology configuration file and establishes an SSH (Secure Shell, a network protocol for secure remote login and file transfer) mutual trust channel to initialize the current multi-pool environment to obtain the target multi-pool environment.

[0056] Step S12: In the target multi-pool environment, read the preset multi-pool encoding table to generate a weighted test case set based on the preset multi-pool encoding table; the preset multi-pool encoding table is a new encoding structure obtained by modifying the historical test encoding table based on the multi-pool related dimensions. The multi-pool related dimensions include the pool relationship, data distribution and cross-pool operation in the distributed storage system.

[0057] In this embodiment, reading a preset multi-pool encoding table to generate a weighted test case set includes: using a preset test case generator to read the preset multi-pool encoding table, and generating a weighted test case set based on external test requests and the pool relationships, data distribution, and cross-pool operations in the preset multi-pool encoding table. Specifically, after completing environment initialization, in the obtained target test environment, the preset multi-pool encoding table is read through the preset test case generator, and a combination algorithm is used to generate the corresponding weighted test case set. During test case generation, conflicting scenarios such as simultaneously testing mirror pool splitting and network isolation need to be filtered out. It should be noted that the preset multi-pool encoding table used in this embodiment is based on the original encoding table, with the addition of multi-pool related dimensions to form a new encoding structure. This includes adding PoolRelation: 0-Independent, 1-Primary / Backup, 2-Hierarchical, 3-Mirror, 4-Random; DataDistribution: 0-Centralized Single Pool, 1-Uniform Distribution, 2-Hotspot Distribution, 3-Random; CrossPoolOperation: 0-None, 1-Migration, 2-Replication, 3-Snapshot, 4-Random. It is important to note that when generating the weighted test case set, the test cases need to be tested in conjunction with the current target multi-pool environment to determine if the generated weighted test case set matches the current target multi-pool environment. If they do not match, an exception is indicated, and the testing process is terminated. In this way, a hierarchical combination strategy of "pool relationship → data distribution → fault type" is adopted to control the number of test cases while ensuring coverage.

[0058] Step S13: Create the corresponding target virtual machine based on the weighted test case set, and use the weighted test case set and the target virtual machine to start multi-threaded cross-pool stability testing of the target resource pool to obtain the corresponding system stability test results.

[0059] In this embodiment, a corresponding target virtual machine is created based on a weighted test case set, and multi-threaded cross-pool stability testing of the target resource pool is performed using the weighted test case set and the target virtual machine. This includes: allocating data resources of the distributed storage system based on the test case requirements corresponding to the weighted test case set and external test requests, and constructing a corresponding resource allocation table based on the allocated data resources; creating a corresponding target virtual machine based on the resource allocation table and determining the target resource pool; and performing cross-pool stability testing of the target resource pool using multi-threaded cross-pool stability testing based on the resource allocation table and the target virtual machine. Specifically, when allocating resources, the idle rate of different resource pools is considered first, and resource pools with an idle rate >30% are selected first. In addition, when creating the target multi-pool environment, it is necessary to ensure that resource pools with a primary and backup relationship do not exist simultaneously in the target multi-pool environment. This can prevent the failure of one resource pool from causing the other resource pool to fail simultaneously due to the primary and backup relationship, leading to the collapse of the test environment. The further generated resource allocation tables are shown in Table 1 (PoolStatus table) and Table 2 (MultiPoolCaseStatus table) below:

[0060] Table 1

[0061]

[0062] Table 2

[0063]

[0064] In this embodiment, a multi-threaded approach is used to perform cross-pool stability testing on the target resource pool using a weighted test case set and the target virtual machine. This includes: starting a preset fault thread on the target virtual machine and using fault cases from the weighted test case set to perform cross-pool stability testing on the target resource pool; simultaneously, starting a preset monitoring thread on the target virtual machine to monitor the status of the target resource pool during the cross-pool stability test, and starting a preset main thread to record the status of the target resource pool to generate a resource pool status table. That is, a preset three-thread collaboration is initiated, where the main thread synchronizes a global snapshot every 5 seconds, the monitoring thread uses an exponential backoff strategy to adjust the check frequency, and the fault thread supports a "pause-continue" intervention mode.

[0065] Specifically, cross-pool stability testing is performed on the target resource pool using fault cases from a weighted test case set. Simultaneously, a pre-defined monitoring thread is started on the target virtual machine to monitor the status of the target resource pool during the cross-pool stability test. This includes: determining the target resource pool based on the fault types corresponding to the fault cases in the weighted test case set; fault cases include isolated node network cases for a single pool with a single node, batch service stop cases for multiple nodes in a single pool, simulated network partitioning cases for multi-pool collaboration, and cases for severing inter-pool links in cross-pool networks; inputting the fault cases into the target resource pool to initiate cross-pool stability testing; and recording the status of the target resource pool. Initially, a pre-defined monitoring thread is started based on the target virtual machine to monitor the target resource pool and obtain its current state. Based on the data consistency between the initial and current states, fault information of the target resource pool is determined. If the target resource pool is in a faulty state, the corresponding fault information is marked, and fault recovery is performed on the target resource pool based on an adaptive recovery strategy. If the target resource pool completes fault recovery, the corresponding resource lock is released, and a corresponding fault recovery report is generated. If the target resource pool has not completed fault recovery, the adaptive recovery strategy is adjusted based on the current fault recovery progress until the target resource pool completes fault recovery. That is, as follows... Figure 2 As shown, after determining the target resource pool, the types of fault tests required for that target resource pool are determined. Specific fault types include: fault testing of isolated node networks for a single node in a single pool; fault testing of batch service shutdown for multiple nodes in a single pool; fault testing of simulated network partitioning in multi-pool collaboration; and fault testing of severing inter-pool links in cross-pool networks. Then, the test is initiated. During fault testing, as follows... Figure 3 As shown, the faults in the test cases need to be injected into the target resource pool. The specific injection methods are shown in Table 3 below:

[0066] Table 3

[0067]

[0068] Firstly, for single-node failures, this can be addressed by injecting iptables -A DROP into the target host's firewall.<node_ip> The rules forcibly block node network communication and set up a 300-second automatic recovery circuit breaker mechanism to verify the cluster's ability to isolate basic faults; for multi-node failures, the parallel control command `systemctl stop ceph-osd@` is used. <id>Batch stop the specified OSD (Object Storage Daemon, a component responsible for data storage, replication, recovery and load balancing) service process, with a 600-second timeout window to observe data redistribution and self-healing efficiency; for network partitioning, the vlanisolation strategy is executed to cut off inter-node communication, and the maximum tolerance time of 900 seconds is verified to verify the split-brain protection mechanism and cross-zone data consistency guarantee capability. Further, all injection operations are performed through automated tools and strictly follow the preset timeout fuse strategy to ensure that the test process is controllable and does not affect the stability of the production environment.

[0069] Then, the initial state corresponding to the target resource pool is recorded, and then a monitoring thread is started to monitor the target resource pool. In the monitoring process, whether the resource pool is abnormal is judged by detecting data consistency. Here, the cross-pool consistency algorithm can be used to judge the data consistency, and the cross-pool consistency algorithm is as follows:

[0070] .

[0071] If abnormal, mark the fault propagation path, then upgrade the check level, and judge whether to recover completely. If the recovery is successful, release the corresponding resource lock, and after generating the corresponding recovery report, end the test. Further, as shown in Figure 4 , the generated recovery report can be used to optimize the multi-pool use case generator, so that subsequent use case generation can generate use cases that are more in line with the current test requirements. If the recovery is not successful, corresponding adjustments are made. Specifically, the adaptive recovery strategy is shown in Table 4:

[0072] Table 4

[0073]

[0074] That is, the resource pool recovery progress is continuously monitored, and its check frequency and performed recovery actions are dynamically adjusted according to the current completion percentage. Specifically, when the recovery progress is in the initial stage (0-30%), the check is performed at a higher frequency (every 10 seconds), focusing on performing lightweight "only restart basic service" operations, striving to quickly recover the basic functions of the resource pool. As the recovery enters the middle stage (30-70%), the check interval is adaptively extended to once every 30 seconds, at which time more in-depth "trigger data rebalancing" actions will be started to ensure the distribution and access of data in the resource pool to be normal. Finally, when the recovery is close to completion (70-100%), the check frequency is further reduced to once every 60 seconds, at which time the most stringent "perform full check" operation will be performed to finally verify the data integrity and state of the entire resource pool, to ensure the stability and reliability of the system after recovery. The whole process automatically switches strategies according to the recovery progress, achieving adaptive optimization of check intensity and recovery depth.

[0075] In judging whether the recovery is successful, verification can be based on the following table 5:

[0076] Table 5

[0077]

[0078] Specifically, for the integrity check after the resource pool failure recovery, the following verification process will be performed: First, the data accessibility is verified by initiating a random read test of the FIO tool (Flexible I / O Tester, an open source and powerful I / O performance test tool), requiring that 99.9% of read and write request response times be strictly less than 10 milliseconds. Second, the inode checksum verification value of all metadata is recursively scanned, and consistency matching of all nodes must be achieved, which can confirm that no metadata disorder or loss has occurred during the failure recovery process. Finally, the synchronization state across resource pools will be verified by comparing the latest operation sequence number of the pglog (placement group log), requiring that the difference in unsynchronized operations between pools be no more than 3, proving that the distributed data has achieved final consistency. It should be noted that according to the actual situation and related fault tests performed by the resource pool, the corresponding verification items can be selected for verification, but only when all three verifications are passed does it mean that the resource pool in the target resource environment is fully restored to a usable state and has the ability to carry production business.

[0079] In addition, it should be noted that when determining the target resource pool, the target pool needs to be selected based on weight round robin, and the specific code is as follows:

[0080] .

[0081] In this embodiment, the target virtual machine is used to start a preset monitoring thread to monitor the target resource pool to obtain the current state, including: starting a preset monitoring thread based on the target virtual machine, and based on a preset detection frequency, sequentially obtaining node inventory state information, data copy number information and cross-pool consistency verification information of the target resource pool; determining the current state of the target resource pool based on the node inventory state information, the data copy number information and the cross-pool consistency verification information. When performing stability testing, the preset monitoring thread adopts a three-level checking monitoring mechanism: checking the node survival state every 5 seconds (high-frequency checking); verifying the data copy number every 30 seconds (medium-frequency checking); checking the cross-pool consistency every 2 minutes (low-frequency checking).

[0082] In addition, the method disclosed in the application can also be applied to application stability testing in a multi-cloud scenario. The application in the multi-cloud scenario is determined as the target resource pool, and the method of the application can also complete the application stability testing in the multi-cloud scenario. Other application scenarios include, but are not limited to, edge computing scenarios, hybrid storage systems, etc.

[0083] It can be seen that in this embodiment, an external test request for a distributed storage system is obtained, and the current multi-pool environment is initialized based on the external test request to obtain a target multi-pool environment; in the target multi-pool environment, a preset multi-pool coding table is read to generate a weighted test case set based on the preset multi-pool coding table; the preset multi-pool coding table is a new coding structure obtained by modifying a historical test coding table based on multi-pool related dimensions, including pool relationship, data distribution and cross-pool operation in the distributed storage system; a corresponding target virtual machine is created based on the weighted test case set, and a multi-thread is started using the weighted test case set and the target virtual machine to perform cross-pool stability testing on the target resource pool to obtain a corresponding system stability test result. That is, by initializing the multi-pool environment according to the test request, a target multi-pool environment for resource pool testing is obtained. In the target multi-pool environment, a new multi-pool coding table modified based on multi-pool related dimensions is called to generate a corresponding weighted test case set. Based on the generated weighted test case set, a multi-thread is started to perform cross-pool stability testing on the target resource pool to obtain the final test result. In this way, by using the multi-pool related parameter coding table for case design, the generated cases can cover the multi-pool scenario, and the testing of the distributed system in the multi-pool scenario can be realized.

[0084] Reference Figure 5 The application embodiment also discloses a stability testing device of a distributed storage system, applied to a multi-pool testing engine, including:

[0085] The environment initialization module 11 is used to obtain external test requests for the distributed storage system and initialize the current multi-pool environment based on the external test requests to obtain the target multi-pool environment.

[0086] The test case set generation module 12 is used to read the preset multi-pool encoding table in the target multi-pool environment, and generate a weighted test case set based on the preset multi-pool encoding table. The preset multi-pool encoding table is a new encoding structure obtained by modifying the historical test encoding table based on the multi-pool related dimensions. The multi-pool related dimensions include the pool relationship, data distribution and cross-pool operation in the distributed storage system.

[0087] Resource pool testing module 13 is used to create a corresponding target virtual machine based on a weighted test case set, and use the weighted test case set and the target virtual machine to start multi-threaded cross-pool stability testing of the target resource pool to obtain the corresponding system stability test results.

[0088] As can be seen, in this embodiment, by initializing the multi-pool environment according to the test request, a target multi-pool environment suitable for resource pool testing is obtained. In the target multi-pool environment, a new multi-pool encoding table, modified based on multi-pool related dimensions, is invoked.

[0089] A corresponding weighted test case set is generated. Based on the generated weighted test case set, multi-threaded testing is launched to perform cross-pool stability tests on the target resource pool, obtaining the final test results. In this way, by using a test case design that includes a multi-pool related parameter encoding table, the generated test cases can cover multi-pool scenarios, enabling the testing of distributed systems in multi-pool scenarios.

[0090] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the stability testing method of the distributed storage system disclosed in any of the foregoing embodiments. Furthermore, the electronic device in this embodiment may specifically be an electronic computer.

[0091] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0092] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0093] The operating system 221 is used to manage and control the various hardware devices on the electronic device and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the stability testing method for the distributed storage system executed by the electronic device as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0094] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned stability testing method for a distributed storage system. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0095] Furthermore, this application also discloses a computer program product, including a computer program / instructions; wherein, when the computer program / instructions are executed by a processor, they implement the aforementioned disclosed alarm aggregation method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0097] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0099] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / id>

Claims

1. A stability testing method for a distributed storage system, characterized in that, Applications to multi-pool testing engines include: Obtain an external test request for the distributed storage system, and initialize the current multi-pool environment based on the external test request to obtain the target multi-pool environment; In the target multi-pool environment, a preset multi-pool encoding table is read to generate a weighted test case set based on the preset multi-pool encoding table; the preset multi-pool encoding table is a new encoding structure obtained by modifying the historical test encoding table based on multi-pool related dimensions, and the multi-pool related dimensions include the pool relationship, data distribution and cross-pool operation in the distributed storage system; Based on the weighted test case set, a corresponding target virtual machine is created, and the weighted test case set and the target virtual machine are used to start multi-threaded cross-pool stability testing of the target resource pool to obtain the corresponding system stability test results.

2. The stability testing method for a distributed storage system according to claim 1, characterized in that, The step of obtaining an external test request for the distributed storage system and initializing the current multi-pool environment based on the external test request to obtain the target multi-pool environment includes: Obtain external test requests for the distributed storage system; Based on the external test request, the corresponding pool topology configuration file is loaded, and a corresponding Secure Shell protocol mutual trust channel is established to initialize the current multi-pool environment based on the pool topology configuration file and the Secure Shell protocol mutual trust channel to obtain the target multi-pool environment.

3. The stability testing method for a distributed storage system according to claim 1, characterized in that, The step of reading a preset multi-pool encoding table to generate a weighted test case set based on the preset multi-pool encoding table includes: The preset test case generator reads the preset multi-pool encoding table and generates a weighted test case set based on the external test request and the pool relationships, data distribution and cross-pool operations in the preset multi-pool encoding table.

4. The stability testing method for a distributed storage system according to claim 1, characterized in that, The step of creating a corresponding target virtual machine based on the weighted test case set, and using the weighted test case set and the target virtual machine to start multi-threaded cross-pool stability testing of the target resource pool includes: Based on the weighted test case set and the test case requirements corresponding to the external test request, the data resources of the distributed storage system are allocated, and a corresponding resource allocation table is constructed based on the allocated data resources; Based on the resource allocation table, create the corresponding target virtual machine and determine the target resource pool; Based on the resource allocation table and the target virtual machine, a multi-threaded approach will be used to perform cross-pool stability testing on the target resource pool.

5. The stability testing method for a distributed storage system according to claim 1, characterized in that, The step of using the weighted test case set and the target virtual machine to start multiple threads to perform cross-pool stability testing on the target resource pool includes: A preset fault thread is started using the target virtual machine, and cross-pool stability testing of the target resource pool is performed using fault cases in the weighted test case set. Simultaneously, a preset monitoring thread is started using the target virtual machine to monitor the status of the target resource pool during the cross-pool stability test, and a preset main thread is started to record the status of the target resource pool to generate a resource pool status table.

6. The stability testing method for a distributed storage system according to claim 5, characterized in that, The target resource pool is subjected to cross-pool stability testing using faulty test cases from the weighted test case set. Simultaneously, a preset monitoring thread is started using the target virtual machine to monitor the status of the target resource pool during the cross-pool stability test, including: The target resource pool is determined based on the fault types corresponding to the fault cases in the weighted test case set; the fault cases include isolated node network cases of single pool single node, batch service stop cases of single pool multi-node, simulated network partitioning cases of multi-pool collaboration, and cross-pool network disconnection cases. Input the fault test cases into the target resource pool to initiate cross-pool stability testing; Record the initial state of the target resource pool, and simultaneously start a preset monitoring thread based on the target virtual machine to monitor the target resource pool in order to obtain the current state; The fault information of the target resource pool is determined based on the data consistency between the initial state and the current state; If the target resource pool is in a fault state, the fault information corresponding to the fault state is marked, and the target resource pool is restored based on an adaptive recovery strategy. If the target resource pool completes fault recovery, the corresponding resource lock is released and a corresponding fault recovery report is generated; If the target resource pool has not completed fault recovery, the adaptive recovery strategy is adjusted based on the current fault recovery progress until the target resource pool completes fault recovery.

7. The stability testing method for a distributed storage system according to claim 6, characterized in that, The step of starting a preset monitoring thread based on the target virtual machine to monitor the target resource pool and obtain its current status includes: A preset monitoring thread is started using the target virtual machine, and based on a preset detection frequency, the node inventory status information, data replica count information, and cross-pool consistency verification information of the target resource pool are obtained sequentially. The current state of the target resource pool is determined based on the node inventory status information, the data replica count information, and the cross-pool consistency verification information.

8. A stability testing device for a distributed storage system, characterized in that, Applications to multi-pool testing engines include: The environment initialization module is used to obtain external test requests for the distributed storage system and initialize the current multi-pool environment based on the external test requests to obtain the target multi-pool environment. The test case set generation module is used to read a preset multi-pool encoding table in the target multi-pool environment, and generate a weighted test case set based on the preset multi-pool encoding table; the preset multi-pool encoding table is a new encoding structure obtained by modifying the historical test encoding table based on the multi-pool related dimensions, and the multi-pool related dimensions include the pool relationship, data distribution and cross-pool operation in the distributed storage system. The resource pool testing module is used to create a corresponding target virtual machine based on the weighted test case set, and to use the weighted test case set and the target virtual machine to start multi-threaded cross-pool stability testing of the target resource pool to obtain the corresponding system stability test results.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the stability testing method for the distributed storage system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the stability testing method for the distributed storage system as described in any one of claims 1 to 7.

Images