Storage system testing method and device, equipment and storage medium

Abstract

The invention discloses a storage system testing method, device and equipment and a storage medium, relates to the technical field of chaos testing, is applied to a central control node of a distributed storage system, and comprises the following steps: collecting state information of the distributed storage system in real time after current fault data is injected; processing the state information by using a first model to obtain a current fault influence range, and processing the dependency graph by using a second model to obtain a current fault propagation path; the dependency graph is a graph constructed based on state information; and determining new current fault data according to a relation between a historical test result and the current fault influence range and a relation between the historical test result and the current fault propagation path, and then skipping to the step of collecting state information of the distributed storage system after the current fault data is injected in real time until the test is completed. Therefore, the overall fault injection strategy can be optimized according to the state information after the fault data is injected, and the test efficiency and the coverage rate of the distributed storage system are improved in combination with step jump closed-loop optimization.

Description

Technical Field

[0001] The present invention relates to the field of chaos testing technology, and in particular to a storage system testing method, device, equipment and storage medium. Background Art

[0002] Distributed storage systems are responsible for storing and processing massive amounts of data. As the scale and complexity of systems increase, their reliability and stability become increasingly important. Currently, common chaos fault testing uses manual or fault script-based testing methods, which have the following problems: 1. Blindness: Fault injection is random, making it impossible to accurately test system weaknesses; 2. Static scenarios: Preset failure modes make it difficult to cover complex interactive scenarios. Therefore, current chaos testing methods are difficult to adapt to the dynamic and complex nature of distributed storage systems.

[0003] It can be seen that how to improve the chaos testing effect of distributed storage systems is a problem that those skilled in the art need to solve. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a storage system testing method, apparatus, device and storage medium, which can solve the problems of blindness and low coverage of chaos testing in distributed storage systems.

[0005] To solve the above technical problems, in a first aspect, an embodiment of the present invention provides a storage system testing method, which is applied to a central control node of a distributed storage system, comprising:

[0006] Real-time collection of status information of the distributed storage system after the current fault data is injected;

[0007] The state information is processed using a preset first model to obtain the current fault impact range corresponding to the current fault data, and the dependency graph is processed using a preset second model to obtain the current fault propagation path corresponding to the current fault data. The first model is constructed based on historical fault data, and the second model is constructed based on the fault propagation path corresponding to the historical fault data. The dependency graph is a graph constructed based on the state information that represents the dependency relationships between different nodes.

[0008] Based on the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path, new current fault data is determined from the preset fault data set, and then the process jumps to the step of real-time collection of status information of the distributed storage system after the current fault data is injected, until all fault data in the preset fault data set have been tested.

[0009] In some embodiments, real-time collection of status information of the distributed storage system after the current fault data is injected includes:

[0010] Starting the initial agent program pre-installed on each node of the distributed storage system to perform the registration operation of each initial agent program on the central control node;

[0011] Each initial agent program that has completed the registration operation is determined as a corresponding target agent program, and state information of the distributed storage system after the current fault data is injected is collected in real time through each target agent program.

[0012] In some embodiments, the state information of the distributed storage system after the current fault data is injected is collected in real time by each target agent program, including:

[0013] The original status information of the distributed storage system after the current fault data is injected is collected in real time through each target agent program;

[0014] Obtain the packaged state information transmitted by Kafka in real time; the packaged state information is the information obtained by each target agent program by classifying and packaging the collected original state information based on the system level of the distributed storage system;

[0015] Based on the packaged state information, a dependency graph corresponding to the original state information is constructed using a graph database; the dependency graph represents the real-time state information between nodes in the distributed storage system;

[0016] The betweenness centrality of each node in the distributed storage system is calculated by a preset algorithm to obtain the corresponding calculation results;

[0017] Based on the calculation results, the nodes that meet the preset betweenness centrality conditions are marked to obtain the critical path identification in the dependency graph; the critical path identification indicates whether the fault test needs to record whether the corresponding path is covered;

[0018] The dependency graph including the critical path identifier is determined as target state information corresponding to the distributed storage system after the current fault data is injected.

[0019] In some embodiments, recording whether the fault test covers the corresponding path includes:

[0020] Through code instrumentation technology, a probe is injected into the path corresponding to the critical path identifier, so that the probe can be used to record the fault test to see whether the corresponding path is covered.

[0021] In some embodiments, determining new current fault data from a preset fault data set includes:

[0022] According to historical test results, a target data type is determined from several data types corresponding to fault data in a preset fault data set; the fault recovery time corresponding to the fault data of the target data type in the historical test results meets a preset time threshold;

[0023] The fault data corresponding to the target data type is determined as the new current fault data.

[0024] In some embodiments, based on the relationship between historical test results of the distributed storage system, the current fault impact range, and the current fault propagation path, new current fault data is determined from a preset fault data set, including:

[0025] Determine a number of test results corresponding to the current fault data from historical test results of the distributed storage system;

[0026] Comparing the current fault impact range and the current fault propagation path with a plurality of test results to obtain a corresponding first comparison result;

[0027] Determine whether the current fault data meets the preset test expectations based on the first comparison result;

[0028] If the current fault data does not meet the preset test expectations, the parameters of the current fault data are adjusted, and the adjusted fault data is determined as the new current fault data;

[0029] If the current fault data meets the preset test expectation, new current fault data is determined from the preset fault data set.

[0030] In some embodiments, if the current fault data is fault data obtained by adjusting parameters of previous fault data, determining new current fault data from a preset fault data set includes:

[0031] Comparing the current fault impact range and the current fault propagation path with the fault impact range and the fault propagation path corresponding to the previous fault data to obtain a second comparison result;

[0032] If the second comparison result does not meet the preset parameter adjustment expectation, the parameters of the current fault data are adjusted, and the adjusted fault data is determined as the new current fault data;

[0033] If the second comparison result indicates that the preset parameter adjustment expectation is met, new current fault data is determined from the preset fault data set.

[0034] In a second aspect, an embodiment of the present invention provides a storage system testing device, which is applied to a central control node of a distributed storage system, comprising:

[0035] The status information collection module is used to collect the status information of the distributed storage system in real time after the current fault data is injected;

[0036] An information processing module is configured to process the state information using a preset first model to obtain the current fault impact range corresponding to the current fault data, and to process the dependency graph using a preset second model to obtain the current fault propagation path corresponding to the current fault data. The first model is constructed based on historical fault data, and the second model is constructed based on the fault propagation path corresponding to the historical fault data. The dependency graph is a graph constructed based on the state information that represents the dependency relationships between different nodes.

[0037] The jump module is used to determine new current fault data from the preset fault data set based on the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path, and then jump to the step of real-time collection of status information of the distributed storage system after the current fault data is injected, until all fault data in the preset fault data set have been tested.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0039] Memory for storing computer programs;

[0040] The processor is configured to execute a computer program to implement the steps of the above-mentioned storage system testing method.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the storage system testing method described above are implemented.

[0042] It can be seen from the above technical solution that in this application, the central control node of the distributed storage system can collect the status information of the distributed storage system after the current fault data is injected in real time; then use the pre-set first model to process the status information to obtain the current fault impact range corresponding to the current fault data, and use the pre-set second model to process the dependency graph to obtain the current fault propagation path corresponding to the current fault data; the first model is a model constructed based on historical fault data, the second model is a model constructed based on the fault propagation path corresponding to the historical fault data, and the dependency graph is a graph constructed based on the status information to characterize the dependency relationship between different nodes; then, according to the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path, new current fault data can be determined from the preset fault data set, and then jump to the step of real-time collection of the status information of the distributed storage system after the current fault data is injected, until all fault data in the preset fault data set have been tested. In this way, this application can determine the fault data that needs to be injected subsequently based on the state information of the distributed storage system after the fault data is injected, as well as the corresponding fault impact range and fault propagation path dynamics, to optimize the overall fault injection strategy; and combine step jumping for closed-loop optimization, reducing manual intervention in fault injection and fault results, and improving the testing efficiency and coverage of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 This is a flow chart of a storage system testing method disclosed in this application;

[0045] Figure 2 A flowchart of a specific storage system testing method disclosed in this application;

[0046] Figure 3 This is a schematic structural diagram of a storage system testing device disclosed in this application;

[0047] Figure 4 This application discloses a structural diagram of an electronic device. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] The terms "including" and "having," as used in the present description and accompanying drawings, and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0050] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] like Figure 1 As shown, this embodiment discloses a storage system testing method, which is applied to a central control node of a distributed storage system, including:

[0052] Step S11: collecting status information of the distributed storage system in real time after the current fault data is injected.

[0053] In this embodiment, it can be understood that the distributed storage system may include a storage layer, a network layer, a control layer, a business layer, etc.; in the process of performing chaos testing on the distributed storage system, the status information of the storage system can be collected in real time after the current fault data is injected; the status information here may include information such as node performance, disk health and data distribution of the storage layer, bandwidth and latency, connection status and other information of the network layer, metadata service load, distributed protocol status and other information of the control layer, and client access and other information of the business layer.

[0054] In a specific embodiment, real-time collection of status information of a distributed storage system after the current fault data is injected can include: starting an initial agent program pre-set on each node of the distributed storage system to perform a registration operation of each initial agent program at a central control node; determining each initial agent program that has completed the registration operation as a corresponding target agent program, and collecting the status information of the distributed storage system after the current fault data is injected in real time through each target agent program. Specifically, the status information of the storage system after the current fault data is injected can be collected through a lightweight agent program (Agent) pre-deployed in the distributed storage system; the agent program automatically registers with the central control node at startup, synchronizes configuration parameters, and the registered agent program is recorded as the target agent program; during the chaos test of the distributed storage system, the target agent program can be used to collect multi-dimensional indicators of the storage system after the current fault data is injected in real time, including information on the storage layer, network layer, control layer, and business layer, and the collected multi-modal information is fused to obtain the status information of the corresponding distributed storage node.

[0055] In another specific embodiment, each target agent collects status information of the distributed storage system after the current fault data is injected in real time, which may include: collecting the original status information of the distributed storage system after the current fault data is injected in real time through each target agent; obtaining the packaged status information transmitted by Kafka in real time; the packaged status information is the information obtained by each target agent classifying and packaging the collected original status information based on the system hierarchy of the distributed storage system; based on each packaged status information, using a graph database to construct a dependency graph corresponding to the original status information; the dependency graph represents the real-time status information between each node in the distributed storage system; the betweenness centrality of each node in the distributed storage system is calculated by a preset algorithm to obtain corresponding calculation results; according to the calculation results, the nodes that meet the preset betweenness centrality conditions are marked to obtain a critical path identifier in the dependency graph; the critical path identifier indicates whether the fault test needs to be recorded to cover the corresponding path; the dependency graph containing the critical path identifier is determined as the target status information corresponding to the distributed storage system after the current fault data is injected. Specifically, during the process of collecting status information of a distributed storage system through an agent program, the agent program collects raw status information from various locations in the storage system and classifies it according to the different layers (storage layer, network layer, etc.) corresponding to the raw status information. The classification results are then packaged into information in a preset format (such as Protobuf (a data exchange format independent of language and interactive platform)), thus obtaining packaged status information. The central control node then uses a graph database to construct a dependency graph corresponding to each node in the distributed storage system based on the packaged status information transmitted by Kafka (a distributed publish-subscribe messaging system). It is understood that the dependency graph constructed through the above steps can represent the real-time status information and dependencies between nodes in the distributed storage system after the current fault data is injected. Furthermore, to facilitate the subsequent statistical analysis of specific test information for chaos testing, some key nodes can be marked according to the dependency graph to obtain critical path identifiers. It is understood that the critical path identifier in the dependency graph indicates whether the distributed storage system's fault test covers the path, so that when uncovered paths are subsequently discovered, new fault data corresponding to the scenario of the path can be generated for fault testing. In this way, this solution can collect the status information of the distributed storage system after the fault data is injected in real time through the agent program. After processing by the central control node, a dependency graph can be constructed and the critical path can be marked, and finally the real-time target status information corresponding to the distributed storage system after the current fault data is injected can be obtained.

[0056] In a specific embodiment, recording whether a fault test covers a corresponding path may include: injecting a probe into a path corresponding to a critical path identifier through code instrumentation technology, so as to use the probe to record whether the fault test covers the corresponding path. Specifically, in the process of determining whether a certain path is covered by a fault test, a probe may be injected into a path corresponding to a critical path identifier in a dependency graph through code instrumentation technology, so as to use the probe to track and record whether the fault test covers the path. Furthermore, it is understandable that if an uncovered path is found, corresponding fault data may be generated, and the priority of the fault data may be increased to optimize the coverage and test effect of the fault test of the distributed storage system.

[0057] Step S12: Use a pre-set first model to process the status information to obtain the current fault impact range corresponding to the current fault data, and use a pre-set second model to process the dependency graph to obtain the current fault propagation path corresponding to the current fault data; the first model is a model constructed based on historical fault data, the second model is a model constructed based on the fault propagation path corresponding to the historical fault data, and the dependency graph is a graph constructed based on the status information to characterize the dependency relationship between different nodes.

[0058] In this embodiment, the above steps can be used to obtain real-time status information of the distributed storage system after the current fault data is injected. Afterwards, the status information can be processed by pre-setting a first model and a second model to predict the fault impact range and fault propagation path of the distributed storage system in the future due to the injection of the current fault data. Specifically, the first model can be an LSTM (Long Short-Term Memory) network, which uses historical fault injection data (such as node failures and network partitions) to construct a supervised learning data set and train the first model. The second model can be a graph neural network, which is a model constructed using the fault propagation path corresponding to the historical fault data and is used for spatial propagation analysis.

[0059] In a specific embodiment, the first model and the second model can be pre-trained, that is, the model training is performed using historical data, and then directly used in the subsequent fault testing process; and when the first model and the second model are used to predict faults, the first model and the second model can be fine-tuned in combination with the current fault data and the corresponding predicted fault propagation path. Specifically, in the process of using fault data B to perform fault testing on the distributed storage system (using the first model B and the second model B to process the relevant status information), the first model B and the second model B are fine-tuned using the test results corresponding to the previous fault data A to obtain the first model C and the second model C; thereafter, when using the next fault data C to perform fault testing, the first model C and the second model C are used to process the corresponding status information, and the first model C and the second model C are fine-tuned using the test results corresponding to the fault data B; and so on. This can also gradually optimize the adaptability of the model without affecting the use of the model.

[0060] Step S13: Based on the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path, determine the new current fault data from the preset fault data set, and then jump to the step of real-time collection of status information of the distributed storage system after the current fault data is injected, until all the fault data in the preset fault data set have been tested.

[0061] In this embodiment, the above steps can be used to process the real-time status information of the distributed storage system after the current fault data is injected with the help of the first model and the second model, and the current fault impact range and the corresponding current fault propagation path corresponding to the current fault data can be predicted. Then, the fault test strategy is analyzed and optimized in combination with the historical fault test results, the current fault impact range and the current fault propagation path; for example, if a certain type of fault causes the recovery time of the storage system to exceed the specified threshold, the test priority of this type of fault data can be adjusted accordingly; and the priority of the fault data used in the subsequent test process can be adjusted and optimized in combination with the critical path identifier; considering these factors, new current fault data can be determined from the preset fault data set, and then the new current fault data can be used to perform fault testing on the distributed storage system until the pre-prepared fault data have completed the corresponding test; or the current test situation has met the corresponding test requirements, such as when facing a certain business, the fault recovery capability of the distributed storage system has met the business requirements.

[0062] In a specific embodiment, determining new current fault data from a preset fault data set may include: determining a target data type from several data types corresponding to fault data in the preset fault data set based on historical test results; the fault recovery time corresponding to the fault data of the target data type in the historical test results meets a preset time threshold; and determining the fault data corresponding to the target data type as the new current fault data. Specifically, in the process of determining the new current fault data, the impact of test data of different data types on the distributed storage system may be considered, such as when the fault recovery time corresponding to the fault data of the target data type in the historical test results meets a preset time threshold; at this time, the storage system may be tested using test data of this data type; first, the target data type is determined from several data types corresponding to the fault data in the preset fault data set, and then the fault data corresponding to the target data type is determined as the new current fault data. In other words, the priority of the fault data corresponding to the target data type in the preset fault data set may be adjusted so that the corresponding fault data is centrally utilized to perform fault testing on the distributed storage system.

[0063] In another specific embodiment, determining new current fault data from a preset fault data set based on the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path can include: determining several test results corresponding to the current fault data from the historical test results of the distributed storage system; using the current fault impact range and the current fault propagation path to compare with the several test results to obtain a corresponding first comparison result; judging whether the current fault data meets the preset test expectations based on the first comparison result; if the current fault data does not meet the preset test expectations, adjusting the parameters of the current fault data, and determining the adjusted fault data as the new current fault data; if the current fault data meets the preset test expectations, determining new current fault data from the preset fault data set. Specifically, in the process of determining the new current fault data, several test results in the historical test results that are associated with the current fault data can be considered; then, the several test results are compared with the current fault impact range and the current fault propagation path to obtain a corresponding first comparison result; it can be understood that the first comparison result represents the difference between the fault test result corresponding to the current fault data and the historical test result. If the current test effect of the distributed storage system is worse than the effect of the historical test, and the difference is within a certain threshold, the parameters of the current fault data can be adjusted, and the adjusted fault data can be determined as the new current fault data to perform a fault test on the distributed storage system again. Correspondingly, if the current test effect is not much different from the historical test effect, other fault data can be used to perform a fault test on the distributed storage system, that is, new current fault data can be determined from the preset fault data set, and the corresponding fault test operation can be performed. This can also achieve a comprehensive fault test on the distributed storage system by adjusting the parameters of the fault data.

[0064] In a specific embodiment, if the current fault data is fault data obtained by adjusting parameters of the previous fault data, determining the new current fault data from the preset fault data set may include: using the current fault impact range and the current fault propagation path to compare with the fault impact range and fault propagation path corresponding to the previous fault data to obtain a second comparison result; if the second comparison result indicates that it does not meet the preset parameter adjustment expectations, then adjusting the parameters of the current fault data and determining the adjusted fault data as the new current fault data; if the second comparison result indicates that it meets the preset parameter adjustment expectations, then determining the new current fault data from the preset fault data set. Specifically, if the current fault data for the fault test of the distributed storage system is fault data obtained by adjusting parameters of the previous fault data, it is necessary to consider the difference between the fault tests before and after the adjustment; using the fault impact range and fault propagation path corresponding to the current fault data to compare with the fault impact range and fault propagation path corresponding to the previous fault data, a second comparison result indicating the difference in the test effect of the fault data before and after the parameter adjustment can be obtained; then, by analyzing the second comparison result, it is possible to choose to adjust the parameters of the current fault data again or to determine new fault data from the preset fault data set. It can be understood that if the second comparison result indicates that the current test effect is better than the previous test effect and meets the corresponding effect change expectations, the new fault data can be used to perform fault testing on the distributed storage system; correspondingly, if the second comparison result indicates that the current test effect is not better than the previous test effect, the parameters of the current fault data can be re-adjusted to perform fault testing on the distributed storage system again using the adjusted fault data; in this way, the impact of the fault data on the test effect before and after the parameters can be comprehensively considered, and as many scenarios as possible can be covered to perform fault testing on the distributed storage system.

[0065] It can be seen that the present application can determine the fault data that needs to be injected subsequently based on the state information of the distributed storage system after the fault data is injected, as well as the corresponding fault impact range and fault propagation path dynamics, so as to optimize the overall fault injection strategy; and combine step jumping for closed-loop optimization, reduce manual intervention in fault injection and fault results, and improve the testing efficiency and coverage of the distributed storage system.

[0066] like Figure 2 As shown, this embodiment discloses a storage system testing method, which specifically includes:

[0067] In this embodiment, when the distributed storage system performs fault testing, it involves links such as agent programs, Kafka consumption, topological relationships, fault engines, adapters, and feedback loops; specifically, each node of the distributed storage system can deploy a lightweight agent program to collect multi-dimensional indicators of the distributed storage system in real time, including the storage layer, network layer, control layer, business layer, etc., and perform multimodal data fusion; the agent program can classify the collected raw data by layer (storage / network / control) and package it in Protobuf format, and transmit it to the central control node in real time through Kafka (here it generally refers to topological relationships, fault engines, adapters, feedback loops, etc.), and Kafka can perform operations such as abnormal data detection and data standardization, and pass the standardized data to the topological relationship link; in the topological relationship link, the configuration metadata can be parsed to generate an initial topological map, and a dependency relationship map can be constructed using a graph database. The resulting topology data (dependency graph, etc.) is then passed to the fault engine. This engine processes the storage system's topology data using a pre-trained LSTM model to predict the impact of future faults. This model uses reinforcement learning for global policy optimization. Furthermore, the fault engine passes the corresponding fault predictions to the adapter, which then requests dependency chain analysis. This dependency chain analysis primarily uses graph neural networks to identify fault propagation paths, such as node downtime -> metadata service overload -> client request timeout. LSTM is used for short-term predictions, while graph neural networks are used for spatial propagation analysis. Both models use reinforcement learning for global policy optimization and can combine historical data training with real-time data fine-tuning to improve model adaptability.

[0068] Furthermore, during the adaptive adjustment process of fault testing, fault prediction results and fault impact paths can be combined to dynamically adjust the corresponding fault testing strategy. This includes adjusting the priority of fault data. For example, if a certain fault causes the system recovery time to exceed a specified threshold, its priority can be adjusted. Furthermore, code instrumentation can be used to identify uncovered paths, generate fault data related to the new scenario, and insert it into a high-priority queue. Code instrumentation tracing primarily involves injecting probes into key services to record request paths. Subsequent agents can provide feedback on the adjustment results and tracked data. Furthermore, within the feedback loop, resilience profile generation and automated remediation recommendations are introduced. Resilience profile construction defines resilience dimensions, including metrics such as availability, recoverability, and data consistency of distributed storage systems. Visual scorecards are generated by comparing scores for each dimension before and after the fault test. Based on the fault type, changes to the test strategy configuration can be recommended. After the remediation is performed, chaos testing can be rerun to verify the improved resilience score. In specific embodiments, a graph can be constructed based on historical work orders to recommend verified modification solutions for similar scenarios.

[0069] It can be seen that this application can determine the subsequent fault data that needs to be injected based on the state information of the distributed storage system after the fault data is injected, as well as the corresponding fault impact range and fault propagation path dynamics, to optimize the overall fault injection strategy; and combine step jumps for closed-loop optimization; based on the feedback summary data, a resilience portrait that characterizes the fault response capability of the distributed storage system can be generated, and corresponding repair suggestions can be given in combination with historical work orders. This can improve the fault testing effect of the distributed system, reduce the failure rate and the risk of data loss, reduce manual intervention in fault injection and fault results, and improve the testing efficiency and coverage of the distributed storage system.

[0070] like Figure 3 As shown, this embodiment discloses a storage system testing device, which is applied to a central control node of a distributed storage system, including:

[0071] The status information collection module 11 is used to collect the status information of the distributed storage system in real time after the current fault data is injected;

[0072] The information processing module 12 is configured to process the state information using a preset first model to obtain the current fault impact range corresponding to the current fault data, and to process the dependency graph using a preset second model to obtain the current fault propagation path corresponding to the current fault data. The first model is constructed based on historical fault data, the second model is constructed based on the fault propagation path corresponding to the historical fault data, and the dependency graph is a graph constructed based on the state information to represent the dependency relationships between different nodes.

[0073] The jump module 13 is used to determine new current fault data from the preset fault data set based on the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path, and then jump to the step of real-time collection of status information of the distributed storage system after the current fault data is injected, until all fault data in the preset fault data set have been tested.

[0074] It can be seen that the present application can determine the fault data that needs to be injected subsequently based on the state information of the distributed storage system after the fault data is injected, as well as the corresponding fault impact range and fault propagation path dynamics, so as to optimize the overall fault injection strategy; and combine step jumping for closed-loop optimization, reduce manual intervention in fault injection and fault results, and improve the testing efficiency and coverage of the distributed storage system.

[0075] In a specific embodiment, the status information collection module 11 includes:

[0076] An agent program registration unit is used to start the initial agent program pre-set on each node of the distributed storage system to perform the registration operation of each initial agent program at the central control node;

[0077] The status information collection submodule is used to determine each initial agent program that has completed the registration operation as a corresponding target agent program, and collect the status information of the distributed storage system after the current fault data is injected in real time through each target agent program.

[0078] In another specific embodiment, the status information collection submodule may include:

[0079] The original information collection unit is used to collect the original status information of the distributed storage system after the current fault data is injected in real time through each target agent program;

[0080] The packaged information acquisition unit is used to obtain the packaged status information transmitted by Kafka in real time; the packaged status information is the information obtained by each target agent program by classifying and packaging the collected original status information based on the system level of the distributed storage system;

[0081] A relationship graph construction unit is used to construct a dependency graph corresponding to the original state information based on the packaged state information using a graph database; the dependency graph represents the real-time state information between the nodes in the distributed storage system;

[0082] A node calculation unit is used to calculate the betweenness centrality of each node in the distributed storage system using a preset algorithm to obtain a corresponding calculation result;

[0083] The node marking unit is used to mark the nodes that meet the preset betweenness centrality conditions according to the calculation results, so as to mark the critical path identification in the dependency graph; the critical path identification indicates whether the fault test needs to record whether the corresponding path is covered;

[0084] The information determination unit is used to determine the dependency graph containing the critical path identifier as the target state information corresponding to the distributed storage system after the current fault data is injected.

[0085] In a specific embodiment, the process of obtaining a critical path identifier through a node marking unit includes: injecting a probe into a path corresponding to the critical path identifier through code instrumentation technology, so as to use the probe to record faults and test whether the corresponding path is covered.

[0086] In a specific embodiment, the jump module 13 may include:

[0087] A data type determination unit is configured to determine a target data type from a plurality of data types corresponding to fault data in a preset fault data set according to historical test results; the fault recovery time corresponding to the fault data of the target data type in the historical test results satisfies a preset time threshold;

[0088] The fault data determining unit is configured to determine the fault data corresponding to the target data type as new current fault data.

[0089] In a specific embodiment, the jump module 13 may include:

[0090] a test result determination unit, configured to determine a number of test results corresponding to the current fault data from historical test results of the distributed storage system;

[0091] A first comparison unit is configured to compare a current fault impact range and a current fault propagation path with a plurality of test results to obtain a corresponding first comparison result;

[0092] A first judging unit, configured to judge whether the current fault data meets a preset test expectation according to the first comparison result;

[0093] A first parameter adjustment unit is configured to adjust parameters of the current fault data when the current fault data does not meet the preset test expectation, and determine the adjusted fault data as new current fault data;

[0094] The new fault data determination submodule is used to determine new current fault data from the preset fault data set when the current fault data meets the preset test expectation.

[0095] In another specific embodiment, the new fault data determination submodule may include:

[0096] A second comparison unit is configured to compare the current fault impact range and the current fault propagation path with the fault impact range and the fault propagation path corresponding to the previous fault data to obtain a second comparison result;

[0097] A second parameter adjustment unit is configured to adjust the parameters of the current fault data when the second comparison result does not meet the preset parameter adjustment expectation, and determine the adjusted fault data as the new current fault data;

[0098] The new fault data determining unit is configured to determine new current fault data from the preset fault data set when the second comparison result indicates that the current fault data meets the preset parameter adjustment expectation.

[0099] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4This is a block diagram of an electronic device according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of use of this application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the storage system testing method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device in this embodiment may specifically be an electronic computer.

[0100] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0101] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0102] The operating system 221 is used to manage and control the hardware devices on the electronic device, as well as the computer program 222, which can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the storage system testing method performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0103] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned storage system testing method. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0104] Furthermore, the present application also discloses a computer program product, including a computer program / instruction, which implements the aforementioned storage system testing method when executed by a processor.

[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0106] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0108] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0109] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A storage system testing method, characterized in that: Central control nodes used in distributed storage systems include: Real-time collection of status information of the distributed storage system after the current fault data is injected; The state information is processed using a preset first model to obtain a current fault impact range corresponding to the current fault data, and the dependency graph is processed using a preset second model to obtain a current fault propagation path corresponding to the current fault data; the first model is a model constructed based on historical fault data, the second model is a model constructed based on the fault propagation path corresponding to the historical fault data, and the dependency graph is a graph constructed based on the state information that represents the dependency relationships between different nodes; Based on the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path, new current fault data is determined from the preset fault data set, and then the process jumps to the step of real-time collection of status information of the distributed storage system after the current fault data is injected, until all fault data in the preset fault data set have been tested.

2. The storage system testing method according to claim 1, wherein: The real-time collection of status information of the distributed storage system after the current fault data is injected includes: Starting an initial agent program pre-set on each node of the distributed storage system to perform a registration operation of each initial agent program on the central control node; Each of the initial agent programs that has completed the registration operation is determined as a corresponding target agent program, and state information of the distributed storage system after the current fault data is injected is collected in real time through each of the target agent programs.

3. The storage system testing method according to claim 2, wherein: The real-time collection of status information of the distributed storage system after the current fault data is injected by each target agent program includes: Collecting original status information of the distributed storage system after the current fault data is injected in real time through each target agent program; Obtaining packaged state information transmitted in real time by Kafka; the packaged state information is information obtained by each target agent program by classifying and packaging the collected original state information based on the system level of the distributed storage system; Based on each of the packaged state information, a dependency graph corresponding to the original state information is constructed using a graph database; the dependency graph represents the real-time state information between each node in the distributed storage system; Calculating the betweenness centrality of each node in the distributed storage system using a preset algorithm to obtain corresponding calculation results; According to the calculation results, nodes that meet the preset betweenness centrality conditions are marked to obtain critical path identifiers in the dependency graph; the critical path identifiers indicate whether the corresponding paths need to be recorded for fault testing; The dependency graph including the critical path identifier is determined as target state information corresponding to the distributed storage system after the current fault data is injected.

4. The storage system testing method according to claim 3, wherein: The recording fault test to determine whether the corresponding path is covered includes: A probe is injected into the path corresponding to the critical path identifier by using code instrumentation technology, so as to use the probe to record faults and test whether the corresponding path is covered.

5. The storage system testing method according to any one of claims 1 to 4, characterized in that: The determining of new current fault data from the preset fault data set includes: According to the historical test results, a target data type is determined from several data types corresponding to the fault data in the preset fault data set; the fault recovery time corresponding to the fault data of the target data type in the historical test results meets a preset time threshold; The fault data corresponding to the target data type is determined as new current fault data.

6. The storage system testing method according to claim 1, wherein: The determining new current fault data from a preset fault data set according to a relationship between historical test results of the distributed storage system, the current fault impact range, and the current fault propagation path includes: Determining a plurality of test results corresponding to the current fault data from historical test results of the distributed storage system; Comparing the current fault impact range and the current fault propagation path with the plurality of test results to obtain a corresponding first comparison result; Determining whether the current fault data meets preset test expectations based on the first comparison result; If the current fault data does not meet the preset test expectation, adjusting parameters of the current fault data, and determining the adjusted fault data as new current fault data; If the current fault data meets the preset test expectation, new current fault data is determined from the preset fault data set.

7. The storage system testing method according to claim 6, wherein: If the current fault data is fault data obtained by adjusting parameters of the previous fault data, then determining new current fault data from the preset fault data set includes: Comparing the current fault impact range and the current fault propagation path with the fault impact range and the fault propagation path corresponding to the previous fault data to obtain a second comparison result; If the second comparison result does not meet the preset parameter adjustment expectation, adjusting the parameters of the current fault data, and determining the adjusted fault data as the new current fault data; If the second comparison result indicates that the preset parameter adjustment expectation is met, new current fault data is determined from the preset fault data set.

8. A storage system testing device, characterized in that: Central control nodes used in distributed storage systems include: A status information collection module, configured to collect status information of the distributed storage system in real time after the current fault data is injected; an information processing module, configured to process the state information using a preset first model to obtain a current fault impact range corresponding to the current fault data, and to process a dependency graph using a preset second model to obtain a current fault propagation path corresponding to the current fault data; the first model is a model constructed based on historical fault data, the second model is a model constructed based on the fault propagation path corresponding to the historical fault data, and the dependency graph is a graph constructed based on the state information that represents the dependency relationships between different nodes; A jump module is used to determine new current fault data from a preset fault data set based on the relationship between the historical test results of the distributed storage system and the current fault impact range and the current fault propagation path, and then jump to the step of real-time collection of status information of the distributed storage system after the current fault data is injected, until all fault data in the preset fault data set have been tested.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the storage system testing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the storage system testing method according to any one of claims 1 to 4.

Images