A method, device, equipment and readable storage medium for determining fault impact

Abstract

The present invention discloses a method, device, equipment and readable storage medium for determining the impact of a fault, which is applied to the field of computer system technology, including: using a virtual machine template in a hyper-converged system to batch deploy business virtual machines to obtain deployed business virtual machines; based on the test tool in the virtual machine template, performing a business stress test for each deployed business virtual machine to obtain a business virtual machine with business stress; injecting a target fault into the business virtual machine with business stress to perform a fault impact test on performance to obtain performance test data; analyzing the performance test data to determine the performance impact parameter corresponding to each target fault. Compared with the current inability to determine the impact of a fault on performance in a hyper-converged system, the present invention can detect the impact of a fault on the performance of a hyper-converged system by deploying a business virtual machine, starting business stress and fault injection, thereby improving the comprehensiveness of determining the impact of a fault on a hyper-converged system.

Description

Technical Field

[0001] The present invention relates to the technical field of computer systems, and in particular to a method, device, equipment and readable storage medium for determining a fault impact. Background Art

[0002] Since the hyper-converged system is an integrated hardware and software infrastructure, its reliability is affected by multiple factors such as servers, hard disks, network cards, networks, and software. It is important to focus on hardware failure testing, which is of great significance for improving system stability, ensuring data security, optimizing operation and maintenance efficiency, and ensuring business continuity. However, currently, we only focus on whether the business is restored. Or we pay little attention to the overall recovery time of the system. This leads to insufficient assessment of the impact of failure scenarios.

[0003] It can be seen that how to improve the comprehensiveness of determining the impact of failures on hyper-converged systems is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and readable storage medium for determining the impact of a fault, which solves the technical problem in the prior art that the impact of a fault on the performance of a hyper-converged system cannot be determined.

[0005] In order to solve the above technical problems, the present invention provides a method for determining the impact of a fault, comprising:

[0006] Deploy business virtual machines in batches using virtual machine templates in the hyper-converged system to obtain deployed business virtual machines;

[0007] Based on the test tool in the virtual machine template, a business stress test is performed on each deployed business virtual machine to obtain a business virtual machine with business stress;

[0008] Injecting the target fault into the business virtual machine with business pressure to test the impact of the fault on performance, and obtaining performance test data;

[0009] The performance test data is analyzed to determine the performance impact parameter corresponding to each target fault.

[0010] In some embodiments, a business stress test is performed for each deployed business virtual machine based on the test tool in the virtual machine template to obtain a business virtual machine with business stress, including:

[0011] Based on different types of testing tools in the virtual machine template, a business stress test is performed for each deployed business virtual machine to obtain the business virtual machine with business stress; wherein the type of testing tool is determined based on the type of business stress.

[0012] In some embodiments, after analyzing the performance test data to determine the performance impact parameter corresponding to each target fault, the method further includes:

[0013] Compare the performance impact parameter corresponding to each target fault with the set minimum performance impact parameter;

[0014] When the performance impact parameter is lower than the set minimum performance impact parameter, determining that the priority of the current target fault is a low priority;

[0015] When the performance impact parameter is not lower than the set minimum performance impact parameter, it is determined that the priority of the current target fault is a high priority.

[0016] In some embodiments, analyzing the performance test data to determine the performance impact parameter corresponding to each target fault includes:

[0017] Performing data space impact analysis, cache space analysis and data recovery analysis on the performance test data to determine a plurality of performance impact parameters corresponding to the target fault;

[0018] Comprehensive processing is performed based on the multiple performance impact parameters to determine the performance impact parameter corresponding to each target fault.

[0019] In some embodiments, analyzing the performance test data to determine the performance impact parameter corresponding to each target fault includes:

[0020] Determine the predicted performance impact parameters corresponding to each target fault using the target machine learning model;

[0021] Analyzing the performance test data to obtain test performance influencing parameters;

[0022] Comprehensively processing the predicted performance impact parameter and the tested performance impact parameter to determine the performance impact parameter corresponding to each target fault;

[0023] The training process of the target machine learning model includes:

[0024] Obtaining historical target failures and corresponding historical actual performance data when the historical target failures occurred;

[0025] Cleaning the historical actual performance data to obtain cleaned historical actual performance data;

[0026] Using the historical target failures and the corresponding historical actual performance data to train a machine learning model to obtain an initial machine learning model;

[0027] The performance of the initial machine learning model is determined using a validation set, and the parameters of the initial machine learning model are adjusted based on the validation results to obtain the target machine learning model.

[0028] In some embodiments, the target failure includes at least two of node power failure, node network disconnection, disk unplugging, node deletion, node expansion, and node online.

[0029] In some embodiments, injecting a target fault into the business virtual machine with business pressure to test the impact of the fault on performance to obtain performance test data includes:

[0030] The target fault is injected into the business virtual machine with business pressure to test the impact of the fault on performance, and performance test data corresponding to the hyper-converged storage and performance test data corresponding to the virtual machine are obtained.

[0031] The embodiment of the present invention further provides a fault impact determination device, which is applied to a hyper-converged system, including:

[0032] A deployment module is used to deploy business virtual machines in batches using virtual machine templates in the hyper-converged system to obtain deployed business virtual machines;

[0033] Starting a business pressure module, which is used to perform a business pressure test on each deployed business virtual machine based on the test tool in the virtual machine template to obtain a business virtual machine with business pressure;

[0034] A fault injection module is used to inject a target fault into the business virtual machine with business pressure to test the impact of the fault on performance and obtain performance test data;

[0035] The performance impact determination module is used to analyze the performance test data and determine the performance impact parameter corresponding to each target fault.

[0036] The embodiment of the present invention further provides a fault impact determination device, including:

[0037] Memory for storing computer programs;

[0038] A processor is used to execute the computer program to implement the steps of the above-mentioned fault impact determination method.

[0039] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned fault impact determination method are implemented.

[0040] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned fault impact determination method when executed by a processor.

[0041] To solve the above technical problems, an embodiment of the present invention provides a method for determining fault impact, which is applied to a hyper-converged system and may include: using a virtual machine template in the hyper-converged system to batch deploy business virtual machines to obtain deployed business virtual machines; performing a business stress test on each deployed business virtual machine based on a test tool in the virtual machine template to obtain a business virtual machine with business stress; injecting a target fault into the business virtual machine with business stress to test the impact of the fault on performance to obtain performance test data; analyzing the performance test data to determine the performance impact parameters corresponding to each target fault.

[0042] It can be seen from the above technical scheme that the beneficial effect of the present invention is that compared with the current situation where it is impossible to determine the impact of faults on the performance of a hyper-converged system, the present invention can detect the impact of faults on the performance of a hyper-converged system by deploying business virtual machines, starting business pressure and fault injection, and improves the comprehensiveness of determining the impact of faults on the hyper-converged system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of a method for determining a fault impact provided by an embodiment of the present invention;

[0045] Figure 2 An example flow chart of a method for determining a fault impact provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of a fault impact determination device provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of a fault impact determination device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0050] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0051] Next, a fault impact determination method provided by an embodiment of the present invention is described in detail. Figure 1 A flowchart of a method for determining a fault impact provided by an embodiment of the present invention is applied to a hyper-converged system. The method may include:

[0052] S101, deploying business virtual machines in batches using virtual machine templates in a hyper-converged system to obtain deployed business virtual machines.

[0053] The execution subject of this embodiment is a hyper-converged system. The hyper-converged system, based on the hardware reconstruction platform and software-defined system design, realizes the high integration and unified management of computing, storage, network, and security resources, and provides an efficient, stable, safe and easy-to-use integrated solution. It has the characteristics of decoupling, reconfiguration, automation, integrity, and elasticity. It can help customers quickly build cloud data centers, reduce complexity, and deliver flexibly and quickly. Thanks to the streamlined distributed architecture, integrated deployment method and elastic expansion capabilities, it has been recognized by more and more users. The maturity of hyper-convergence technology has been significantly improved, accelerating its entry into key business application scenarios. Providing low-latency, high-concurrency performance support and meeting the high requirements of core businesses at the high availability and continuity level will still be the focus of future attention. The virtual machine template in this embodiment is a pre-configured virtual machine instance, which includes an operating system, software, configuration, and settings. When creating a new virtual machine, you can use the virtual machine template to quickly deploy an environment consistent with the template. The following is a detailed explanation of the virtual machine template: A virtual machine template is a snapshot of a virtualization environment containing a specific configuration and software environment. It can be used as the basis for creating a new virtual machine to ensure that all virtual machines created based on the template have the same initial state. Using virtual machine templates can significantly improve deployment efficiency because there is no need to reconfigure the operating system and application for each new virtual machine. This reduces the possibility of human error and improves the reliability and stability of the system. The business virtual machine in this embodiment is a software entity that simulates and runs a virtual computer on a physical computer. The business virtual machine in this embodiment is written as a business virtual machine to distinguish it from the virtual machine template.

[0054] S102: Perform a business stress test on each deployed business virtual machine based on the test tool in the virtual machine template to obtain a business virtual machine with business stress.

[0055] The virtual machine template in this embodiment includes a test tool. This embodiment does not limit the specific number of tests. For example, the test tools in this embodiment may be multiple; or the test tool in this embodiment may be single. This embodiment does not limit the specific test tools. For example, the test tool in this embodiment may be a fio stress testing tool; or the test tool in this embodiment may be a fio stress testing tool and a sysbench stress testing tool. The fio stress testing tool is a tool for I / O testing and hardware stress / performance verification. The sysbench stress testing tool is an open source, modular, cross-platform multi-threaded performance testing tool. These two stress testing tools are used in performance testing to simulate real-world business loads to evaluate the performance of the system under high-stress conditions.

[0056] It should be further explained that, in order to improve the accuracy of business pressure, the above-mentioned test tool based on the virtual machine template performs a business pressure test for each deployed business virtual machine to obtain a business virtual machine with business pressure, which may include: based on different types of test tools in the virtual machine template, performing a business pressure test for each deployed business virtual machine to obtain a business virtual machine with business pressure; wherein, based on the type of business pressure, the type of the test tool is determined. In this embodiment, determining the type of the test tool based on the type of business pressure means that each test tool has a main simulated business type, so that business pressure can be injected based on multiple different types of test tools, which can improve the comprehensiveness of the business pressure test. For example, the fio stress test tool mainly simulates a typical IO (input and output) model. Sysbench is a model that mainly simulates database oltp (online transaction processing) business. Hyper-convergence core value delivery scenarios, such as the financial industry, medical industry, and transportation industry. Summarizing the business model, it mainly includes small block random read and write, large block sequential read and write, or mixed read and write. Therefore, the IO model is made into a fio configuration file to form a configuration file library. If there is no special requirement, the 64k mixed read and write model is used as the default stress model. The Sysbench test commands are also written to the configuration file of the virtual machine template.

[0057] S103, injecting a target fault into a business virtual machine with business pressure to test the impact of the fault on performance, and obtaining performance test data.

[0058] This embodiment does not limit the specific target fault. The target fault in this embodiment can be various faults that may occur in chaos engineering, and the fault can be obtained from the fault library. The target fault in this embodiment can be node power failure, node network disconnection, disk removal, node deletion, node expansion and node online, etc. The target fault can include at least two of node power failure, node network disconnection, disk removal, node deletion, node expansion and node online. This embodiment does not limit the specific performance test data. For example, the performance test data in this embodiment is the trough average value; or the performance test data in this embodiment can also be data space, cache space, fault recovery time, etc. It can be understood that in order to improve the comprehensiveness of performance test data acquisition, the target fault can be injected into a business virtual machine with business pressure to test the impact of the fault on performance, and obtain the performance test data corresponding to the hyper-converged storage, and the performance test data corresponding to the virtual machine. In this embodiment, the hyper-converged storage is a key component of the hyper-converged system. It is pooled and managed in the hyper-converged infrastructure through software-defined storage resources. Therefore, determining the performance test data of the hyper-converged storage can improve the comprehensiveness of determining the impact of the fault on performance in the hyper-converged system. The virtual machine in this embodiment is an entity that runs applications and services in the hyper-converged system. The performance of the virtual machine directly reflects the performance of the entire system. This embodiment does not limit the performance test data corresponding to the virtual machine. For example, the test data corresponding to the virtual machine in this embodiment can be the trough duration. For ease of understanding, please refer to Table 1, which is a performance test data corresponding to a virtual machine provided in an embodiment of the present invention. Table 1 provides quantitative data on the impact on the performance of the virtual machine in the hyper-converged system under a specific fault condition, that is, after the SSD is unplugged. Each column in the table represents a different performance indicator: Trough duration (min): refers to the number of minutes the performance trough lasts. In this example, the trough duration is 3 minutes. Average trough value before fault: refers to the average value of the performance trough before the fault occurs. Here, the average value of the trough before the fault is 5000. Average trough value after fault: refers to the average value of the performance trough after the fault occurs. After the SSD is unplugged, the trough average value drops to 1000. Performance degradation ratio: By comparing the average value of the performance trough before and after the fault, the percentage of performance degradation is calculated. In this case, the performance dropped by 80%, which means that the system performance is significantly reduced after the SSD is unplugged. The data in this table can be used to analyze the specific impact of unplugging the SSD on the performance of the hyper-converged system

[0059] Table 1 Performance test data corresponding to a virtual machine

[0060]

[0061] This embodiment does not limit the performance test data corresponding to the hyper-converged storage. For example, the performance test data corresponding to the hyper-converged storage in this embodiment may be data space, cache space, and data recovery. Please refer to Table 2, which is a performance test data corresponding to a hyper-converged storage provided in an embodiment of the present invention.

[0062] Table 2 Performance test data corresponding to a hyper-converged storage

[0063]

[0064] It should be further explained that, based on any of the above embodiments, the above-mentioned injection of the target fault into a business virtual machine with business pressure to test the impact of the fault on performance and obtain performance test data may include: injecting the target fault into a business virtual machine with business pressure to test the impact of the fault on performance, and monitoring the business virtual machine with business pressure, and determining stable performance test data based on the monitoring data; wherein, the stable performance test data is the impact of the fault injection on the performance that fluctuates within a set range. The reason why this embodiment obtains stable data is to collect data on system performance under fault conditions. The key to this process is to determine stable performance test data, that is, after the fault is injected, the fluctuation of system performance is within a controllable and predictable range. Such data can help understand the performance of the system under actual fault conditions and improve the accuracy of performance test data.

[0065] S104, analyzing the performance test data to determine the performance impact parameter corresponding to each target fault.

[0066] This embodiment does not limit the specific process of analyzing the performance test data to determine the performance impact parameters corresponding to each target fault. For example, the performance test data can be analyzed to obtain multiple types of performance test data, thereby obtaining the performance impact parameters corresponding to each performance test data, and averaging the multiple performance impact parameters to obtain the performance impact parameters corresponding to each target fault. Or in this embodiment, the performance test data can be analyzed to obtain multiple types of performance test data, thereby obtaining the performance impact parameters corresponding to each performance test data, and after multiplying the multiple performance impact parameters and the corresponding weight values, they are summed to obtain the performance impact parameters corresponding to each target fault. The performance impact parameter in this embodiment can be a performance impact score; or the performance impact parameter in this embodiment can be a performance degradation ratio.

[0067] It should be further explained that, based on any of the above embodiments, the above analysis of the performance test data to determine the performance impact parameters corresponding to each target fault may include: performing data space impact analysis, cache space analysis and data recovery analysis on the performance test data to determine multiple performance impact parameters corresponding to the target fault; performing comprehensive processing based on multiple performance impact parameters to determine the performance impact parameters corresponding to each target fault. The data space impact analysis in this embodiment refers to determining the performance of the storage system before and after fault injection, including the storage read and write speed, space utilization and performance under high load. The cache performance test result analysis in this embodiment determines the change in cache space performance by deeply analyzing the cache hit rate, response time, system resource usage and other indicators before and after fault injection. The data recovery analysis in this embodiment refers to evaluating the time and resource consumption required for data recovery after a system failure. The data recovery analysis can determine the impact of the failure on the system. This embodiment does not limit the specific comprehensive processing method. For example, the comprehensive processing method in this embodiment may be to directly calculate the average; or the comprehensive processing method in this embodiment may be to multiply each performance impact parameter by the corresponding weight based on the set data space impact analysis, cache space analysis, and data recovery analysis, and then perform addition processing to obtain the performance impact parameter corresponding to each target fault. This embodiment performs data space impact analysis, cache space analysis, and data recovery analysis on the performance test data before and after fault injection, thereby achieving multi-faceted analysis and improving the accuracy of determining the performance impact parameters.

[0068] It should be further explained that, based on any of the above embodiments, the above analysis of the performance test data to determine the performance impact parameter corresponding to each target fault may include:

[0069] S1041, using the target machine learning model to determine the predicted performance impact parameters corresponding to each target fault.

[0070] This embodiment can be implemented by training a model to identify the association between the target fault and the performance parameter. For example, the model is trained using historical data so that it can predict the change of performance indicators such as response time and throughput according to the fault type. The training process of the target machine learning model in this embodiment may include: obtaining historical target faults and the corresponding historical actual performance data when the historical target fault occurs; cleaning the historical actual performance data to obtain the cleaned historical actual performance data; training the machine learning model using the historical target faults and the corresponding historical actual performance data to obtain the initial machine learning model; determining the performance of the initial machine learning model using the validation set, and adjusting the parameters of the initial machine learning model based on the validation results to obtain the target machine learning model. This embodiment does not limit the specific machine learning model. For example, the machine learning model in this embodiment may be a linear regression model; or the machine learning model in this embodiment may also be a neural network model. This embodiment does not limit the specific historical actual performance data. For example, the historical actual performance data in this embodiment may be data space performance data; or the historical actual performance data in this embodiment may also be data recovery performance data. This embodiment does not limit the specific cleaning method. For example, the cleaning in this embodiment may process missing values; or the cleaning in this embodiment may also process abnormal values. In order to improve the prediction accuracy, this embodiment can make predictions based on the actual operating environment and faults of the current system. Accordingly, during training, it is necessary to obtain the actual operating environment corresponding to the fault, so that accurate predictions can be made based on the actual operating environment and faults.

[0071] S1042, analyzing the performance test data to obtain test performance influencing parameters.

[0072] S1043, comprehensively process the predicted performance impact parameters and the tested performance impact parameters to determine the performance impact parameters corresponding to each target fault.

[0073] This embodiment does not limit the specific method of comprehensive processing of the predicted performance impact parameters and the tested performance impact parameters. For example, the comprehensive processing in this embodiment may be averaging processing; the comprehensive processing in this embodiment may also be data fusion processing. When determining the performance impact parameters corresponding to each target fault, this embodiment will perform comprehensive processing based on the predicted performance impact parameters and the performance impact parameters obtained by the test to obtain the performance impact parameters corresponding to each target fault, thereby making the performance impact parameters determined more accurately.

[0074] It should be further explained that, based on any of the above embodiments, in order to improve the processing efficiency of faults that have a greater impact on performance, after analyzing the performance test data and determining the performance impact parameters corresponding to each target fault, it can also include: comparing the performance impact parameters corresponding to each target fault with the set minimum performance impact parameters; when the performance impact parameter is lower than the set minimum performance impact parameter, determining the priority of the current target fault to be a low priority; when the performance impact parameter is not lower than the set minimum performance impact parameter, determining the priority of the current target fault to be a high priority. This embodiment does not limit the specific minimum performance impact parameter. For example, the minimum performance impact parameter in this embodiment is 70; or the minimum performance impact parameter in this embodiment can also be 80. In this embodiment, the performance impact parameter corresponding to each target fault is compared with the set minimum performance impact parameter to divide the priority of the fault, so that when multiple faults occur, high-priority faults can be processed first to reduce the impact on system performance.

[0075] A method for determining the impact of a fault provided by an embodiment of the present invention may include: S101, using a virtual machine template in a hyper-converged system to batch deploy business virtual machines to obtain deployed business virtual machines; S102, based on a test tool in the virtual machine template, performing a business stress test for each deployed business virtual machine to obtain a business virtual machine with business stress; S103, injecting a target fault into a business virtual machine with business stress to perform a fault impact test on performance to obtain performance test data; S104, analyzing the performance test data to determine the performance impact parameter corresponding to each target fault. Compared with the current inability to determine the impact of a fault on performance in a hyper-converged system, the present invention can detect the impact of a fault on the performance of a hyper-converged system by deploying business virtual machines, starting business stress and fault injection, thereby improving the comprehensiveness of determining the impact of a fault on a hyper-converged system.

[0076] In order to make the present invention easier to understand, please refer to Figure 2 , Figure 2 An example flow chart of a method for determining a fault impact provided by an embodiment of the present invention may specifically include:

[0077] S201, importing a virtual machine template into a hyper-converged system; wherein the virtual machine template includes a test tool and a business model configuration file.

[0078] This embodiment deploys the execution machine, imports the prepared virtual machine template into the hyper-converged system, and configures network information. The execution virtual machine can add two network cards, eth0 network card is a private network segment for connecting to the business virtual machine, and eth1 network card is a public network ip (address) for external ssh (network protocol for accessing remote computers) access, which is convenient for operation. All subsequent operations such as stress startup of business virtual machines, hyper-converged system fault injection, performance indicator monitoring, and test result analysis are performed on the execution machine.

[0079] S202, using the virtual machine template in the hyper-converged system to batch deploy business virtual machines, and evenly distribute the business virtual machines on each host.

[0080] This embodiment performs business pressure virtual machine deployment, and has built-in test tools and business model configuration files in the virtual machine template, and installed the Python library Paramiko (remote call). In the execution machine, the virtual machine template is imported into the hyper-converged system by calling the API interface of the hyper-converged system. Use the template to deploy virtual machines in batches, and the virtual machines are evenly distributed on each host to start business pressure. This embodiment 1) Pressure model: Simulate business pressure through two pressure testing tools, fio and sysbench. Fio mainly simulates typical IO models. Sysbench mainly simulates database oltp business. Hyper-convergence core value delivery scenarios, such as the financial industry, medical industry, and transportation industry. Summarize the business model, which mainly includes small block random read and write, large block sequential read and write, or mixed read and write. Therefore, the typical IO model is made into a fio configuration file to form a configuration file library. If there is no special requirement, the 64k mixed read and write model is used as the default pressure model. The Sysbench test command is also written to the configuration file. 2) Batch deployment of virtual machines: Through the execution machine, call the API interface of the hyper-converged system to obtain the scale of the hyper-converged cluster, and deploy virtual machines in batches to ensure that the virtual machines are evenly distributed on each host. The virtual machine turns on the DHCP function and automatically obtains an IP address to ensure private network connectivity with the execution machine. 3) Start business stress: Obtain the IP address of the business virtual machine through the layer 2 broadcast of the execution machine, write it into the configuration file, and then call the paramiko module to remotely execute commands in the business virtual machine to start stress testing. At the same time, use the iostat command to count the IO load once a second and write it into a file.

[0081] S203, starting a service stress test for each service virtual machine service to obtain a service virtual machine with service stress.

[0082] S204, injecting multiple faults into the business virtual machines with business pressure, and monitoring the business virtual machines with business pressure to obtain performance test data; wherein the faults include node power failure, network disconnection, disk unplugging, node deletion, node expansion, and node online; the performance test data includes the performance test data of the hyper-converged storage and the performance test data of the business virtual machines.

[0083] This embodiment performs fault injection. Typical fault types include node power failure, network disconnection, disk unplugging, node deletion, node expansion, and node online. The fault library can be enriched as needed. 1) Fault injection: Two types of faults are distinguished according to the fault type. Faults that will cause the business virtual machine to crash, such as host power failure, network disconnection, and node deletion; Faults that will not cause the business virtual machine to crash, such as single hard disk failure, node online, and capacity expansion. By executing the shutdown command to simulate node power failure, the iplink command to simulate network disconnection, and the command / sys / block / sd* / device / state to offline to simulate a bad disk, etc. 2) Fault recovery: The fault is also recovered through the corresponding command. This embodiment includes monitoring data in two dimensions of hyper-converged storage SDS and virtual machines. It should be noted that: the time of monitoring indicator statistics. After the business virtual machine pressure is started, the indicator statistics are performed, and after the indicator is relatively stable, continue to wait for 5 minutes to perform fault injection. After obtaining the completion of fault recovery, the monitoring is ended. 1) Hyper-converged storage monitoring: The core of the hyper-converged all-in-one machine is hyper-converged storage, which is also the focus of monitoring. The monitored indicators include SDS storage IOPS (number of input and output operations per second), throughput, data disk space usage, cache space usage, data recovery volume, data recovery rate, recovery time, data migration rate, time, etc. See the two tables below. The above indicators are obtained through system interface calls. 2) Virtual machine monitoring: Use the iostat command in each virtual machine to count the io situation once per second and save it to a file. The monitoring data of the virtual machine needs to be statistically analyzed by distinguishing the fault type. If the fault causes the host to crash, select one virtual machine on other normal hosts for data recovery. If the crash cannot crash the corresponding performance of the host, data performance statistics are collected from normal virtual machines. Monitor hyper-converged storage, including fault recovery rate, fault recovery time, recovery data volume, performance degradation ratio, monitor the IO performance degradation ratio and degradation duration of the virtual machine. If the fault does not cause the virtual machine to crash, select one virtual machine from the host where the fault is injected and other normal hosts for IO load statistics and analysis. This embodiment distinguishes whether a fault will cause a virtual machine to crash, selects virtual machines to be monitored differently, monitors only virtual machines that will not crash, and determines the monitoring time according to the impact range of the fault time, thereby reducing unnecessary data statistics and performing accurate testing.

[0084] S205, determining the impact of the fault on the performance based on the performance test data obtained through monitoring, and obtaining a performance impact report and performance impact parameters.

[0085] This embodiment collects monitoring data, evaluates the impact of the fault on the system and virtual machine performance, and outputs a report. This embodiment can determine the performance impact report and performance impact parameters by comprehensively evaluating and analyzing the virtual machine performance degradation ratio, trough duration, hyper-converged storage IOPS, throughput, data disk space usage, cache space usage, data recovery volume, data recovery rate, recovery time, data migration rate, time, etc. in the performance test data. Provide more targeted data support for system optimization and business operation and maintenance. This embodiment can establish a performance baseline based on the performance data during normal operation to provide a reference for subsequent performance comparison. Fault impact analysis: Compare the performance data before and after the fault and analyze the specific impact of the fault on the system performance. For example, if the IOPS drops from 5000 to 1000 after the SSD is unplugged, the performance drops by 80%, which indicates that the SSD is critical to the system performance. Performance degradation ratio calculation: Calculate the performance degradation ratio, which can be obtained by (pre-fault performance value-post-fault performance value / pre-fault performance value*100%). Performance impact report generation: The analysis results are organized into a report, including fault description, performance data comparison, performance degradation ratio, possible cause analysis, and recommended improvement measures.

[0086] The performance impact determination method provided by the embodiment of the present invention has the following beneficial effects by linking the execution machine deployment, business pressure simulation, fault injection and performance monitoring: 1. Close to the real business pressure simulation, improve the quality of simulation test. The business model selects the core value scenario of hyper-convergence, abstracted as an IO model, close to the real usage, and the simulation test effect is more effective. 2. Comprehensively monitor indicators from two dimensions: virtual machine and hyper-convergence storage.

[0087] The following is an introduction to a fault impact determination device provided in an embodiment of the present invention. The fault impact determination device described below and the fault impact determination method described above can be referenced to each other.

[0088] Figure 3 A schematic diagram of a structure of a fault impact determination device provided by an embodiment of the present invention may include:

[0089] A deployment module 100 is used to deploy business virtual machines in batches using virtual machine templates in a hyper-converged system to obtain deployed business virtual machines;

[0090] Starting the business pressure module 200, which is used to perform a business pressure test on each deployed business virtual machine based on the test tool in the virtual machine template to obtain a business virtual machine with business pressure;

[0091] A fault injection module 300 is used to inject a target fault into the business virtual machine with business pressure to test the impact of the fault on performance and obtain performance test data;

[0092] The performance impact determination module 400 is used to analyze the performance test data and determine the performance impact parameter corresponding to each target fault.

[0093] Further, based on the above embodiment, starting the service pressure module 200 may include:

[0094] Start the business pressure unit, which is used to perform business pressure testing for each deployed business virtual machine based on different types of testing tools in the virtual machine template to obtain the business virtual machine with business pressure; wherein the type of testing tool is determined based on the type of business pressure.

[0095] Further, based on any of the above embodiments, the above fault impact determination device may further include:

[0096] A comparison module, used to compare the performance impact parameter corresponding to each target fault with the set minimum performance impact parameter;

[0097] A low priority determination module, configured to determine that the priority of the current target fault is a low priority when the performance impact parameter is lower than the set minimum performance impact parameter;

[0098] The high priority determination module is used to determine that the priority of the current target fault is a high priority when the performance impact parameter is not lower than the set minimum performance impact parameter.

[0099] Further, based on any of the above embodiments, the performance impact determination module 400 may include:

[0100] A multi-aspect analysis unit, used to perform data space impact analysis, cache space analysis and data recovery analysis on the performance test data to determine multiple performance impact parameters corresponding to the target fault;

[0101] The performance impact parameter determination unit is used to perform comprehensive processing based on the multiple performance impact parameters to determine the performance impact parameter corresponding to each target fault.

[0102] Further, based on any of the above embodiments, the performance impact determination module 400 may include:

[0103] A prediction performance impact parameter determination unit, used to determine the prediction performance impact parameter corresponding to each target fault using a target machine learning model;

[0104] Test performance analysis, used to analyze the performance test data to obtain test performance influencing parameters;

[0105] A comprehensive processing unit, used for comprehensively processing the predicted performance influencing parameters and the tested performance influencing parameters to determine the performance influencing parameters corresponding to each target fault;

[0106] Among them, the training process of the target machine learning model includes: obtaining historical target failures and the corresponding historical actual performance data when the historical target failures occur; cleaning the historical actual performance data to obtain the cleaned historical actual performance data; using the historical target failures and the corresponding historical actual performance data to train the machine learning model to obtain an initial machine learning model; using the verification set to determine the performance of the initial machine learning model, and adjusting the parameters of the initial machine learning model based on the verification results to obtain the target machine learning model.

[0107] Further, based on any of the above embodiments, the target failure includes at least two of node power failure, node network disconnection, disk unplugging, node deletion, node expansion, and node online.

[0108] Further, based on any of the above embodiments, the above fault injection module 300 may include:

[0109] The fault injection unit is used to inject the target fault into the business virtual machine with business pressure to test the impact of the fault on performance, and obtain performance test data corresponding to the hyper-converged storage and performance test data corresponding to the virtual machine.

[0110] It should be noted that the order of the modules and units in the above-mentioned fault impact determination device can be changed without affecting the logic.

[0111] Figure 3 The description of the features in the corresponding embodiments can be found in Figure 3 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0112] The fault impact determination device provided by the embodiment of the present invention may include: a deployment module 100, which is used to batch deploy business virtual machines using a virtual machine template in a hyper-converged system to obtain deployed business virtual machines; a business pressure start-up module 200, which is used to perform a business pressure test for each deployed business virtual machine based on the test tool in the virtual machine template to obtain a business virtual machine with business pressure; a fault injection module 300, which is used to inject a target fault into the business virtual machine with business pressure to perform a fault impact test on performance to obtain performance test data; a performance impact determination module 400, which is used to analyze the performance test data to determine the performance impact parameter corresponding to each target fault. Compared with the current inability to determine the impact of a fault on performance in a hyper-converged system, the present invention can detect the impact of a fault on the performance of a hyper-converged system by deploying a business virtual machine, starting business pressure and fault injection, thereby improving the comprehensiveness of determining the impact of a fault on a hyper-converged system.

[0113] A fault impact determination device provided by an embodiment of the present invention is introduced below. The fault impact determination device described below and the fault impact determination method described above can refer to each other.

[0114] Figure 4 A schematic diagram of a fault impact determination device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the fault impact determination device includes: a memory 60 for storing a computer program;

[0115] The processor 61 is used to implement the steps of the fault impact determination method in the above embodiment when executing a computer program.

[0116] The fault impact determination device provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0117] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0118] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the fault impact determination method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to data required for fault impact determination, etc.

[0119] In some embodiments, the fault impact determination device may further include a display screen 62 , an input and output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0120] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the fault impact determination device, and may include more or less components than those shown in the figure.

[0121] It is understandable that if the fault impact determination method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0122] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned fault impact determination method are implemented.

[0123] The above is a detailed introduction to a method for determining the impact of a fault provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0124] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0125] The above is a detailed introduction to a fault impact determination method, device, equipment and readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for determining the impact of a fault, characterized in that: Applicable to hyper-converged systems, including: Deploy business virtual machines in batches using virtual machine templates in the hyper-converged system to obtain deployed business virtual machines; Based on the test tool in the virtual machine template, a business stress test is performed on each deployed business virtual machine to obtain a business virtual machine with business stress; Injecting the target fault into the business virtual machine with business pressure to test the impact of the fault on performance, and obtaining performance test data; The performance test data is analyzed to determine the performance impact parameter corresponding to each target fault.

2. The fault impact determination method according to claim 1, characterized in that: Based on the test tool in the virtual machine template, a business stress test is performed for each deployed business virtual machine to obtain a business virtual machine with business stress, including: Based on different types of testing tools in the virtual machine template, a business stress test is performed for each deployed business virtual machine to obtain the business virtual machine with business stress; wherein the type of testing tool is determined based on the type of business stress.

3. The fault impact determination method according to claim 1, characterized in that: After analyzing the performance test data to determine the performance impact parameter corresponding to each target fault, the method further includes: Compare the performance impact parameter corresponding to each target fault with the set minimum performance impact parameter; When the performance impact parameter is lower than the set minimum performance impact parameter, determining that the priority of the current target fault is a low priority; When the performance impact parameter is not lower than the set minimum performance impact parameter, it is determined that the priority of the current target fault is a high priority.

4. The fault impact determination method according to any one of claims 1 to 3, characterized in that: The performance test data is analyzed to determine the performance impact parameters corresponding to each target fault, including: Performing data space impact analysis, cache space analysis and data recovery analysis on the performance test data to determine a plurality of performance impact parameters corresponding to the target fault; Comprehensive processing is performed based on the multiple performance impact parameters to determine the performance impact parameter corresponding to each target fault.

5. The fault impact determination method according to claim 1, characterized in that: The performance test data is analyzed to determine the performance impact parameters corresponding to each target fault, including: Determine the predicted performance impact parameters corresponding to each target fault using the target machine learning model; Analyzing the performance test data to obtain test performance influencing parameters; Comprehensively processing the predicted performance impact parameter and the tested performance impact parameter to determine the performance impact parameter corresponding to each target fault; The training process of the target machine learning model includes: Obtaining historical target failures and corresponding historical actual performance data when the historical target failures occurred; Cleaning the historical actual performance data to obtain cleaned historical actual performance data; Using the historical target failures and the corresponding historical actual performance data to train a machine learning model to obtain an initial machine learning model; The performance of the initial machine learning model is determined using a validation set, and the parameters of the initial machine learning model are adjusted based on the validation results to obtain the target machine learning model.

6. The fault impact determination method according to claim 1, characterized in that: The target failure includes at least two of node power failure, node network disconnection, disk unplugging, node deletion, node expansion and node online.

7. The fault impact determination method according to claim 1, characterized in that: Injecting the target fault into the business virtual machine with business pressure to test the impact of the fault on performance, and obtaining performance test data, including: The target fault is injected into the business virtual machine with business pressure to test the impact of the fault on performance, and performance test data corresponding to the hyper-converged storage and performance test data corresponding to the virtual machine are obtained.

8. A fault impact determination device, characterized in that: Applicable to hyper-converged systems, including: A deployment module is used to deploy business virtual machines in batches using virtual machine templates in the hyper-converged system to obtain deployed business virtual machines; Starting a business pressure module, which is used to perform a business pressure test on each deployed business virtual machine based on the test tool in the virtual machine template to obtain a business virtual machine with business pressure; A fault injection module is used to inject a target fault into the business virtual machine with business pressure to test the impact of the fault on performance and obtain performance test data; The performance impact determination module is used to analyze the performance test data and determine the performance impact parameter corresponding to each target fault.

9. A fault impact determination device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the fault impact determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the fault impact determination method according to any one of claims 1 to 7 are implemented.

Images