Fault test method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of cloud computing fault testing, in particular to a fault testing method and device, electronic equipment and a storage medium. Obtaining at least one to-be-injected fault component in the cloud platform, and setting a set load corresponding to each to-be-injected fault component; obtaining at least one fault operation corresponding to each to-be-injected fault component; and under the condition that each to-be-injected fault component is in a set load, injecting each fault operation to each to-be-injected fault component, and monitoring a fault test result corresponding to each to-be-injected fault component. Therefore, component and fault operation can be refined, and the granularity of fault testing can be refined. And there is no need to randomly construct some fault stock-by-stock injection clusters. In addition, the fault test results corresponding to the to-be-injected fault components are monitored, so that some fault test experiences can be accumulated according to the test results, and subsequent fault tests are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing fault testing, and in particular to a fault testing method, device, electronic equipment and storage medium. Background Art

[0002] In recent years, the cloud computing field has shown a rapid development momentum, and the competition among major cloud computing vendors is fierce. In addition to the continuous enrichment and innovation of product forms, failure rate and failure recovery efficiency have become the focus of the industry. The performance of these two aspects is crucial to the reliability and availability of cloud computing services, and directly affects customer satisfaction and business stability. Therefore, in this highly competitive market, cloud computing providers are scrambling to improve their fault management and recovery capabilities to meet the growing needs of customers, driving continuous progress and innovation in the field of cloud computing.

[0003] Currently, during the product testing phase, multiple means are usually used to simulate various real-world network environments and usage scenarios to reveal as many product reliability issues as possible. In addition to conventional API stress testing, we also conduct fault injection testing at the same time. This parallel testing method is designed to verify the performance specifications of the product, while also exposing the stability and reliability of the product in the face of various unpredictable failure situations. This comprehensive testing strategy not only helps to discover and solve potential problems, but also injects a certain amount of performance pressure into the cloud platform, because many real-world network failures are often directly related to the performance of the system under high pressure.

[0004] However, most of the existing fault tests are clumsy and randomly construct some faults and inject them into the cluster. Then, the reliability of the components is verified by observing the success rate of specific business requests. The granularity is coarse and it is impossible to accumulate some fault testing experience.

[0005] Therefore, how to perform fine-grained fault testing on the cluster has become an urgent problem to be solved. Summary of the invention

[0006] In view of this, the present invention provides a fault testing method, device, electronic device and storage medium to solve the problem of how to perform fine-grained fault testing on a cluster.

[0007] In a first aspect, the present invention provides a fault testing method, the method comprising:

[0008] Obtain at least one fault component to be injected in the cloud platform, and set a set load corresponding to each fault component to be injected;

[0009] Obtain at least one fault operation corresponding to each fault component to be injected;

[0010] When each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected, and the fault test result corresponding to each fault component to be injected is monitored.

[0011] The fault testing method provided in the embodiment of the present application obtains at least one fault component to be injected in the cloud platform, so that faults can be injected into each fault component to be injected in a targeted manner. The set load corresponding to each fault component to be injected is set, so that each fault component to be injected can be fault tested under various loads. Then, at least one fault operation corresponding to each fault component to be injected is obtained; when each fault component to be injected is under the set load, each fault operation is injected into each fault component to be injected, thereby achieving refinement to components and fault operations, and refining the granularity of fault testing. There is no need to randomly construct some faults and inject them into the cluster all at once. In addition, the fault test results corresponding to each fault component to be injected are monitored, so that some fault testing experience can be accumulated based on the test results, which is convenient for subsequent fault testing.

[0012] In an optional implementation, when each fault component to be injected is under a set load, injecting each fault operation into each fault component to be injected includes:

[0013] When each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected according to a preset sequence.

[0014] The fault testing method provided in the embodiment of the present application injects each fault operation into each fault component to be injected in a preset order when each fault component to be injected is under a set load, thereby avoiding injecting each fault operation into each fault component to be injected at the same time, causing the cloud platform to be unable to work normally.

[0015] In an optional implementation, the preset sequence includes obtaining the sequence of each fault component to be injected, and when each fault component to be injected is under a set load, injecting each fault operation into each fault component to be injected according to the preset sequence, including:

[0016] When each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected according to the order in which each fault component to be injected is obtained.

[0017] The fault testing method provided in the embodiment of the present application, when each fault component to be injected is under a set load, injects each fault operation into each fault component to be injected in the order in which each fault component to be injected is obtained, thereby avoiding injecting each fault operation into each fault component to be injected at the same time, causing the cloud platform to be unable to work normally.

[0018] In an optional implementation, the preset sequence includes the sequence of each fault component to be injected specified by the user, and when each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected according to the preset sequence, including:

[0019] When each of the components to be injected with faults is under a set load, each fault operation is injected into each of the components to be injected with faults according to the order of the components to be injected with faults specified by the user.

[0020] The fault testing method provided in the embodiment of the present application, when each fault component to be injected is under a set load, injects each fault operation into each fault component to be injected in the order of each fault component to be injected specified by the user, so that the order of injecting fault operations into each fault component to be injected can be set according to user needs to meet user needs.

[0021] In an optional implementation, the preset order includes the order of the current loads corresponding to the fault components to be injected from large to small. When the fault components to be injected are under the set load, the fault operations are injected into the fault components to be injected according to the preset order, including:

[0022] Monitor the current load corresponding to each component to be injected with faults;

[0023] Compare the current loads corresponding to the components to be injected with faults;

[0024] When each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected in descending order of the current load corresponding to each fault component to be injected.

[0025] The fault testing method provided in the embodiment of the present application monitors the current load corresponding to each fault component to be injected, so that the current load corresponding to each fault component to be injected can be obtained. The current loads corresponding to each fault component to be injected are compared; when each fault component to be injected is under the set load, each fault operation is injected into each fault component to be injected in the order from large to small according to the current load corresponding to each fault component to be injected. Thereby, it is possible to avoid injecting each fault operation into each fault component to be injected at the same time, causing the cloud platform to be unable to work normally. In addition, it is also possible to avoid injecting fault operations into fault components to be injected with smaller current loads first, causing the overall load of the cloud platform to increase, which may cause the load of fault components to be injected with larger current loads to also increase. Finally, the fault operation is injected into the fault component to be injected with larger current loads, causing the fault component to be injected with larger current loads to be unable to work normally.

[0026] In an optional implementation, monitoring the fault test results corresponding to each fault component to be injected includes:

[0027] Monitoring the working parameters of each fault component to be injected when recovering each fault operation and the sub-success rate of each fault component to be injected when recovering each fault operation;

[0028] For each fault component to be injected, determine the fault recovery success rate corresponding to the fault component to be injected according to the sub-success rate of recovery of each fault operation by the fault component to be injected;

[0029] According to the failure recovery success rate and working parameters corresponding to each failure component to be injected, the failure test result corresponding to each failure component to be injected is determined.

[0030] The fault testing method provided by the embodiment of the present application monitors the working parameters when each fault component to be injected recovers each fault operation and the sub-success rate at which each fault component to be injected recovers each fault operation. Thus, it is possible to determine whether each fault component to be injected is abnormal according to the working parameters when each fault component to be injected recovers each fault operation, and to determine the fault recovery success rate corresponding to the fault component to be injected according to the sub-success rate at which each fault component to be injected recovers each fault operation. Then, for each fault component to be injected, the fault recovery success rate corresponding to the fault component to be injected is determined according to the sub-success rate at which the fault component to be injected recovers each fault operation, thereby ensuring the accuracy of the fault recovery success rate corresponding to the determined fault component to be injected. Then, according to the fault recovery success rate and working parameters corresponding to each fault component to be injected, the fault test results corresponding to each fault component to be injected are determined, thereby ensuring the accuracy of the fault test results corresponding to each fault component to be injected, and then some fault test experience can be accumulated according to the fault test results, which is convenient for subsequent fault testing.

[0031] In an optional implementation, determining the fault test result corresponding to each fault component to be injected according to the fault recovery success rate and working parameters corresponding to each fault component to be injected includes:

[0032] If the fault recovery success rate corresponding to each component to be injected with faults is greater than or equal to the preset fault recovery success rate, and the working parameters are within the preset range, then the fault test result corresponding to each component to be injected with faults is determined to be a test success;

[0033] If the fault recovery success rate corresponding to each component to be injected with faults is greater than or equal to the preset fault recovery success rate, and there is a working parameter that exceeds the preset range, then the fault test result corresponding to each component to be injected with faults is determined to be a test success, and a fault warning is output for the working parameter that exceeds the preset range;

[0034] If the fault recovery success rate corresponding to the fault component to be injected is lower than the preset fault recovery success rate, the fault test determines that the fault test result corresponding to each fault component to be injected is a test failure, and outputs the reason for the test failure based on the working parameters and fault recovery success rate corresponding to each fault group to be injected.

[0035] The fault testing method provided in the embodiment of the present application, if the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and the working parameters are within the preset range, then the fault test result corresponding to each fault component to be injected is determined to be a test success, ensuring the accuracy of the fault test result corresponding to each fault component to be injected as a test success. If the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and there are working parameters exceeding the preset range, then the fault test result corresponding to each fault component to be injected is determined to be a test success, and a fault warning is output for the working parameters exceeding the preset range, ensuring the accuracy of the generated fault warning, and then the test experience can be accumulated according to the fault warning. If there is a fault recovery success rate corresponding to the fault component to be injected that is less than the preset fault recovery success rate, the fault test determines that the fault test result corresponding to each fault component to be injected is a test failure, and outputs the test failure reason according to the working parameters and fault recovery success rate corresponding to each fault group to be injected, ensuring the accuracy of the output test failure reason.

[0036] In a second aspect, the present invention provides a fault testing device, the device comprising:

[0037] A first acquisition module is used to acquire at least one fault component to be injected in the cloud platform, and set a set load corresponding to each fault component to be injected;

[0038] A second acquisition module is used to acquire at least one fault operation corresponding to each fault component to be injected;

[0039] The injection module is used to inject each fault operation into each fault component to be injected when each fault component to be injected is under a set load, and monitor the fault test result corresponding to each fault component to be injected.

[0040] The fault testing device provided in the embodiment of the present application obtains at least one fault component to be injected in the cloud platform, so that faults can be injected into each fault component to be injected in a targeted manner. The set load corresponding to each fault component to be injected is set, so that each fault component to be injected can be fault tested under various loads. Then, at least one fault operation corresponding to each fault component to be injected is obtained; when each fault component to be injected is under the set load, each fault operation is injected into each fault component to be injected, thereby achieving refinement to components and fault operations, and refining the granularity of fault testing. There is no need to randomly construct some faults and inject them into the cluster all at once. In addition, the fault test results corresponding to each fault component to be injected are monitored, so that some fault testing experience can be accumulated based on the test results, which is convenient for subsequent fault testing.

[0041] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the fault testing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the fault testing method of the first aspect or any corresponding embodiment thereof.

[0043] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the fault testing method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a fault testing method according to an embodiment of the present invention;

[0046] Figure 2 is a flow chart of another fault testing method according to an embodiment of the present invention;

[0047] Figure 3 is a timing diagram of a fault testing method according to an embodiment of the present invention;

[0048] Figure 4 is an overall flow chart of a fault testing method according to an embodiment of the present invention;

[0049] Figure 5 is a structural block diagram of a fault testing device according to an embodiment of the present invention;

[0050] Figure 6 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] In recent years, the cloud computing field has shown a rapid development momentum, and the competition among major cloud computing vendors is fierce. In addition to the continuous enrichment and innovation of product forms, failure rate and failure recovery efficiency have become the focus of the industry. The performance of these two aspects is crucial to the reliability and availability of cloud computing services, and directly affects customer satisfaction and business stability. Therefore, in this highly competitive market, cloud computing providers are scrambling to improve their fault management and recovery capabilities to meet the growing needs of customers, driving continuous progress and innovation in the field of cloud computing.

[0053] Currently, during the product testing phase, multiple means are usually used to simulate various real-world network environments and usage scenarios to reveal as many product reliability issues as possible. In addition to conventional API stress testing, we also conduct fault injection testing at the same time. This parallel testing method is designed to verify the performance specifications of the product, while also exposing the stability and reliability of the product in the face of various unpredictable failure situations. This comprehensive testing strategy not only helps to discover and solve potential problems, but also injects a certain amount of performance pressure into the cloud platform, because many real-world network failures are often directly related to the performance of the system under high pressure.

[0054] However, most of the existing fault tests are clumsy and randomly construct some faults and inject them into the cluster. Then, the reliability of the components is verified by observing the success rate of specific business requests. The granularity is coarse and it is impossible to accumulate some fault testing experience.

[0055] Therefore, how to perform fine-grained fault testing on the cluster has become an urgent problem to be solved.

[0056] Based on the above problems, an embodiment of the present application provides a fault testing method, which obtains at least one fault component to be injected in the cloud platform, so that faults can be injected into each fault component to be injected in a targeted manner. The set load corresponding to each fault component to be injected is set, so that each fault component to be injected can be fault tested under various loads. Then, at least one fault operation corresponding to each fault component to be injected is obtained; when each fault component to be injected is under the set load, each fault operation is injected into each fault component to be injected, thereby achieving refinement to components and fault operations, and refining the granularity of fault testing. There is no need to randomly construct some faults and inject them into the cluster all at once. In addition, the fault test results corresponding to each fault component to be injected are monitored, so that some fault testing experience can be accumulated based on the test results, which is convenient for subsequent fault testing.

[0057] It should be noted that the method for fault testing provided in the embodiment of the present application, the execution subject may be a device for fault testing, and the device for fault testing may be implemented as part or all of an electronic device through software, hardware, or a combination of software and hardware, wherein the electronic device may be a server or a terminal, wherein the server in the embodiment of the present application may be a single server or a server cluster composed of multiple servers, and the terminal in the embodiment of the present application may be a smart phone, a personal computer, a tablet computer, a wearable device, an intelligent robot, or other intelligent hardware devices. In the following method embodiments, the execution subject is an electronic device as an example for explanation.

[0058] According to an embodiment of the present invention, a fault testing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In this embodiment, a fault testing method is provided, which can be used for the above-mentioned electronic equipment. Figure 1 is a flow chart of a fault testing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0060] Step S101, obtaining at least one fault component to be injected in the cloud platform, and setting a set load corresponding to each fault component to be injected.

[0061] Optionally, the electronic device may receive at least one fault component to be injected in the cloud platform input by a user.

[0062] Optionally, the electronic device may also receive a faulty component type in the cloud platform input by a user, and then determine at least one faulty component to be injected according to the faulty component type.

[0063] Exemplarily, the user may input the fault component type based on the following function:

[0064] component=[glance,cinder,nova,neutron];

[0065] Among them, glance represents the type of faulty component used to complete image upload and download requests; cinder represents the type of faulty component used to complete volume creation, deletion, and query requests; nova represents the type of faulty component used to complete volume cloud host creation, deletion, and query requests; neutron represents the type of faulty component used to complete network resource creation, deletion, and query requests. The electronic device determines at least one faulty component to be injected based on the type of the faulty component.

[0066] Then, the electronic device may set the set load corresponding to each fault component to be injected according to the fault test requirements or according to the user input.

[0067] It should be noted that the set load corresponding to each fault component to be injected can be changed according to the fault test requirements or user input, so that the scene pressure of the fault test can be controlled more flexibly. In addition, the set load corresponding to each fault component to be injected can be the same or different, and the embodiment of the present application does not specifically limit the set load corresponding to each fault component to be injected.

[0068] Step S102: obtaining at least one fault operation corresponding to each fault component to be injected.

[0069] Specifically, the electronic device may receive at least one fault operation corresponding to each fault component to be injected input by the user, or may generate at least one fault operation corresponding to each fault component to be injected according to the fault component type corresponding to each fault component to be injected.

[0070] Exemplarily, the user may input at least one fault operation corresponding to each fault component to be injected based on the following function.

[0071] fault_operation=[power_off,reboot,kill_process];

[0072] Among them, power_off means shutting down, reboo means restarting, and kill_process means deleting the process.

[0073] Step S103 , when each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected, and the fault test result corresponding to each fault component to be injected is monitored.

[0074] Specifically, when each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected. Then, the electronic device monitors each fault component to be injected and determines the fault test result corresponding to each fault component to be injected.

[0075] This step will be described in detail below.

[0076] The fault testing method provided in the embodiment of the present application obtains at least one fault component to be injected in the cloud platform, so that faults can be injected into each fault component to be injected in a targeted manner. The set load corresponding to each fault component to be injected is set, so that each fault component to be injected can be fault tested under various loads. Then, at least one fault operation corresponding to each fault component to be injected is obtained; when each fault component to be injected is under the set load, each fault operation is injected into each fault component to be injected, thereby achieving refinement to components and fault operations, and refining the granularity of fault testing. There is no need to randomly construct some faults and inject them into the cluster all at once. In addition, the fault test results corresponding to each fault component to be injected are monitored, so that some fault testing experience can be accumulated based on the test results, which is convenient for subsequent fault testing.

[0077] In this embodiment, a fault testing method is provided, which can be used for the above-mentioned electronic equipment. Figure 2 is a flow chart of a fault testing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0078] Step S201, obtaining at least one fault component to be injected in the cloud platform, and setting a set load corresponding to each fault component to be injected.

[0079] For details about this step, please refer to the above description of step S101, which will not be elaborated here.

[0080] Step S202: obtaining at least one fault operation corresponding to each fault component to be injected.

[0081] For details about this step, please refer to the above description of step S102, which will not be elaborated here.

[0082] Step S203 , when each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected, and the fault test result corresponding to each fault component to be injected is monitored.

[0083] Specifically, the above step S203 may include the following steps:

[0084] Step S2031 , when each fault component to be injected is under a set load, inject each fault operation into each fault component to be injected in a preset order.

[0085] In an optional implementation manner of the present application, the preset order includes the order of obtaining each fault component to be injected, and the above step S2031 may include the following steps:

[0086] Step a1: when each fault component to be injected is under a set load, inject each fault operation into each fault component to be injected according to the order in which each fault component to be injected is obtained.

[0087] Specifically, when each of the components to be injected with faults is under a set load, the electronic device may inject each fault operation into each of the components to be injected with faults according to the order in which each of the components to be injected with faults is acquired.

[0088] The order of obtaining the faulty components to be injected may be the order in which the user inputs the faulty components to be injected, or the order in which the electronic device obtains the faulty component types corresponding to the faulty components to be injected. Then, according to the order in which the faulty component types corresponding to the faulty components to be injected are obtained, each fault operation is injected into each faulty component to be injected.

[0089] In an optional implementation manner of the present application, the preset order includes the order of each component to be injected with faults specified by the user, and the above step S2031 may include the following steps:

[0090] Step a2: when each fault component to be injected is under a set load, inject each fault operation into each fault component to be injected according to the order of each fault component to be injected specified by the user.

[0091] Specifically, the electronic device may receive the order of each fault component to be injected specified by the user, and when each fault component to be injected is under a set load, inject each fault operation into each fault component to be injected according to the order of each fault component to be injected specified by the user.

[0092] Optionally, the electronic device can also receive an injection sequence corresponding to each fault component type specified by the user, and then, when each fault component to be injected is under a set load, inject each fault operation into each fault component to be injected corresponding to each fault component type according to the injection sequence corresponding to each fault component type specified by the user.

[0093] Exemplarily, the user specifies glance as the first injected fault component type, cinder as the third injected fault component type, nova as the fourth injected fault component type, and neutron as the second injected fault component type.

[0094] In an optional implementation manner of the present application, the preset order includes the order of the current loads corresponding to the faulty components to be injected from large to small, and the above step S2031 may include the following steps:

[0095] Step a3: monitor the current load corresponding to each component to be injected with faults.

[0096] Specifically, the electronic device can monitor the current in the circuit using a Hall effect sensor or a current transformer, the electronic device can also monitor the voltage in the circuit using a voltage divider or a voltage transformer, and the electronic device can also monitor the temperature using a thermocouple, a resistance temperature detector (RTD), or a thermistor (NTC / PTC).

[0097] Then, the electronic device determines the current load corresponding to each component to be injected with faults according to the monitored current and / or voltage and / or temperature.

[0098] Step a4, comparing the current loads corresponding to the faulty components to be injected.

[0099] Specifically, the electronic device may compare the current loads corresponding to the faulty components to be injected, and then sort the faulty components to be injected in descending order of the current loads.

[0100] Step a5: when each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected in descending order of the current load corresponding to each fault component to be injected.

[0101] Specifically, when each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected in a descending order of the current load corresponding to each fault component to be injected.

[0102] Step S2032, monitoring the working parameters of each to-be-injected fault component when recovering each fault operation and the sub-success rate of each to-be-injected fault component recovering each fault operation.

[0103] Specifically, the electronic device can use a SCADA (Supervisory Control and Data Acquisition) system to monitor in real time the working parameters of each component to be injected with faults when recovering each fault operation. Among them, the working parameters may include parameters such as temperature, pressure, current, voltage, flow, speed, etc. Then, the electronic device can also use a log recording system to record the number of attempts and successes of the fault operation.

[0104] The electronic device uses a data analysis tool to process and analyze the collected operating parameters. Specifically, the electronic device can monitor each operating parameter, compare each operating parameter with a preset range, and when the operating parameter exceeds the preset range, the system should issue an alarm.

[0105] In addition, the electronic device can also calculate a sub-success rate according to the number of attempts and the number of successes of each fault operation of each component to be injected with faults, where sub-success rate = number of successes / number of attempts.

[0106] Step S2033: for each fault component to be injected, determine the fault recovery success rate corresponding to the fault component to be injected according to the sub-success rate of recovery of each fault operation by the fault component to be injected.

[0107] Specifically, for each fault component to be injected, the electronic device may add up the sub-success rates of recovery of each fault operation according to the fault component to be injected, and then calculate the average index to determine the fault recovery success rate corresponding to the fault component to be injected.

[0108] Step S2034, determining the fault test results corresponding to the components to be injected with faults according to the fault recovery success rates and working parameters corresponding to the components to be injected with faults.

[0109] Specifically, the above step S2034 may include the following situations:

[0110] In one case, if the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and the working parameters are within the preset range, the fault test result corresponding to each fault component to be injected is determined to be a test success.

[0111] Specifically, the electronic device receives a preset fault recovery success rate input by a user, and may also set the preset fault recovery success rate according to specific circumstances.

[0112] The electronic device compares the fault recovery success rate corresponding to each component to be injected with the fault with the preset fault recovery success rate.

[0113] If the fault recovery success rate corresponding to each faulty component to be injected is greater than or equal to the preset fault recovery success rate, and the working parameters corresponding to each faulty component to be injected are within the preset range, the electronic device determines that the fault test result corresponding to each faulty component to be injected is a successful test.

[0114] In another case, if the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and there is a working parameter that exceeds the preset range, then the fault test result corresponding to each fault component to be injected is determined to be a successful test, and a fault warning is output for the working parameter that exceeds the preset range.

[0115] Specifically, if the fault recovery success rate corresponding to each component to be injected with faults is greater than or equal to the preset fault recovery success rate, and there is an operating parameter that exceeds the preset range, the electronic device determines that the fault test result corresponding to each component to be injected with faults is a test success. The electronic device determines the operating parameter that exceeds the preset range, and then outputs a fault warning for the operating parameter that exceeds the preset range.

[0116] In another case, if the fault recovery success rate corresponding to the fault component to be injected is lower than the preset fault recovery success rate, the fault test determines that the fault test result corresponding to each fault component to be injected is a test failure, and outputs the reason for the test failure based on the working parameters corresponding to each fault group to be injected and the fault recovery success rate.

[0117] Specifically, if there is a fault recovery success rate corresponding to a fault component to be injected that is less than a preset fault recovery success rate, the electronic device determines a target fault component to be injected that has a fault recovery success rate less than the preset fault recovery success rate.

[0118] The electronic device determines the sub-success rate of the target to-be-injected fault component recovering each fault operation. The electronic device compares the sub-success rate of the target to-be-injected fault component recovering each fault operation with a preset sub-success rate. The electronic device determines the fault operation whose sub-success rate in the target to-be-injected fault component is less than the preset sub-success rate as the target fault operation.

[0119] In addition, the electronic device determines that the operating parameters that exceed the preset range in the fault component to be injected are target operating parameters. Then, the electronic device determines the reason for the failure to recover the target fault operation based on the target operating operation, thereby outputting the reason for the test failure.

[0120] The fault testing method provided in the embodiment of the present application, when each fault component to be injected is under a set load, injects each fault operation into each fault component to be injected in the order in which each fault component to be injected is obtained, thereby avoiding injecting each fault operation into each fault component to be injected at the same time, causing the cloud platform to be unable to work normally.

[0121] Alternatively, when each fault component to be injected is under a set load, each fault operation is injected into each fault component to be injected according to the order of each fault component to be injected specified by the user, so that the order of injecting fault operations into each fault component to be injected can be set according to user needs to meet user needs.

[0122] Alternatively, the current load corresponding to each fault component to be injected is monitored, so that the current load corresponding to each fault component to be injected can be obtained. The current loads corresponding to each fault component to be injected are compared; when each fault component to be injected is under the set load, each fault operation is injected into each fault component to be injected in the order of the current load corresponding to each fault component to be injected from large to small. This can avoid injecting each fault operation into each fault component to be injected at the same time, causing the cloud platform to be unable to work normally. In addition, it can also avoid injecting fault operations into fault components to be injected with smaller current loads first, resulting in an increase in the overall load of the cloud platform, which may cause the load of fault components to be injected with larger current loads to also increase. Finally, the fault operation is injected into the fault component to be injected with larger current loads, causing the fault component to be injected with larger current loads to be unable to work normally.

[0123] Then, monitor the working parameters of each fault component to be injected when recovering each fault operation and the sub-success rate of each fault component to be injected when recovering each fault operation. Therefore, it is possible to determine whether each fault component to be injected is abnormal based on the working parameters of each fault component to be injected when recovering each fault operation, and determine the fault recovery success rate corresponding to the fault component to be injected based on the sub-success rate of each fault component to be injected when recovering each fault operation. Then, for each fault component to be injected, determine the fault recovery success rate corresponding to the fault component to be injected based on the sub-success rate of each fault operation recovered by the fault component to be injected, thereby ensuring the accuracy of the fault recovery success rate corresponding to the fault component to be injected. If the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and the working parameters are within the preset range, then determine that the fault test result corresponding to each fault component to be injected is a test success, thereby ensuring the accuracy of the fault test result corresponding to each fault component to be injected being a test success. If the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and there are working parameters that exceed the preset range, then the fault test results corresponding to each fault component to be injected are determined to be test success, and a fault warning is output for the working parameters that exceed the preset range, ensuring the accuracy of the generated fault warning, and then test experience can be accumulated based on the fault warning. If there is a fault recovery success rate corresponding to a fault component to be injected that is less than the preset fault recovery success rate, the fault test determines that the fault test results corresponding to each fault component to be injected are test failures, and outputs the test failure reasons based on the working parameters and fault recovery success rates corresponding to each fault group to be injected, ensuring the accuracy of the output test failure reasons.

[0124] In order to better introduce the fault testing method provided in the embodiment of the present application, Figure 3As shown, an embodiment of the present application provides a timing diagram of a fault testing method. Specifically, the stress generation tool (Perf-A) can inject simulated user requests into the cloud platform to simulate the platform performance under high load or normal load conditions. The purpose is to test the performance and stability of the cloud platform to ensure that it can still work normally under high concurrency. The fault injection module (Fault(s)) can inject fault operations into each fault component to be injected. By introducing these faults, the fault recovery capability and fault tolerance mechanism of the cloud platform can be tested. The fault timing generation module can generate a timing sequence for injected faults. The result evaluation module generates an evaluation result and returns the evaluation result.

[0125] The overall flow chart of the fault testing method can be shown as follows: Figure 4 As shown. Combining multiple sets of fault data sets, the main design of this method is as follows:

[0126] cluster_fault_classifier(component,fault_operation,model), which is the main function of the system device, includes three main variables:

[0127] Among them, component = [glance, cinder, nova, neutron], fault_operation = [power_off, reboot, kill_process], model = [Sequential, manul, passive], the internal functions of this function mainly include calling the request simulation method of the corresponding component through the passed component parameter to generate a certain pressure (the pressure can be customized), starting the timing at the same time as the pressure is started, and entering the fault injection method selector at the same time. Three fault injection methods are given by default. The Sequential method injects a fault at a fixed period of the regularity of the time node. The manul method is mainly open to the user to define the order and time point of the fault injection. The passive method is more complicated. It will start a component monitoring program at the same time as the component pressure is started, and decide what kind of fault to trigger injection through the monitoring program.

[0128] Among them, the pressure generation module: the main function of this module is to issue a certain request pressure to each component interface. The pressure is adjustable and the request type can be customized. The business request Perf(s) type can be divided into the following types according to the component: a. glance common image upload and download requests; b. cinder common volume creation, deletion, and query requests; c. nova cloud host creation, deletion, query and other requests; d. neutron network resource creation, deletion, and query requests.

[0129] Fault generation module: This module is mainly used to inject faults. For the timing of fault injection, the corresponding timing module can be used;

[0130] a: The internal implementation of the Sequential method is mainly to call each fault_operation method in sequence;

[0131] b: The internal structure of the Manul method is mainly exposed to the user to define the order of each falut_operation method;

[0132] c: The internal structure of the Passive method is relatively complex, and it mainly calls the monitoring method to obtain the monitoring status of each component. (Falut_operation supports customization and updating to prepare for adding new fault scenarios).

[0133] Specifically, the monitor method selects specific component monitoring indicators, and sets the priority of the component fault suite to the highest through pop after the judgment conditions are met. At the same time, some other order adjustment methods are set to flexibly adjust the component failure order. After setting a comprehensive evaluation of multiple monitoring indicators, the order of injected faults is finally generated, and the execution of the entire process continues. At the same time, this method can also synchronously call the pressure generation method to dynamically adjust the pressure of Perf, and more flexibly control the scenario pressure of fault testing.

[0134] In the present embodiment, a fault testing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0135] This embodiment provides a fault testing device, such as Figure 5 As shown, including:

[0136] The first acquisition module 301 is used to acquire at least one fault component to be injected in the cloud platform, and set a set load corresponding to each fault component to be injected;

[0137] The second acquisition module 302 is used to acquire at least one fault operation corresponding to each fault component to be injected;

[0138] The injection module 303 is used to inject each fault operation into each fault component to be injected when each fault component to be injected is under a set load, and monitor the fault test result corresponding to each fault component to be injected.

[0139] In some optional implementations, the injection module 303 is specifically configured to inject each fault operation into each fault component to be injected in a preset order when each fault component to be injected is under a set load.

[0140] In some optional implementations, the preset order includes the order of obtaining each fault component to be injected, and the injection module 303 is specifically used to inject each fault operation into each fault component to be injected according to the order of obtaining each fault component to be injected when each fault component to be injected is under a set load.

[0141] In some optional embodiments, the preset order includes the order of each fault component to be injected specified by the user, and the injection module 303 is specifically used to inject each fault operation into each fault component to be injected according to the order of each fault component to be injected specified by the user when each fault component to be injected is under a set load.

[0142] In some optional embodiments, the preset order includes the order of the current loads corresponding to each fault component to be injected from large to small, and the injection module 303 is specifically used to monitor the current loads corresponding to each fault component to be injected; compare the current loads corresponding to each fault component to be injected; and when each fault component to be injected is at the set load, inject each fault operation into each fault component to be injected in the order of the current loads corresponding to each fault component to be injected from large to small.

[0143] In some optional implementations, the injection module 303 is specifically used to monitor the working parameters of each fault component to be injected when recovering from each fault operation and the sub-success rate of each fault component to be injected recovering from each fault operation; for each fault component to be injected, the fault recovery success rate corresponding to the fault component to be injected is determined according to the sub-success rate of the fault component to be injected recovering from each fault operation; according to the fault recovery success rate and working parameters corresponding to each fault component to be injected, the fault test result corresponding to each fault component to be injected is determined.

[0144] In some optional embodiments, the injection module 303 is specifically used to determine that the fault test results corresponding to each fault component to be injected are test success if the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and the working parameters are within the preset range; if the fault recovery success rate corresponding to each fault component to be injected is greater than or equal to the preset fault recovery success rate, and there are working parameters exceeding the preset range, then the fault test results corresponding to each fault component to be injected are determined to be test success, and a fault warning is output for the working parameters exceeding the preset range; if there are fault recovery success rates corresponding to fault components to be injected that are less than the preset fault recovery success rate, the fault test determines that the fault test results corresponding to each fault component to be injected are test failures, and outputs the reasons for the test failure based on the working parameters and fault recovery success rates corresponding to each fault group to be injected.

[0145] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0146] The fault testing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0147] The embodiment of the present invention also provides an electronic device having the above Figure 5 Fault test setup shown.

[0148] See also Figure 6 , Figure 6 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0149] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0150] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0151] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0153] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0154] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0155] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0156] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0157] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A fault testing method, characterized in that: The method comprises: Obtain at least one fault component to be injected in the cloud platform, and set a set load corresponding to each fault component to be injected; Obtaining at least one fault operation corresponding to each of the fault components to be injected; When each of the fault components to be injected is under the set load, each of the fault operations is injected into each of the fault components to be injected, and the fault test results corresponding to each of the fault components to be injected are monitored.

2. The method according to claim 1, characterized in that Injecting each of the fault operations into each of the fault components to be injected when each of the fault components to be injected is under the set load includes: In the case where each of the components to be injected with faults is under the set load, each of the fault operations is injected into each of the components to be injected with faults in a preset order.

3. The method according to claim 2, characterized in that The preset sequence includes obtaining the sequence of each of the to-be-injected fault components, and when each of the to-be-injected fault components is under the set load, injecting each of the fault operations into each of the to-be-injected fault components according to the preset sequence, including: In the case that each of the to-be-injected fault components is under the set load, each of the fault operations is injected into each of the to-be-injected fault components in the order in which the to-be-injected fault components are acquired.

4. The method according to claim 2, characterized in that: The preset sequence includes the sequence of the fault components to be injected specified by the user. When the fault components to be injected are under the set load, injecting the fault operations into the fault components to be injected according to the preset sequence includes: In the case that each of the to-be-injected fault components is under the set load, each of the fault operations is injected into each of the to-be-injected fault components according to the order of the to-be-injected fault components specified by the user.

5. The method according to claim 2, characterized in that: The preset sequence includes the order of the current loads corresponding to the fault components to be injected from large to small. When the fault components to be injected are at the set load, injecting the fault operations into the fault components to be injected according to the preset sequence includes: Monitoring the current load corresponding to each of the components to be injected with faults; Comparing the current loads corresponding to the components to be injected with faults; In the case that each of the to-be-injected fault components is under the set load, each of the fault operations is injected into each of the to-be-injected fault components in the descending order of the current loads corresponding to each of the to-be-injected fault components.

6. The method according to claim 1, characterized in that The monitoring of the fault test results corresponding to each of the faulty components to be injected includes: Monitoring the working parameters of each of the faulty components to be injected when recovering each of the faulty operations and the sub-success rate of each of the faulty components to be injected when recovering each of the faulty operations; For each of the fault components to be injected, determining the fault recovery success rate corresponding to the fault component to be injected according to the sub-success rate of the fault component to be injected recovering each of the fault operations; According to the failure recovery success rate corresponding to each of the failure components to be injected and the working parameters, the failure test results corresponding to each of the failure components to be injected are determined.

7. The method according to claim 6, characterized in that The determining, according to the failure recovery success rate corresponding to each of the failure components to be injected and the working parameters, the failure test results corresponding to each of the failure components to be injected, comprises: If the fault recovery success rate corresponding to each of the components to be injected with faults is greater than or equal to the preset fault recovery success rate, and the working parameters are within the preset range, then the fault test result corresponding to each of the components to be injected with faults is determined to be a test success; If the fault recovery success rate corresponding to each of the components to be injected with faults is greater than or equal to the preset fault recovery success rate, and there is an operating parameter that exceeds the preset range, then the fault test result corresponding to each of the components to be injected with faults is determined to be a test success, and a fault warning is output for the operating parameter that exceeds the preset range; If the fault recovery success rate corresponding to the fault component to be injected is lower than the preset fault recovery success rate, the fault test determines that the fault test result corresponding to each fault component to be injected is a test failure, and outputs the reason for the test failure based on the working parameters corresponding to each fault group to be injected and the fault recovery success rate.

8. A fault testing device, characterized in that: The device comprises: A first acquisition module is used to acquire at least one fault component to be injected in the cloud platform, and set a set load corresponding to each fault component to be injected; A second acquisition module is used to acquire at least one fault operation corresponding to each of the fault components to be injected; The injection module is used to inject each of the fault operations into each of the fault components to be injected when each of the fault components to be injected is under the set load, and monitor the fault test results corresponding to each of the fault components to be injected.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the fault testing method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the fault testing method according to any one of claims 1 to 7.

Images