Fault testing method and device, electronic equipment and medium
Through pre-trained fault prediction models and fault injection strategies, fault testing in chaos engineering is automated, the problem of manual dependency is solved, efficiency and accuracy is improved, potential service failures are discovered in advance, and strategies are optimized to avoid risks.
Patent Information
- Application Number
- CN202411072880.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-06
AI Technical Summary
In existing chaos engineering, it relies on manual failure strategies, which are inefficient and error-prone, and rely too much on manual experience and knowledge, making it difficult to efficiently discover potential failures and optimization strategies of services.
A pre-trained fault prediction model is used to determine potential target failures based on performance data, and a fault injection strategy is generated to automatically perform fault injection on nodes of the analysis service, reducing manual participation.
Automatic fault testing is realized, efficiency and accuracy are improved, dependence on labor is reduced, potential service failures can be detected in advance, and strategies can be optimized to avoid potential risks.
Smart Images

Figure CN119088656A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of testing technology, and in particular to a fault testing method, device, electronic equipment and medium. Background Art
[0002] Chaos engineering is to proactively create failures for a service and observe the performance of the service under different failure pressures, so as to discover the weaknesses and vulnerabilities of the service in advance and determine optimization strategies in advance to avoid potential risks.
[0003] At present, when executing chaos engineering, we mainly rely on manual fault strategy formulation and inject faults into the service according to the fault strategy. For example, we manually control the shutdown of some servers running the service. This method relies too much on manual operation, which is inefficient. It also relies on manual experience and knowledge, which is prone to errors. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a fault testing method, device, electronic device and medium to reduce the dependence of chaos engineering on manual work. The specific technical solution is as follows:
[0005] In a first aspect of an embodiment of the present application, a fault testing method is provided, the method comprising:
[0006] Get the performance data of the service to be analyzed;
[0007] Determine potential target faults of the service to be analyzed based on the performance data using a pre-trained fault prediction model;
[0008] Based on the target fault, generating a fault injection strategy, wherein the fault injection strategy is a strategy for controlling the target fault to occur in the service to be analyzed;
[0009] The fault injection strategy is executed on the node to be analyzed that runs the service to be analyzed.
[0010] Optionally, obtaining the performance data of the service to be analyzed includes:
[0011] Obtaining the operating parameters generated by the node to be analyzed running the service to be analyzed within a preset time period;
[0012] Feature extraction is performed on the operating parameters to obtain performance data of the service to be analyzed.
[0013] Optionally, the using a pre-trained fault prediction model to determine a potential target fault of the service to be analyzed based on the performance data includes:
[0014] Inputting the performance data into the fault prediction model to obtain the probability of each fault existing in the service to be analyzed output by the fault prediction model;
[0015] From the probabilities of each failure, a failure corresponding to the maximum probability is selected as a potential target failure of the service to be analyzed.
[0016] Optionally, the fault prediction model is trained by the following steps:
[0017] Acquire a sample performance parameter and a training label of the sample performance parameter, wherein the training label indicates a potential sample failure of a sample service to which the sample performance parameter belongs;
[0018] Inputting the sample performance parameter into a preset machine learning model to obtain an output result of the machine learning model, wherein the output result represents a potential failure predicted for the sample service;
[0019] Determine a loss value based on the training label and the output result;
[0020] Adjust the model parameters of the machine learning model based on the loss value, return to the step of obtaining sample performance parameters and training labels of the sample performance parameters, and use the current machine learning model as the fault prediction model until the machine learning model converges.
[0021] Optionally, generating a fault injection strategy based on the target fault includes:
[0022] Obtain risk information related to a target failure of the service to be analyzed;
[0023] The target fault, the risk information and the preset policy restriction condition are input into a preset policy generation model to obtain the fault injection policy output by the policy generation model.
[0024] Optionally, after executing the fault injection strategy on the node to be analyzed that runs the service to be analyzed, the method further includes:
[0025] Obtaining current values and historical values of preset parameter items of the service to be analyzed;
[0026] Using a preset optimization algorithm, based on the current value, the historical value and a preset optimization target for the preset parameter item, modify the fault injection strategy;
[0027] The modified fault injection strategy is executed on the node to be analyzed.
[0028] Optionally, the using a preset optimization algorithm to modify the fault injection strategy based on the current value, the historical value, and a preset optimization target for the preset parameter item includes:
[0029] Training a target performance prediction model based on the current value, the fault injection strategy corresponding to the current value, the historical value, and the historical fault injection strategy corresponding to the historical value;
[0030] Determining predicted values of preset parameter items corresponding to a plurality of candidate fault injection strategies based on the target performance prediction model;
[0031] Based on the predicted value corresponding to each candidate fault injection strategy and the preset optimization target, a candidate fault injection strategy is selected as the modified fault injection strategy.
[0032] Optionally, after executing the modified fault injection strategy on the node to be analyzed, the method further includes:
[0033] Determining whether the fault injection strategy has converged based on the current value and / or the number of modifications of the fault injection strategy;
[0034] If not, return to the step of obtaining the current value and the historical value of the preset parameter item of the service to be analyzed.
[0035] According to a second aspect of an embodiment of the present application, a fault testing device is provided, the device comprising:
[0036] An acquisition module is used to obtain the performance data of the service to be analyzed;
[0037] A determination module, configured to determine a potential target fault of the service to be analyzed based on the performance data acquired by the acquisition module using a pre-trained fault prediction model;
[0038] A generating module, configured to generate a fault injection strategy based on the target fault determined by the determining module, wherein the fault injection strategy is a strategy for controlling the target fault to occur in the service to be analyzed;
[0039] An execution module is used to execute the fault injection strategy generated by the generation module on the node to be analyzed that runs the service to be analyzed.
[0040] Optionally, the acquisition module is specifically used to:
[0041] Obtaining the operating parameters generated by the node to be analyzed running the service to be analyzed within a preset time period;
[0042] Feature extraction is performed on the operating parameters to obtain performance data of the service to be analyzed.
[0043] Optionally, the determining module is specifically used to:
[0044] Inputting the performance data into the fault prediction model to obtain the probability of each fault existing in the service to be analyzed output by the fault prediction model;
[0045] From the probabilities of each failure, a failure corresponding to the maximum probability is selected as a potential target failure of the service to be analyzed.
[0046] Optionally, the device further comprises a training module, wherein the training module is used to:
[0047] Acquire a sample performance parameter and a training label of the sample performance parameter, wherein the training label indicates a potential sample failure of a sample service to which the sample performance parameter belongs;
[0048] Inputting the sample performance parameter into a preset machine learning model to obtain an output result of the machine learning model, wherein the output result represents a potential failure predicted for the sample service;
[0049] Determine a loss value based on the training label and the output result;
[0050] Adjust the model parameters of the machine learning model based on the loss value, return to the step of obtaining sample performance parameters and training labels of the sample performance parameters, and use the current machine learning model as the fault prediction model until the machine learning model converges.
[0051] Optionally, the generating module is specifically used for:
[0052] Obtain risk information related to a target failure of the service to be analyzed;
[0053] The target fault, the risk information and the preset policy restriction condition are input into a preset policy generation model to obtain the fault injection policy output by the policy generation model.
[0054] Optionally, the device further comprises:
[0055] The acquisition module is further configured to acquire the current value and the historical value of the preset parameter item of the service to be analyzed after executing the fault injection strategy on the node to be analyzed running the service to be analyzed;
[0056] A modification module, configured to modify the fault injection strategy using a preset optimization algorithm based on the current value, the historical value, and a preset optimization target for the preset parameter item;
[0057] The execution module is further used to execute the modified fault injection strategy on the node to be analyzed.
[0058] Optionally, the modification module is specifically used to:
[0059] Training a target performance prediction model based on the current value, the fault injection strategy corresponding to the current value, the historical value, and the historical fault injection strategy corresponding to the historical value;
[0060] Determining predicted values of preset parameter items corresponding to a plurality of candidate fault injection strategies based on the target performance prediction model;
[0061] Based on the predicted value corresponding to each candidate fault injection strategy and the preset optimization target, a candidate fault injection strategy is selected as the modified fault injection strategy.
[0062] Optionally, the device further comprises:
[0063] A judgment module, configured to determine whether the fault injection strategy has converged based on the current value and / or the number of times the fault injection strategy has been modified after the modified fault injection strategy is executed on the node to be analyzed;
[0064] The calling module is used for calling the obtaining module to execute the step of obtaining the current value and the historical value of the preset parameter item of the service to be analyzed if the judgment result is non-convergence.
[0065] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0066] Memory, used to store computer programs;
[0067] The processor is used to implement the fault testing method described in any one of the first aspects above when executing the program stored in the memory.
[0068] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the fault testing method described in any one of the first aspects is implemented.
[0069] According to a fifth aspect of the embodiments of the present application, there is also provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the fault testing methods described in the first aspect.
[0070] The fault testing method, device, electronic device and medium provided in the embodiments of the present application can use a pre-trained fault prediction model to determine the potential target faults of the service to be analyzed based on the performance data of the service to be analyzed. Then a fault injection strategy is generated to cause the target fault to occur in the service to be analyzed, and the fault injection strategy is executed on the node to be analyzed that runs the service to be analyzed. Since the target fault is a possible fault of the service to be analyzed, the fault injection strategy is executed for the node to be analyzed so that the target fault occurs in the service to be analyzed. The performance of the service to be analyzed when the target fault occurs can be obtained in advance, so as to avoid the potential risks of the service to be analyzed, which has implementation significance. That is, the embodiments of the present application can inject faults into the service to be analyzed in a targeted manner without the need for human intervention, thereby reducing the dependence of chaos engineering on manual work. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0072] Figure 1 A flowchart of a fault testing method provided in an embodiment of the present application;
[0073] Figure 2 A flowchart of a method for training a fault prediction model provided in an embodiment of the present application;
[0074] Figure 3 A flowchart of a method for modifying a fault injection strategy provided in an embodiment of the present application;
[0075] Figure 4 An exemplary schematic diagram of a fault testing process provided in an embodiment of the present application;
[0076] Figure 5 A schematic diagram of the structure of a fault testing device provided in an embodiment of the present application;
[0077] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0078] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0079] First, the service system where the service to be analyzed involved in the embodiment of the present application is located is described.
[0080] The service system can be a system built on a service mesh, microservice architecture, or distributed architecture. Among them, the service mesh is an infrastructure layer used to implement inter-service communication, load balancing, service discovery, security and other functions, which can be widely used in cloud native applications and microservice scenarios. Microservice architecture is an architectural model that splits an application into multiple small services. Distributed architecture is an architectural model that divides a large system into multiple small services.
[0081] The service system includes multiple services, each of which runs in one or more nodes. When a service runs in multiple nodes, each node runs the service independently. The nodes can be containers, virtual machines, or physical servers.
[0082] When the service system is built based on the service grid, an agent is also running in the node where the service is located. When two services communicate, they can communicate through the agent in the node where the two services are located. For example, when service A needs to send a message to service B, it can first send the message to agent a in the container where service A is located, and agent a then forwards the message to agent b in the container where service B is located, and then agent b forwards the message to service B.
[0083] When the service system is built based on a microservice architecture or a distributed architecture, services can communicate with each other through inter-service communication mechanisms, such as message queues.
[0084] In such a complex service system, injecting faults into the service system through chaos engineering and analyzing and optimizing the system is an important means to ensure the resilience and reliability of the service system.
[0085] In order to reduce the dependence of chaos engineering on manual work, the present application embodiment provides a fault testing method, which can be applied to a scheduling node, and the scheduling node can be any node in the above service system. Figure 1 As shown, the method comprises the following steps:
[0086] S101. Obtain performance data of a service to be analyzed.
[0087] The service to be analyzed may be an application that needs to be analyzed and optimized, or a software module such as a microservice that can be independently deployed and provide functions.
[0088] Performance data is data that can reflect the operating efficiency and / or stability of the service to be analyzed. The specific method of obtaining performance data can be referred to the description below.
[0089] S102: Determine potential target faults of the service to be analyzed based on performance data using a pre-trained fault prediction model.
[0090] Among them, the potential target failure of the service to be analyzed is: the service to be analyzed has not appeared at present, but if no maintenance or adjustment measures are taken, it may occur in the future. For example, the potential target failure of the service to be analyzed can be: response timeout, failure to verify user identity, or failure to obtain a third-party payment page, etc.
[0091] The specific method for determining the target fault can be referred to the description below.
[0092] S103: Generate a fault injection strategy based on the target fault, wherein the fault injection strategy is a strategy for controlling the target fault to occur in the service to be analyzed.
[0093] The fault injection strategy can include multiple controls on the nodes to be analyzed that run the services to be analyzed. For example, if the target fault is the inability to return a complete web page to the terminal, the fault injection strategy can be: controlling the CPU usage to not exceed 50%, controlling the memory usage to not exceed 40%, and setting the packet loss time to reduce by 10 milliseconds.
[0094] The specific method of generating a fault injection strategy can be referred to the description below.
[0095] S104: Execute a fault injection strategy on the node to be analyzed that runs the service to be analyzed.
[0096] There may be one or more nodes to be analyzed that run the service to be analyzed.
[0097] When there is only one node to be analyzed, the fault injection strategy may include the fault injected into the node.
[0098] When there are multiple nodes to be analyzed, the nodes to be injected with faults may be all or part of the nodes to be analyzed. In this case, the fault injection strategy may include the node identifiers to be injected with faults and the faults to be injected into the nodes. For example, the nodes to be analyzed include: node 1, node 2, and node 3, and the fault injection strategy includes: node 1, controlling the CPU usage to not exceed 50%, and controlling the memory usage to not exceed 40%.
[0099] The fault injection strategy is implemented on the node to be analyzed, which injects faults into the node to be analyzed, thereby causing the target fault to occur in the service to be analyzed.
[0100] After executing the fault injection strategy on the node to be analyzed, the changes in the performance parameters of the service to be analyzed can be observed, so as to obtain the stability of the service to be analyzed when the target fault occurs, and then the service to be analyzed can be optimized in advance, the ability of the service to be analyzed to respond to the target fault can be improved, and potential risks can be avoided.
[0101] The embodiments of the present application can use a pre-trained fault prediction model to determine the potential target faults of the service to be analyzed based on the performance data of the service to be analyzed. Then a fault injection strategy is generated to cause the target fault to occur in the service to be analyzed, and the fault injection strategy is executed on the node to be analyzed that runs the service to be analyzed. Since the target fault is a fault that may occur in the service to be analyzed, the fault injection strategy is executed for the node to be analyzed so that the target fault occurs in the service to be analyzed. The performance of the service to be analyzed when the target fault occurs can be obtained in advance, so as to avoid the potential risks of the service to be analyzed, which has implementation significance. That is, the embodiments of the present application can inject faults into the service to be analyzed in a targeted manner without the need for human intervention, thereby reducing the dependence of chaos engineering on manual work.
[0102] The following is a detailed description of the fault testing method provided in the embodiment of the present application:
[0103] The above-mentioned method of obtaining the performance data of the service to be analyzed in S101 can be implemented as follows: obtaining the operating parameters generated by the node to be analyzed running the service to be analyzed within a preset time period, and then performing feature extraction on the operating parameters to obtain the performance data of the service to be analyzed.
[0104] The scheduling node can collect the operation logs of the nodes to be analyzed from the log system, and extract the operation parameters generated by running the service to be analyzed within a preset time period from the operation logs. In addition, the scheduling node can collect the operation parameters generated by running the service to be analyzed within a preset time period from the monitoring system.
[0105] The preset time period may be the previous scheduling cycle, for example, if S101 is executed once every hour, the preset time period may be the hour before the current time. The operating parameters generated by running the service to be analyzed within the preset time period include: the operating parameters generated by running the service to be analyzed at multiple time points included in the preset time period.
[0106] In the embodiment of the present application, the operation parameter is the data actually generated during the operation of the service to be analyzed. For example, the operation data includes at least one of the following data: central processing unit (CPU) utilization rate, memory utilization rate, network delay, response time, throughput and error rate, etc. Among them, the CPU utilization rate is the proportion of the time used by the CPU to execute the service to be analyzed per unit time, which can reflect the busyness and load of the CPU. The memory utilization rate is the proportion of the memory space used by the service to be analyzed in the total memory space, which can reflect the occupancy of the service to be analyzed by the service to be analyzed, and the memory utilization rate can affect the speed and stability of the operation of the service to be analyzed. The network delay is the time required for data to be sent from other services to the service to be analyzed, or the time required for data to be sent from the service to be analyzed to other services. The response time is the time required for the service to be analyzed to process the received request and respond to the request. The throughput is the number of requests processed by the service to be analyzed per unit time. The error rate is the probability of an error in the request processed by the service to be analyzed.
[0107] After obtaining the operation data, the scheduling node can also perform data preprocessing on the obtained operation data. For example, since there may be some duplication between the operation parameters in the operation log and the operation parameters in the monitoring system, the data preprocessing can include deduplication of the operation parameters obtained from the log system and the monitoring system. For another example, data preprocessing includes: if there is a missing value in the operation parameter, fill in the missing position with zero, or delete the missing data item, or use the mean of the data before and after the missing position to fill the missing position. For another example, data preprocessing includes: denoising the operation parameters. In addition to data cleaning processes such as deduplication, missing processing and denoising, data preprocessing can also include normalization of the operation parameters to facilitate the subsequent determination of potential failures of the service to be analyzed.
[0108] Afterwards, the scheduling node extracts features from the operating data, and the performance data of the service to be analyzed includes: the proportion of CPU usage exceeding the CPU usage threshold, the proportion of memory usage exceeding the memory usage threshold, the proportion of high-latency requests, and the sliding average and differential values of operating parameters. Among them, the proportion of high-latency requests indicates the proportion of requests received by the service to be analyzed whose response time exceeds the timeout period. Extracting time series features such as the sliding average and differential values of operating parameters can reflect the temporal dependency of operating parameters, so as to obtain the changing trend and periodicity of operating parameters.
[0109] There are one or more nodes to be analyzed where the service to be analyzed is located, that is, there are one or more nodes to be analyzed running the service to be analyzed. When there are multiple nodes to be analyzed, the operating parameters generated by each node to be analyzed running the service to be analyzed within a preset time period can be obtained respectively, and then the operating parameters of each node to be analyzed can be extracted respectively, and the extracted feature data is used as the performance data of the service to be analyzed.
[0110] Through the above method, since the data volume of the operating parameters of the service to be analyzed is huge, including a large amount of redundant data and data that is meaningless to fault analysis, the embodiment of the present application performs feature extraction on the operating parameters of the service to be analyzed, and can extract feature data that is beneficial to fault analysis from a large number of operating parameters as performance parameters, thereby reducing the amount of data used in subsequent fault analysis and improving the efficiency of fault analysis.
[0111] In the embodiment of the present application, the above S102 uses the pre-trained fault prediction model based on the performance data to determine the potential target fault of the service to be analyzed, including the following steps:
[0112] Step 1: Input the performance data into the fault prediction model to obtain the probability of each fault existing in the service to be analyzed output by the fault prediction model.
[0113] After the performance data is input into the fault prediction model, the fault prediction model can output multiple probabilities, each probability being the probability of a fault in the service to be analyzed. For example, the output of the fault prediction model is: 0.1, 0.2, 0.1, 0.1, 0.5, which means that the probability of fault 1 is 0.1, the probability of fault 2 is 0.2, the probability of fault 3 is 0.1, the probability of fault 4 is 0.1, and the probability of fault 5 is 0.5.
[0114] The output result of the fault prediction model can be expressed as: P(Fault)=f(Feature_i), where P(Fault) represents the probability of each fault existing in the service to be analyzed, f() represents the fault prediction model, and Feature_i represents the performance parameter.
[0115] Step 2: From the probability of each failure, select the failure corresponding to the maximum probability as the potential target failure of the service to be analyzed.
[0116] For example, the output results of the fault prediction model are: 0.1, 0.2, 0.1, 0.1, 0.5, then fault 5 corresponding to 0.5 is selected as the potential target fault of the service to be analyzed.
[0117] Through the above method, the embodiment of the present application can use the fault prediction model to predict the potential target faults of the service to be analyzed, without manually performing fault analysis on the service to be analyzed, thereby reducing the reliance of fault analysis on manual work. Moreover, since manual analysis relies on human experience and knowledge, the stability of the fault analysis results is poor and the accuracy is low. The embodiment of the present application avoids the reliance on manual experience and knowledge, thus improving the stability and accuracy of the fault analysis results.
[0118] In the examples of this application, see Figure 2 The fault prediction model used in the above S102 can be trained in advance through the following steps:
[0119] S201: Obtain sample performance parameters and training labels of the sample performance parameters, wherein the training labels indicate potential sample failures of the sample services to which the sample performance parameters belong.
[0120] The method of obtaining the sample performance parameters may refer to the above-mentioned process of obtaining the performance parameters of the service to be analyzed in S101, which will not be described in detail here.
[0121] After obtaining the sample performance parameters, you can obtain the training labels manually marked for each sample performance parameter.
[0122] S202: Input the sample performance parameters into a preset machine learning model to obtain an output result of the machine learning model, wherein the output result indicates a potential failure predicted for the sample service.
[0123] Optionally, the machine learning model may be: Random forest, Support Vector Machine (SVM) or Neural Network (NN), etc.
[0124] S203: Determine the loss value based on the training labels of the sample performance parameters and the output results of the machine learning model.
[0125] The training labels of the sample performance parameters and the output results of the machine learning model can be substituted into the preset loss function to calculate the loss value. For example, the preset loss function can be: Mean Absolute Error (MAE) loss function, Mean Squared Error (MSE) loss function or Cross-Entropy Loss function, etc.
[0126] S204. Adjust the model parameters of the machine learning model based on the loss value, return to S201 to obtain sample performance parameters and training labels of the sample performance parameters, until the machine learning model converges, and use the current machine learning model as a fault prediction model.
[0127] Back propagation can be used to adjust the model parameters of the fault prediction model based on the gradient descent method, including adjusting the weights and biases of each network layer of the fault prediction model, and then return to S201 to obtain the next batch of sample performance parameters and their training labels, and perform the next round of iterative training.
[0128] Before adjusting the model parameters, it is also possible to determine whether the machine learning model has converged. If it has converged, the current machine learning model is used as a fault prediction model. If it has not converged, the model parameters of the machine learning model are adjusted based on the loss value. For example, when the current number of iterations reaches a preset number of times, it is determined that the machine learning model has converged, otherwise it is determined that the machine learning model has not converged. For another example, when the loss value calculated in the current round of iterations is less than the loss value threshold, it is determined that the machine learning model has converged, otherwise it is determined that the machine learning model has not converged. For another example, when the stability of the loss value calculated in the previous preset round of iterations is higher than the stability threshold, it is determined that the machine learning model has converged, otherwise it is determined that the machine learning model has not converged. Alternatively, it is also possible to determine whether the machine learning model has converged by other means, which is not specifically limited in the embodiments of the present application.
[0129] After the machine learning model converges, the test performance parameters can also be input into the machine learning model to obtain the output results of the machine learning model, and the loss value is calculated based on the output results and the training labels of the test performance parameters. If the loss value is greater than the preset threshold, return to step one to continue training; if the loss value is less than or equal to the preset threshold, the current machine learning model is used as a fault prediction model. Among them, the method for obtaining the test performance parameters and their training labels can refer to the above-mentioned S201 method for obtaining sample performance parameters and training labels of sample performance parameters, which will not be repeated here.
[0130] Through the above method, the embodiment of the present application can pre-train a fault prediction model that can predict faults based on performance parameters. Moreover, during the training process, the output results of the fault prediction model can be closer and closer to the training labels, making the output results of the fault prediction model more and more accurate during the training process, thereby ensuring the accuracy of subsequent fault predictions.
[0131] In an embodiment of the present application, the above-mentioned S103 generates a fault injection strategy based on the target fault, which can be implemented as follows: obtaining risk information related to the target fault of the service to be analyzed, and then inputting the target fault, risk information and preset policy constraints into a preset policy generation model to obtain the fault injection strategy output by the policy generation model.
[0132] Among them, the risk information related to the target failure of the service to be analyzed may include at least one of the following: the number, location and proportion of high-risk nodes, the service name of the service to be analyzed and the duration of the risk, etc. Among them, the high-risk node is: a node that meets the preset high-risk conditions among the nodes to be analyzed. For example, the preset high-risk conditions include at least one of the following: the CPU usage exceeds the CPU usage threshold for a duration exceeding the preset duration, the memory usage exceeds the memory usage threshold for a duration exceeding the preset duration, and the network delay exceeds the preset delay, etc. The duration of the risk is: the duration that the node to be analyzed meets the above high-risk conditions.
[0133] The preset restriction conditions include multiple aspects. For example, there are restriction conditions on fault type, injection intensity, injection range, injection duration, injection frequency, recovery strategy, etc. Among them, the restriction condition on fault type is the multiple fault types that can be selected for injection, such as CPU current limiting, memory leakage, network delay, service restart, etc. Injection intensity indicates the degree of restriction on CPU, memory and network delay, for example, the restriction condition on injection intensity is: limit the CPU usage range to 0-50%, and increase the network delay range to 20-100 milliseconds. The restriction condition on injection range is the service and node into which the fault can be injected. The injection duration indicates the duration range of each fault injection, for example, the restriction condition on injection duration is 5-10 minutes. The injection frequency indicates the number of injections that can be selected per unit time, for example, the restriction condition on injection frequency is that 3-5 faults can be injected per day. The restriction condition on recovery strategy is the automatic recovery strategy that can be selected after fault injection, for example, automatic restart, dynamic adjustment of resources allocated to each service or node, etc.
[0134] The fault injection strategy can be recorded as Param_init=g(Fault Report, Rules), where Param_init represents the fault injection strategy, g() represents the strategy generation model, Fault Report represents the target fault and risk information, and Rules represents the preset strategy constraints.
[0135] The above strategy generation model can be trained through sample fault information and fault labels. The sample fault information includes potential faults and risk information of the sample system, and the fault label is a fault injection strategy that is manually executed on the sample system based on the needs of sample fault information analysis. The specific training method of the strategy generation model can refer to the training method of the above fault prediction model, which will not be repeated here.
[0136] Through the above method, the embodiment of the present application can use the policy generation model to automatically generate a fault injection strategy based on the potential target faults, risk information and preset policy constraints of the service to be analyzed. The generation process of the fault injection strategy does not need to rely on manual work, thereby simplifying user operations and improving the efficiency, stability and accuracy of generating fault injection strategies. Moreover, compared to directly shutting down some physical devices where the service to be analyzed is located, the embodiment of the present application can control parameters such as the type, intensity, duration and frequency of fault injection, achieve more fine-grained fault injection and control, achieve more precise control of chaos engineering, and improve the effectiveness of the recovery strategy.
[0137] In the embodiment of the present application, after the fault injection strategy is executed on the node to be analyzed in S104, the scheduling node may also optimize the fault injection strategy, such as Figure 3 As shown, the optimization process includes the following steps:
[0138] S301: Obtain current values and historical values of preset parameter items of the service to be analyzed.
[0139] The preset parameter items can be set according to actual business needs. For example, if the business needs have high requirements on the response time of the service to be analyzed, the preset parameter item can be the response time.
[0140] S302: Use a preset optimization algorithm to modify the fault injection strategy based on current values, historical values, and preset optimization targets for preset parameter items.
[0141] The preset optimization goal may be set according to actual business requirements. For example, the preset optimization goal is to minimize the response time.
[0142] For example, a preset optimization algorithm such as Bayesian or genetic algorithm can be used to modify the fault injection strategy based on the current value, historical value and preset optimization target of the preset parameter item. Among them, the Bayesian optimization algorithm is an optimization algorithm that finds the maximum or minimum value of the preset parameter item. By continuously trying new fault injection strategies, the best fault injection strategy can be gradually adjusted, that is, after the adjusted fault injection strategy is injected into the node to be analyzed, it is closer to the situation where the target fault exists in the service to be analyzed in the real scenario. Genetic algorithm is an optimization algorithm that simulates natural selection and genetic mechanism. It finds the optimal fault injection strategy by simulating operations such as selection, crossover and mutation in the biological evolution process.
[0143] S303: Execute the modified fault injection strategy on the node to be analyzed.
[0144] pass Figure 3The method shown in the embodiment of the present application can modify the fault injection strategy, that is, optimize the fault injection strategy so that after the modified fault injection strategy is injected into the node to be analyzed, it is closer to the situation where the target fault exists in the service to be analyzed in the real scenario, thereby improving the accuracy of chaos engineering.
[0145] In the embodiment of the present application, the above S302 uses a preset optimization algorithm to modify the fault injection strategy based on the current value, the historical value and the preset optimization target of the preset parameter item, including the following steps:
[0146] Step 1: Based on the current value, the fault injection strategy corresponding to the current value, the historical value, and the historical fault injection strategy corresponding to the historical value, a target performance prediction model is trained.
[0147] The current value can be used as the training label of the fault injection strategy corresponding to the current value, and the historical value can be used as the training label of the historical fault injection strategy corresponding to the historical value. Then, based on each fault injection strategy and its training label, the preset machine learning model is trained to obtain the trained target performance prediction model. The training method of the target performance prediction model can refer to the training method of the above-mentioned fault prediction model, which will not be repeated here.
[0148] Step 2: Determine predicted values of preset parameter items corresponding to multiple candidate fault injection strategies based on the target performance prediction model.
[0149] The fault injection strategy includes multiple parameter items, and each parameter item can have multiple values. For example, the fault injection strategy includes: fault type, injection intensity, injection range, injection duration, injection frequency and recovery strategy, etc. The value of the fault type can be CPU current limiting, memory leakage, network delay and / or service restart, and the value range of the injection intensity is [0,50%]. The combination of each parameter item value of the fault injection strategy can be used as a candidate fault injection strategy; or only the combination of various parameter item values of the fault injection strategy, each parameter item value combination that has not been selected can be used as a candidate fault injection strategy. Alternatively, the fault injection strategy can be controlled to include multiple parameter items, some parameter item values are fixed, and some parameter item values can be modified, such as setting that only the injection intensity and injection time in the fault injection strategy can be modified. At this time, each combination of the parameter item values that can be modified in the fault injection strategy can be combined with the non-modifiable parameter item value combination as a candidate fault injection strategy.
[0150] Alternatively, the candidate fault injection strategy may be determined by other means, which is not specifically limited in the embodiment of the present application.
[0151] Afterwards, the scheduling node may input each candidate fault injection strategy into the target performance prediction model respectively, and obtain the predicted value of the preset parameter item corresponding to the candidate fault injection strategy output by the target performance prediction model.
[0152] Step 3: Based on the predicted value corresponding to each candidate fault injection strategy and the preset optimization target, select a candidate fault injection strategy as the modified fault injection strategy.
[0153] In one implementation, the prediction value corresponding to each candidate fault injection strategy can be substituted into a preset acquisition function to obtain an acquisition function value, and then the candidate fault injection strategy with the largest acquisition function value is selected as the modified fault injection strategy. The acquisition function value is positively correlated with the correlation between the candidate fault injection strategy and the preset optimization target, that is, the larger the acquisition function value, the closer the candidate fault injection strategy is to the preset optimization target.
[0154] In the embodiment of the present application, after the target performance prediction model determines the predicted values of the preset parameter items corresponding to the multiple candidate fault injection strategies, the predicted mean mu and the predicted standard deviation sigma are obtained. Then, the target historical value that best meets the optimization goal is obtained, such as obtaining the minimum value of the response time. If the predicted value corresponding to the candidate fault injection strategy is more in line with the optimization goal than the target historical value, the immediate improvement imp corresponding to the predicted value can be calculated as follows: mu-mu_sample_opt-xi, where mu_sample_opt is the target historical value and xi is a preset hyperparameter, such as xi=0.01. Then, the standardized immediate improvement Z=imp / sigma is calculated. Then, the expected improvement is calculated using the Cumulative Distribution Function (CDF) and Probability Density Functions (PDF) of the Gaussian distribution, and ei = imp*norm.cdf(Z) + sigma*norm.pdf(Z), where ei represents the expected improvement value corresponding to each candidate fault injection strategy, norm.cdf(Z) represents the probability below the Z value, and sigma*norm.pdf(Z) represents the additional improvement that may be obtained due to uncertainty. Then, the candidate fault injection strategy with the maximum ei is determined by minimizing the fault loss function. At this time, the modified fault injection strategy can be recorded as: Param_opt = argmin_Param h(Param, Real-timeData, Historical Data), where Param_opt is the modified fault injection strategy, argmin_Param represents the value of Param when the fault loss function is minimized, h() represents the preset fault loss function, Param represents each candidate fault injection strategy, Real-time Data represents the current value of the preset parameter item, and Historical Data represents the historical value of the preset parameter item.
[0155] Among them, the fault loss function can be designed in combination with the preset optimization goal, accuracy and coverage. Among them, accuracy represents the prediction accuracy of the target performance prediction model, and coverage represents the breadth of fault types covered by fault injection. After minimizing the fault loss function, a fault injection strategy with higher accuracy and wider coverage can be obtained. Higher coverage can make more fault types used during fault injection, making the test of the service to be analyzed more comprehensive, so as to discover various weaknesses of the service to be analyzed, so that the service to be analyzed and the service system where the service to be analyzed is located can be optimized more targeted in the future, ensuring the robustness of the service to be analyzed and the service system.
[0156] For example, the preset fault loss function can be recorded as: def loss_function(param,real_time_data,historical_data). Among them, the accuracy is recorded as: accuracy_score=calculate_accuracy(param,real_time_data), calculate_accuracy represents the accuracy calculation formula, which can be set according to actual needs. The coverage is recorded as: coverage_score=calculate_coverage(param,historical_data), calculate_coverage represents the coverage calculation formula, which can be set according to actual needs. Taking the fault injection strategy including injection intensity and injection time as an example, accuracy=-abs(50-cpu_limit), coverage=duration, where abs() represents the calculation of absolute value, cpu_limit represents the injection intensity for the CPU, and duration represents the injection time. The loss function value is recorded as: return(alpha*accuracy_score+beta*coverage_score), where alpha and beta are preset hyperparameters. For example, alpha and beta are both 0.5. The closer return is to 0, the closer the fault injection strategy is to the optimization target, and the higher the accuracy and coverage.
[0157] When designing the fault loss function, the impact of fault injection can also be considered, including the injection range and injection duration, so that when minimizing the fault loss function, the number of nodes affected by the fault injection and the duration of the fault injection can be reduced to avoid the problem of large-scale node crashes, thereby reducing the impact of fault injection on the actual business execution of the analysis service, reducing the impact on users, and ensuring user experience.
[0158] Through the above method, the embodiment of the present application can select a candidate fault injection strategy that better meets the preset optimization goals as a modified fault injection strategy based on the current values of the preset parameter items of the service to be analyzed and their fault injection strategies, as well as the historical values and their historical fault injection strategies, thereby improving the accuracy of the fault injection strategy.
[0159] In the embodiment of the present application, the scheduling node can monitor the operation of the service to be analyzed and continuously adjust the fault injection strategy, thereby continuously optimizing the fault injection strategy until the fault injection strategy converges. That is, after the scheduling node executes the modified fault injection strategy on the node to be analyzed in S303, it can also determine whether the fault injection strategy converges based on the current value and / or the number of modifications of the fault injection strategy.
[0160] For example, it may be determined whether the number of modifications of the fault injection strategy reaches a preset number of modifications. If so, it is determined that the fault injection strategy has converged; otherwise, it is determined that the fault injection strategy has not converged.
[0161] For another example, it may be determined whether the current value of the preset parameter item of the service to be analyzed is less than the preset value. If so, it is determined that the fault injection strategy has converged; otherwise, it is determined that the fault injection strategy has not converged.
[0162] For another example, it can be determined whether the number of modifications of the fault injection strategy reaches the preset number of modifications, and whether the current value of the preset parameter item of the service to be analyzed is less than the preset value. If the judgment results are both yes, it is determined that the fault injection strategy has converged; if any judgment result is no, it is determined that the fault injection strategy has not converged.
[0163] If the fault injection strategy converges, the experiment is considered complete and the fault injection strategy will not be modified further.
[0164] If the fault injection strategy has not converged, the process returns to the step of obtaining the current value and the historical value of the preset parameter item of the service to be analyzed in S301 to continue modifying the fault injection strategy.
[0165] Through the above method, the embodiment of the present application can continue to optimize the fault injection strategy when the fault injection strategy has not converged, thereby improving the accuracy of the fault injection strategy. On the contrary, when the fault injection strategy converges, the benefit of optimizing the fault injection strategy at this time is small, that is, it is difficult to improve the accuracy of the fault injection strategy to a greater extent, so the fault injection strategy may not be optimized to save computing resources.
[0166] See also Figure 4 The following is an example of the overall process of the fault testing method provided in the embodiment of the present application, combined with actual application scenarios:
[0167] The scheduling node includes: cluster management module, data analysis module and intelligent scheduling module.
[0168] The cluster management module pre-registers multiple service nodes in the service grid. Each service node runs at least one service. Subsequently, based on the registration information of the service node, the operating parameters generated by the running services of each service node can be collected in real time. The cluster management module can also determine whether the service node has a fault based on the real-time operating parameters of the service node, and record the attribute data of the faulty service node when the service node fails.
[0169] The data analysis module pre-trains the fault prediction model, and determines the potential target faults of the service to be analyzed based on the real-time performance data of the service to be analyzed through the fault prediction model, and determines the risk information related to the target faults of the service to be analyzed.
[0170] The intelligent scheduling module inputs the target fault, risk information and preset policy constraints into the preset policy generation model to obtain the fault injection policy generated by the policy generation model. And calls the fault injection interface provided by the cluster management module to send the fault injection policy to the cluster management module, so that the cluster management module executes the fault injection policy on the node to be analyzed running the service to be analyzed. The intelligent scheduling module also obtains the current value and historical value of the preset parameter item of the service to be analyzed, and uses the preset optimization algorithm to modify the fault injection policy based on the current value, historical value and preset optimization target of the preset parameter item, and then calls the fault injection interface provided by the cluster management module to send the modified fault injection policy to the cluster management module, so that the cluster management module executes the modified fault injection policy on the node to be analyzed.
[0171] It can be seen that the embodiments of the present application can simplify manual operations in chaos engineering, reduce dependence on manual operations, and improve the efficiency of chaos engineering. Moreover, the embodiments of the present application can conduct experiments on the situation where the target failure of the service to be analyzed occurs in advance before the target failure of the service to be analyzed occurs, so as to optimize the service to be analyzed or the service system where the service to be analyzed is located in advance, reduce the interruption time when the target failure of the service to be analyzed occurs, and reduce the failure risk and operation and maintenance cost in actual operation and maintenance.
[0172] Based on the same inventive concept, corresponding to the above method embodiment, the present application embodiment also provides a fault testing device, such as Figure 5 As shown, the device includes: an acquisition module 501, a determination module 502, a generation module 503 and an execution module 504;
[0173] An acquisition module 501 is used to acquire performance data of a service to be analyzed;
[0174] A determination module 502 is used to determine potential target faults of the service to be analyzed based on the performance data obtained by the acquisition module 501 using a pre-trained fault prediction model;
[0175] A generating module 503 is used to generate a fault injection strategy based on the target fault determined by the determining module 502, where the fault injection strategy is a strategy for controlling the target fault to occur in the service to be analyzed;
[0176] The execution module 504 is used to execute the fault injection strategy generated by the generation module 503 on the node to be analyzed that runs the service to be analyzed.
[0177] Optionally, the acquisition module 501 is specifically used for:
[0178] Obtaining the operating parameters generated by the node to be analyzed running the service to be analyzed within a preset time period;
[0179] Extract features from the operating parameters to obtain performance data of the service to be analyzed.
[0180] Optionally, the determination module 502 is specifically configured to:
[0181] Input the performance data into the fault prediction model to obtain the probability of each fault existing in the service to be analyzed output by the fault prediction model;
[0182] From the probability of each failure, select the failure corresponding to the maximum probability as the potential target failure of the service to be analyzed.
[0183] Optionally, the device further comprises a training module, wherein the training module is used for:
[0184] Obtaining a sample performance parameter and a training label of the sample performance parameter, where the training label indicates a potential sample failure of a sample service to which the sample performance parameter belongs;
[0185] Input the sample performance parameters into a preset machine learning model to obtain an output result of the machine learning model, where the output result represents a potential failure predicted for the sample service;
[0186] Determine the loss value based on the training labels and output results;
[0187] The model parameters of the machine learning model are adjusted based on the loss value, and the step of obtaining the sample performance parameters and the training labels of the sample performance parameters is returned until the machine learning model converges, and the current machine learning model is used as a fault prediction model.
[0188] Optionally, the generating module 503 is specifically used for:
[0189] Obtain risk information related to target failures of the service to be analyzed;
[0190] The target fault, risk information and preset policy constraints are input into the preset policy generation model to obtain the fault injection policy output by the policy generation model.
[0191] Optionally, the device further comprises:
[0192] The acquisition module 501 is further used to acquire the current value and historical value of the preset parameter item of the service to be analyzed after executing the fault injection strategy on the node to be analyzed running the service to be analyzed;
[0193] A modification module, used to modify the fault injection strategy based on current values, historical values and preset optimization targets for preset parameter items using a preset optimization algorithm;
[0194] The execution module 504 is further configured to execute the modified fault injection strategy on the node to be analyzed.
[0195] Optionally, modify the module to:
[0196] Training a target performance prediction model based on the current value, the fault injection strategy corresponding to the current value, the historical value, and the historical fault injection strategy corresponding to the historical value;
[0197] Determine predicted values of preset parameter items corresponding to a plurality of candidate fault injection strategies based on the target performance prediction model;
[0198] Based on the predicted value corresponding to each candidate fault injection strategy and the preset optimization target, a candidate fault injection strategy is selected as the modified fault injection strategy.
[0199] Optionally, the device further comprises:
[0200] A judgment module, used for determining whether the fault injection strategy has converged based on the current value and / or the number of times the fault injection strategy has been modified after executing the modified fault injection strategy on the node to be analyzed;
[0201] The calling module is used to call the obtaining module 501 to execute the step of obtaining the current value and the historical value of the preset parameter item of the service to be analyzed if the judgment result is non-convergence.
[0202] The present application also provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.
[0203] Memory 603, used for storing computer programs;
[0204] The processor 601 is used to implement the method executed by the scheduling node in the method embodiment when executing the program stored in the memory 603.
[0205] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0206] The communication interface is used for communication between the above electronic device and other devices.
[0207] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0208] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0209] In another embodiment provided in the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the fault testing method described in any one of the above embodiments is implemented.
[0210] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the fault testing method described in any one of the above embodiments.
[0211] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0212] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0213] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0214] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A fault testing method, characterized in that: The method comprises: Get the performance data of the service to be analyzed; Determine potential target faults of the service to be analyzed based on the performance data using a pre-trained fault prediction model; Based on the target fault, generating a fault injection strategy, wherein the fault injection strategy is a strategy for controlling the target fault to occur in the service to be analyzed; The fault injection strategy is executed on the node to be analyzed that runs the service to be analyzed.
2. The method according to claim 1, characterized in that The obtaining of the performance data of the service to be analyzed includes: Obtaining the operating parameters generated by the node to be analyzed running the service to be analyzed within a preset time period; Feature extraction is performed on the operating parameters to obtain performance data of the service to be analyzed.
3. The method according to claim 1, characterized in that The using a pre-trained fault prediction model to determine a potential target fault of the service to be analyzed based on the performance data includes: Inputting the performance data into the fault prediction model to obtain the probability of each fault existing in the service to be analyzed output by the fault prediction model; From the probabilities of each failure, a failure corresponding to the maximum probability is selected as a potential target failure of the service to be analyzed.
4. The method according to any one of claims 1 to 3, characterized in that: The fault prediction model is trained by the following steps: Acquire a sample performance parameter and a training label of the sample performance parameter, wherein the training label indicates a potential sample failure of a sample service to which the sample performance parameter belongs; Inputting the sample performance parameter into a preset machine learning model to obtain an output result of the machine learning model, wherein the output result represents a potential failure predicted for the sample service; Determine a loss value based on the training label and the output result; Adjust the model parameters of the machine learning model based on the loss value, return to the step of obtaining sample performance parameters and training labels of the sample performance parameters, and use the current machine learning model as the fault prediction model until the machine learning model converges.
5. The method according to claim 1, characterized in that The generating a fault injection strategy based on the target fault includes: Obtain risk information related to a target failure of the service to be analyzed; The target fault, the risk information and the preset policy restriction condition are input into a preset policy generation model to obtain the fault injection policy output by the policy generation model.
6. The method according to claim 1, characterized in that After executing the fault injection strategy on the node to be analyzed running the service to be analyzed, the method further includes: Obtaining current values and historical values of preset parameter items of the service to be analyzed; Using a preset optimization algorithm, based on the current value, the historical value and a preset optimization target for the preset parameter item, modify the fault injection strategy; The modified fault injection strategy is executed on the node to be analyzed.
7. The method according to claim 6, characterized in that The using a preset optimization algorithm to modify the fault injection strategy based on the current value, the historical value and a preset optimization target for the preset parameter item includes: Training a target performance prediction model based on the current value, the fault injection strategy corresponding to the current value, the historical value, and the historical fault injection strategy corresponding to the historical value; Determining predicted values of preset parameter items corresponding to a plurality of candidate fault injection strategies based on the target performance prediction model; Based on the predicted value corresponding to each candidate fault injection strategy and the preset optimization target, a candidate fault injection strategy is selected as the modified fault injection strategy.
8. The method according to claim 6 or 7, characterized in that: After executing the modified fault injection strategy on the node to be analyzed, the method further includes: Determining whether the fault injection strategy has converged based on the current value and / or the number of modifications of the fault injection strategy; If not, return to the step of obtaining the current value and the historical value of the preset parameter item of the service to be analyzed.
9. A fault testing device, characterized in that: The device comprises: An acquisition module is used to obtain the performance data of the service to be analyzed; A determination module, configured to determine a potential target fault of the service to be analyzed based on the performance data acquired by the acquisition module using a pre-trained fault prediction model; A generating module, configured to generate a fault injection strategy based on the target fault determined by the determining module, wherein the fault injection strategy is a strategy for controlling the target fault to occur in the service to be analyzed; An execution module is used to execute the fault injection strategy generated by the generation module on the node to be analyzed that runs the service to be analyzed.
10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 8 when executing a program stored in a memory.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Active testing method, device and equipment of AI computing power cluster, medium and product
CN119415221A
Micro-service operation control method and device, storage medium and electronic equipment
CN119829363A
