Chaos engineering fault drill scene generation method and device, equipment and storage medium

Abstract

The disclosure provides a chaos engineering fault rehearsal scene generation method and device, equipment and storage medium, which can be applied to the technical field of machine learning and chaos engineering. The method comprises: acquiring a plurality of component data on a measured service link of a measured system, and extracting a plurality of preset dimension fault scene sample sets from the plurality of component data; the fault scene sample sets are respectively applied to the measured system for experiments, the measured system is preset with a plurality of influence indexes for the measured service, and the influence total probability of each fault scene sample on the plurality of influence indexes is calculated based on the experimental results of the plurality of influence indexes; according to the influence total probability of each fault scene sample on the plurality of influence indexes and the preset probability standard value, the fault scene sample set is classified into a positive sample set and a negative sample set; the positive sample set and the negative sample set are input into a pre-constructed automatic arrangement model for training, and a trained automatic arrangement model is obtained, which is used to generate an effective fault rehearsal scene.

Description

Technical Field

[0001] This disclosure relates to the fields of machine learning and chaos engineering, and in particular to a method, apparatus, device, medium, and program product for generating chaos engineering fault simulation scenarios. Background Technology

[0002] Machine learning is the science of studying how to use computers to simulate or implement human learning activities, and the study of automatically improving computer algorithms by utilizing data or past experience.

[0003] Chaos engineering originated from Chaos Monkey, created by Netflix in 2010. By randomly injecting different types of faults into a system, it aims to identify as many fault-prone links as possible, thereby enabling targeted hardening and protection of the system.

[0004] Fault drill scenarios refer to complex fault scenarios that simulate a series of fault events that may occur in a production environment. Fault drill scenarios can be used to inject faults into the system to test its response, thereby improving the system's fault tolerance.

[0005] Chaos engineering is a concept and approach to ensure system robustness and improve system stability. As a practical application of chaos engineering and based on the analysis of historical production failures, real-world failure scenarios are often highly complex and unpredictable. Therefore, the ability to flexibly support the orchestration of various complex failure simulation scenarios is an extremely important aspect. Failure simulation scenarios are constructed by orchestrating a series of individual failure steps within a specific scenario. Currently, failure scenario orchestration is primarily done manually, which is extremely inefficient, error-prone, and highly dependent on individual experience and understanding. This inevitably leads to insufficient coverage of scenarios beyond one's own knowledge, resulting in inadequate simulation effectiveness. Summary of the Invention

[0006] In view of the above problems, this disclosure provides a method, apparatus, equipment, medium and program product for generating chaotic engineering fault simulation scenarios.

[0007] According to the first aspect of this disclosure, a method for generating chaotic engineering fault simulation scenarios is provided, comprising: acquiring multiple component data on the service link under test of the system under test; extracting multiple preset dimensions of fault scenario sample sets from the multiple component data; applying the fault scenario sample sets to the system under test for experiments, wherein the system under test has preset multiple influence indicators for the service under test; calculating the total probability of each fault scenario sample's influence on the multiple influence indicators based on the experimental results of the multiple influence indicators; classifying the fault scenario sample sets into positive sample sets and negative sample sets according to the total probability of each fault scenario sample's influence on the multiple influence indicators and preset probability standard values; and inputting the positive sample sets and negative sample sets into a pre-constructed automatic orchestration model for training to obtain a trained automatic orchestration model for generating effective fault simulation scenarios.

[0008] According to an embodiment of this disclosure, obtaining multiple component data on the service link under test of the system under test includes: obtaining multiple services and components that pass through the service link under test of the system under test, splitting and marking the multiple services and components to obtain the data number of each component and using it as component data.

[0009] According to embodiments of this disclosure, the fault scenario sample set includes a historical scenario sample set, a random scenario sample set, and a manual scenario sample set. Extracting fault scenario sample sets with multiple preset dimensions from the multiple component data includes: obtaining multiple historical fault scenarios from the multiple component data; arranging and simulating the multiple historical fault scenarios to construct a historical scenario sample set; generating a random scenario sample set using a random algorithm; and generating a manual scenario sample set using a manual arrangement method.

[0010] According to embodiments of this disclosure, the plurality of impact metrics include business metrics and resource metrics, the business metrics including TPS and response latency, and the resource metrics including CPU, memory, disk I / O and network.

[0011] According to embodiments of this disclosure, the total probability of the impact of each fault scenario sample on the plurality of impact indicators is calculated using the following formula:

[0012]

[0013] In the formula, n is the total number of influencing indicators; P j，λ P represents the probability of the j-th fault scenario sample influencing the λ-th influencing index. j Let be the total probability of the impact of the j-th fault scenario sample on multiple influencing indicators.

[0014] According to embodiments of this disclosure, the probability of the j-th fault scenario sample affecting the λ-th influence index is calculated using the following formula:

[0015]

[0016] In the formula, P j，λ (0) is the expected value of the indicator, which is obtained manually based on the daily testing experience of the system under test. P j，λ (1) is the actual value of the index, which is taken from the experimental results of the j-th fault scenario sample acting on the tested system.

[0017] According to embodiments of this disclosure, the fault scenario sample set is classified into a positive sample set and a negative sample set based on the total probability of the influence of each fault scenario sample on the multiple influence indicators and a preset probability standard value. This includes: for any fault scenario sample in the fault scenario sample set, if the total probability of the influence of the fault scenario sample on the multiple influence indicators is greater than the preset probability standard value, the fault scenario sample is marked as a positive sample; otherwise, the fault scenario sample is marked as a negative sample.

[0018] According to embodiments of this disclosure, the automatic orchestration model is pre-constructed as follows: A minimum-cost maximum-flow algorithm is used to map the constraints and optimization objectives related to the orchestration methods, thereby constructing the automatic orchestration model; wherein the constraints related to the orchestration methods include minimum-cost constraints and maximum-flow constraints, the minimum-cost constraints being the set of the smallest possible orchestration methods, and the maximum-flow constraints being the fault scenario samples with the highest overall probability of impact; the optimization objective is to find the target scenario sample set that has the highest overall probability of impact on the tested system under the condition of minimizing the space of orchestration method types.

[0019] According to an embodiment of this disclosure, the minimum cost maximum flow algorithm specifically includes the following steps: assigning corresponding weights to each component data based on its position and role in the system under test; selecting a fault scenario sample corresponding to any component data as an initial fault step, calculating the product of the total probability of the fault scenario sample's influence on the multiple influence indicators and the weight of the component data, and determining the product as the fault probability of the initial fault step on the orchestration link; repeating the above fault probability calculation steps, sequentially selecting other fault scenario samples in the fault scenario sample set, calculating the corresponding fault probabilities, and continuously adding them to the orchestration link until the accumulated fault probability on the orchestration link reaches a preset fault probability threshold; and determining the multiple fault scenario samples selected on the orchestration link as the target scenario sample set.

[0020] According to embodiments of this disclosure, the positive and negative sample sets are input into a pre-constructed automatic orchestration model for training, and the training process of the automatic orchestration model is further described as follows: the training process of the automatic orchestration model undergoes multiple rounds of iterative optimization, and in each round of iterative optimization, the negative sample set is sampled according to a preset ratio to adjust the input ratio of the positive and negative sample sets.

[0021] According to embodiments of this disclosure, the method further includes: loading the trained automatic orchestration model into the target system, inputting various parameters of the system under test, and generating an effective fault simulation scenario corresponding to the target system.

[0022] The second aspect of this disclosure provides a chaotic engineering fault simulation scenario generation device, comprising: a sample extraction module, used to acquire multiple component data on the tested service link of the system under test, and extract fault scenario sample sets of multiple preset dimensions from the multiple component data; a probability calculation module, used to apply the fault scenario sample sets to the tested system for experiments, wherein the tested system has preset multiple influence indicators for the tested service, and calculates the total probability of each fault scenario sample's influence on the multiple influence indicators based on the experimental results of the multiple influence indicators; a sample classification module, used to classify the fault scenario sample set into a positive sample set and a negative sample set according to the total probability of each fault scenario sample's influence on the multiple influence indicators and a preset probability standard value; and a model training module, used to input the positive sample set and the negative sample set into a pre-constructed automatic orchestration model for training, to obtain a trained automatic orchestration model for generating effective fault simulation scenarios.

[0023] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0024] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0025] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0026] Based on the chaotic engineering fault simulation scenario generation method, apparatus, equipment, medium, and program products provided in this disclosure, through research on the constituent elements of complex fault scenarios, historical fault data, and machine learning, fault simulation scenarios can be automatically arranged. Simultaneously, based on historical experience, analysis and continuous optimization learning can be performed to construct more effective and high-quality fault simulation scenarios. This disclosure is more flexible and convenient, improves the effectiveness of the arrangement scheme, and reduces the complexity and limitations of manual arrangement. Attached Figure Description

[0027] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0028] Figure 1 The illustration schematically depicts application scenarios of the chaos engineering fault simulation scenario generation method, apparatus, device, medium, and program product according to embodiments of the present disclosure;

[0029] Figure 2 A flowchart illustrating a method for generating chaos engineering fault simulation scenarios according to an embodiment of the present disclosure is shown schematically.

[0030] Figure 3 This schematically illustrates a detailed operation flowchart of a method for generating chaotic engineering fault simulation scenarios according to an embodiment of the present disclosure;

[0031] Figure 4 A flowchart illustrating the process of extracting a sample set of fault scenarios according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 5 A flowchart illustrating the construction of an automatic orchestration model using the minimum cost maximum flow algorithm according to an embodiment of the present disclosure is shown.

[0033] Figure 6 This schematic diagram illustrates the structural block diagram of a chaos engineering fault simulation scenario generation device according to an embodiment of the present disclosure;

[0034] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a method for generating chaotic engineering fault simulation scenarios according to an embodiment of the present disclosure. Detailed Implementation

[0035] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0036] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0037] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0038] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0039] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0040] Research revealed that machine learning technologies are utilized in areas such as intelligent detection, localization, and diagnosis of hardware and software faults. However, these technologies are highly dependent on the specific fault domain and its requirements, and cannot be directly applied to fault simulation and orchestration in chaos engineering.

[0041] Analysis of fault drill scenarios reveals that these scenarios consist of a series of simple fault steps. The arrangement method is mainly based on expert guidance and experience, real historical fault events in the production environment, and the knowledge and understanding of testers. This method has certain limitations, and manual arrangement is difficult and inefficient.

[0042] In view of this, the embodiments of this disclosure provide a method for generating chaotic engineering fault simulation scenarios based on historical data, manual arrangement, monitoring indicators, etc., which is more flexible and convenient, improves the effectiveness of the arrangement scheme, and reduces the complexity and limitations of manual arrangement.

[0043] Figure 1 The illustration schematically depicts an application scenario of the chaos engineering fault simulation scenario generation method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0044] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0045] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0046] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0047] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0048] It should be noted that the chaotic engineering fault simulation scenario generation method provided in this embodiment can generally be executed by server 105. Correspondingly, the chaotic engineering fault simulation scenario generation device provided in this embodiment can generally be located in server 105. The chaotic engineering fault simulation scenario generation method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the chaotic engineering fault simulation scenario generation device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0049] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0050] The following will be based on Figure 1 The described scene, through Figures 2-5 The method for generating chaotic engineering fault simulation scenarios according to the disclosed embodiments is described in detail.

[0051] Figure 2 A flowchart illustrating a method for generating chaotic engineering fault simulation scenarios according to an embodiment of the present disclosure is shown. Figure 3 The diagram illustrates the specific operation flowchart of the method for generating chaotic engineering fault simulation scenarios according to an embodiment of the present disclosure.

[0052] Please see Figure 2 and Figure 3 The chaotic engineering fault simulation scenario generation method in this embodiment includes operations S210 to S240.

[0053] When operating S210, multiple component data are acquired on the service link of the system under test, and multiple preset dimensions of fault scenario sample sets are extracted from the multiple component data.

[0054] In this embodiment, multiple component data are obtained on the service link under test of the system under test, including: obtaining multiple services and components that pass through the service link under test of the system under test, splitting and marking the multiple services and components, obtaining the data number of each component and using it as component data.

[0055] Specifically, in the system under test, the traffic flow from the ingress to the egress of the tested service inevitably passes through multiple services and components. These services and components are then broken down and labeled to obtain the data IDs of each component. These component IDs are used as component data to obtain multiple component data sets. For example, the data IDs of each component are labeled as variable i, and the set of component data is labeled I. This data labeling provides a necessary prerequisite for extracting a sample set of fault scenarios.

[0056] Next, the data related to these components across various dimensions are integrated and summarized to form a fault sample set for each component, which serves as the source data for the automatic orchestration model. In this embodiment, the fault scenario sample set includes a historical scenario sample set, a random scenario sample set, and a manually generated scenario sample set. Using data from these three preset dimensions—historical, random, and manually generated scenarios—as the data source, sample sets for different scenarios are comprehensively abstracted.

[0057] Figure 4 A flowchart illustrating the process of extracting a sample set of fault scenarios according to an embodiment of the present disclosure is shown.

[0058] Specifically, such as Figure 4 As shown, a fault scenario sample set with multiple preset dimensions is extracted from multiple component data, including operations S2101 to S2103.

[0059] In operation S2101, multiple historical fault scenarios are obtained from multiple component data, and the multiple historical fault scenarios are arranged and simulated to construct a historical scenario sample set.

[0060] In operation S2102, a random scene sample set is generated using a random algorithm.

[0061] In operation S2103, a set of artificial scene samples is generated using a manual arrangement method.

[0062] It should be noted that there is no strict order between operations S2101 to S2103; they can be performed in parallel. A realistic sample set of fault scenarios is constructed by simulating a large number of historical fault scenarios. The test sample set of fault scenarios is constructed using two methods: automatically simulating fault scenarios using a random algorithm and manually simulating test scenarios.

[0063] Through the above embodiments, this disclosure analyzes and summarizes historical production events, random scenarios generated by random algorithms, and manually arranged scenarios, and precipitates a sample set for each type of failure scenario. For example, each failure scenario sample is labeled j, and the precipitated scenario sample set corresponding to each component i is denoted as Ji, where J is composed of j.

[0064] In operation S220, the fault scenario sample set is applied to the system under test for experimentation. The system under test has multiple impact indicators preset for the service under test. Based on the experimental results of the multiple impact indicators, the total probability of the impact of each fault scenario sample on the multiple impact indicators is calculated.

[0065] This operation will conduct specific experiments on the tested service using the three scenario sample sets constituted by the above operation S210, and observe the probability of their impact on various influencing indicators.

[0066] In one alternative implementation, multiple impact metrics include business metrics and resource metrics. Business metrics include TPS (Transactions Per Second) and response latency, while resource metrics include CPU, memory, disk I / O, and network. The overall probability of impact on the service under test is then assessed based on these impact metrics, and the data is recorded.

[0067] In this embodiment, the total probability of the impact of each fault scenario sample on multiple impact indicators is calculated using the following formula:

[0068]

[0069] In the formula, n is the total number of influencing indicators; P j，λ P represents the probability of the j-th fault scenario sample influencing the λ-th influencing index. j Let be the total probability of the impact of the j-th fault scenario sample on multiple influencing indicators.

[0070] Specifically, the probability of the j-th fault scenario sample affecting the λ-th influence index is calculated using the following formula:

[0071]

[0072] In the formula, P j，λ (0) is the expected value of the indicator, which is obtained manually based on the daily testing experience of the system under test. P j，λ (1) is the actual value of the index, which is taken from the experimental results of the j-th fault scenario sample acting on the tested system.

[0073] Through the above embodiments, this operation calculates the sum of the probabilities of the failure scenario sample's impact on various impact indicators of the tested system, and obtains the total probability of the sample's impact on all impact indicators of the tested service. This total probability of impact is used for subsequent classification of the sample set.

[0074] In operation S230, based on the total probability of each fault scenario sample's influence on multiple influencing indicators and the preset probability standard value, the fault scenario sample set is classified into a positive sample set and a negative sample set.

[0075] Optionally, the probability standard value is set based on experience and test standards to determine the degree of influence of the sample on the tested service indicator.

[0076] In this embodiment, for any fault scenario sample in the fault scenario sample set, if the total probability of the fault scenario sample's influence on multiple influence indicators is greater than a preset probability standard value, the fault scenario sample is marked as a positive sample; otherwise, the fault scenario sample is marked as a negative sample.

[0077] For example, the total probability P of the impact of the j-th fault scenario sample on multiple influencing indicators calculated above. j This is compared to a set probability standard value A to classify the sample. When P j When the value is greater than A, the j-th fault scenario sample is marked as a positive sample; otherwise, it is marked as a negative sample.

[0078] In operation S240, the positive and negative sample sets are input into the pre-constructed automatic orchestration model for training, resulting in a trained automatic orchestration model used to generate effective fault simulation scenarios.

[0079] This operation uses positive and negative sample data as the data source to continuously optimize the pre-constructed automatic orchestration model, thereby improving its effectiveness.

[0080] In this embodiment, the automatic orchestration model is pre-constructed in the following manner:

[0081] An automatic orchestration model is constructed by mapping the constraints of orchestration methods to the optimization objective using a minimum-cost maximum-flow algorithm. The constraints include minimum-cost and maximum-flow constraints. The minimum-cost constraint is the set of the smallest possible orchestration methods, and the maximum-flow constraint is the set of fault scenario samples with the highest overall probability of impact. The optimization objective is to find the target scenario sample set that maximizes the overall probability of impact on the tested system while minimizing the space of orchestration method types.

[0082] Combining machine learning algorithms, this disclosure selects the minimum cost maximum flow algorithm, which is relatively most suitable for the characteristics of this model, as the basis and makes adaptation modifications to construct an automatic orchestration model.

[0083] Figure 5 The flowchart illustrating the construction of an automatic orchestration model using the minimum cost maximum flow algorithm according to an embodiment of the present disclosure is shown.

[0084] Specifically, such as Figure 5 As shown, the minimum cost maximum flow algorithm includes operations S2401 to S2404.

[0085] In operation S2401, based on the position and role of each component data in the system under test, corresponding weights are set for each component data.

[0086] Assume the system under test contains M components. Then the sample space of the entire tested system has Based on the position and role of each component's data in the system under test, a corresponding weight W is assigned to each component's data. i (1≤i≤M).

[0087] In operation S2402, select a fault scenario sample corresponding to any component data as the initial fault step, calculate the product of the total probability of the impact of the fault scenario sample on multiple impact indicators and the weight of the component data, and determine the product as the fault probability of the initial fault step on the orchestration link.

[0088] Select a fault scenario sample j corresponding to a certain component data i as the initial fault step, and then, based on the total probability of the influence of the fault scenario sample j on multiple influencing indicators and the weight W of the component data i set in the above operation S2401,... i The product of these factors serves as the initial fault step for the fault probability P of the orchestration link. ij .

[0089] In operation S2403, repeat operation S2402 above, sequentially select other fault scenario samples in the fault scenario sample set, calculate the corresponding fault probability and continuously add it to the orchestration link until the accumulated fault probability on the orchestration link reaches the preset fault probability threshold.

[0090] Select other fault scenario samples in sequence, calculate the fault probability of the fault scenario sample to the orchestration link in the same way as the above operation S2402, and continuously add it to the orchestration link until the preset fault probability threshold is reached.

[0091] In operation S2404, multiple fault scenario samples selected on the orchestration link are identified as the target scenario sample set.

[0092] At the moment when the above operation S2403 terminates, the set of various selected fault scenario samples is the set of the smallest sorting method, which is determined as the target scenario sample set.

[0093] Through the above embodiments, the minimum cost maximum flow algorithm is used to select faults during orchestration, and the target scenario sample set is selected based on the fault probability threshold Y set for this orchestration. This disclosure applies machine learning to the field of fault simulation scenario orchestration in chaos engineering, effectively reducing the difficulty of fault scenario orchestration and improving orchestration efficiency.

[0094] In this embodiment, the training of the pre-constructed automatic orchestration model involves inputting positive and negative sample sets into the model. The training process of the automatic orchestration model undergoes multiple rounds of iterative optimization. In each round of optimization, the negative sample set is sampled according to a preset ratio, adjusting the input ratio of the positive and negative sample sets. Thus, in each round of optimization of the automatic orchestration model, a certain proportion of negative samples is sampled, continuously adjusting the ratio of positive to negative samples as sample data for model iteration, ensuring that the loss value of the model's loss function continuously decreases.

[0095] By continuously iterating and optimizing the automatic orchestration model, the goal of automatically generating fault drill scenarios using machine learning methods is achieved. This improves both the efficiency and effectiveness of orchestration capabilities, enabling the automatic orchestration of fault drill scenarios that meet user needs.

[0096] It should be noted that for a given service of a system under test, its components and architecture may change with version iterations and upgrades. Therefore, regarding the automatic orchestration of fault scenarios, it is necessary to distinguish which steps in operations S210 to S240 are mandatory or auxiliary steps based on the specific circumstances.

[0097] For example, when a system under test (SUT) performs its first fault scenario orchestration exercise, all operations S210 through S240 are mandatory. Similarly, when training an orchestration model at a relatively mature stage, if the components, link call relationships, architecture, and scenario sample set of the SUT are fixed, then operations S210 through S230 are not required every time and are considered preparatory steps. Furthermore, when the SUT undergoes a major version upgrade, resulting in changes to components, architecture, or scenario sample set, and a new round of fault scenario orchestration is performed, then all operations S210 through S240 are mandatory.

[0098] It should also be noted that in this embodiment, the fault scenario sample set mainly consists of three types of data: historical events, random scenarios, and artificial scenarios. The data volume is limited and has certain limitations. However, it should be understood that the sample set serves as the foundational source data for the iterative optimization of the automatic orchestration model. A larger and more diverse data set will result in better training performance for the automatic orchestration model. Therefore, in other optional implementations, if the sample set can be expanded, a large number of fault simulation scenarios can be quickly and automatically constructed as a sample set, providing better source data for the automatic orchestration model and significantly improving its effectiveness.

[0099] In an optional implementation, after the above-described operation S240, the chaotic engineering fault simulation scenario generation method further includes operation S250.

[0100] When operating the S250, the trained automatic orchestration model is loaded into the target system, various parameters of the system under test are input, and a valid fault simulation scenario for the target system is generated.

[0101] The target system can be a specific application system. Users can load the trained automatic orchestration model into the application system, inputting multiple component data of the system under test, fault scenario sample sets, fault probability thresholds, and other parameter data. The trained automatic orchestration model can then automatically orchestrate and provide effective fault orchestration scheme data.

[0102] By arranging fault simulation scenarios in chaos engineering, the method provided in this disclosure can effectively solve the problems of high difficulty and low efficiency in manually constructing complex fault scenarios, and greatly improve the efficiency of constructing complex fault scenarios in the field of chaos engineering, thereby improving the efficiency of the tested system in conducting routine fault simulations and providing stronger protection for the stability and fault tolerance of the tested system.

[0103] In addition, the method provided in this disclosure also applies machine learning algorithms to this field. Through continuous learning and optimization of sample data, the arrangement of fault drill scenarios becomes more intelligent and flexible. At the same time, it can automatically arrange and generate more targeted fault drill scenarios, which greatly liberates manpower and effectively makes up for the limitations of manually constructing fault scenarios.

[0104] Based on the above method, this disclosure also provides a chaotic engineering fault simulation scenario generation device. The following will combine... Figure 6 The device is described in detail.

[0105] Figure 6 The diagram illustrates the structure of a chaotic engineering fault simulation scenario generation apparatus according to an embodiment of the present disclosure.

[0106] like Figure 6 As shown, the chaotic engineering fault simulation scenario generation device 600 of this embodiment includes a sample extraction module 610, a probability calculation module 620, a sample classification module 630, and a model training module 640.

[0107] The sample extraction module 610 is used to acquire multiple component data on the service link under test of the system under test, and extract multiple preset dimensions of fault scenario sample sets from the multiple component data. The fault scenario sample sets include historical scenario sample sets, random scenario sample sets, and artificial scenario sample sets. In one embodiment, the sample extraction module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0108] The probability calculation module 620 is used to apply the fault scenario sample set to the system under test for experiments. The system under test has multiple influence indicators preset for the service under test. Based on the experimental results of the multiple influence indicators, the total probability of the influence of each fault scenario sample on the multiple influence indicators is calculated. In one embodiment, the probability calculation module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0109] The sample classification module 630 is used to classify the fault scenario sample set into a positive sample set and a negative sample set based on the total probability of each fault scenario sample's influence on multiple influencing indicators and a preset probability standard value. In one embodiment, the sample classification module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0110] The model training module 640 is used to train a pre-constructed automatic orchestration model by inputting positive and negative sample sets, thereby obtaining a trained automatic orchestration model for generating effective fault simulation scenarios. In one embodiment, the model training module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0111] Through the embodiments of this disclosure, machine learning is applied to the field of fault simulation scenario orchestration in chaos engineering, which effectively reduces the difficulty of fault scenario orchestration and improves orchestration efficiency.

[0112] According to embodiments of this disclosure, any plurality of modules among the sample extraction module 610, probability calculation module 620, sample classification module 630, and model training module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the sample extraction module 610, probability calculation module 620, sample classification module 630, and model training module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the sample extraction module 610, probability calculation module 620, sample classification module 630, and model training module 640 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0113] Figure 7A block diagram schematically illustrates an electronic device suitable for implementing a method for generating chaotic engineering fault simulation scenarios according to an embodiment of the present disclosure.

[0114] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0115] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0116] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0117] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0118] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0119] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the chaos engineering fault simulation scenario generation method provided in the embodiments of this disclosure.

[0120] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0121] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0122] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0123] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0126] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for generating chaotic engineering fault simulation scenarios, comprising: Data from multiple components is acquired on the service link of the system under test, and a sample set of fault scenarios with multiple preset dimensions is extracted from the multiple component data. The fault scenario sample set is applied to the system under test for experimentation. The system under test has multiple impact indicators preset for the service under test. Based on the experimental results of the multiple impact indicators, the total probability of the impact of each fault scenario sample on the multiple impact indicators is calculated. Based on the total probability of the impact of each fault scenario sample on the multiple impact indicators and the preset probability standard value, the fault scenario sample set is classified into a positive sample set and a negative sample set. The positive and negative sample sets are input into a pre-constructed automatic orchestration model for training to obtain a trained automatic orchestration model, which is used to generate effective fault simulation scenarios. The automatic orchestration model is pre-constructed in the following manner: An automatic orchestration model is constructed by mapping the constraints and optimization objectives involved in the orchestration methods using a minimum cost maximum flow algorithm. The constraints involved in the orchestration methods include minimum cost constraints and maximum flow constraints. The minimum cost constraints are the set of the smallest possible orchestration methods, and the maximum flow constraints are the fault scenario samples with the highest total probability of impact. The optimization objective is to find the target scenario sample set that has the highest total probability of impact on the tested system under the condition of minimizing the space of orchestration method types.

2. The method according to claim 1, wherein, The process of acquiring multiple component data on the service link under test of the system under test includes: The system obtains multiple services and components along the service link of the system under test, splits and marks the multiple services and components, obtains the data number of each component, and uses it as component data.

3. The method according to claim 1, wherein, The fault scenario sample set includes a historical scenario sample set, a random scenario sample set, and a manually generated scenario sample set. Multiple preset dimensions of fault scenario sample sets are extracted from the data of the multiple components, including: Multiple historical fault scenarios are obtained from the data of the multiple components, and the multiple historical fault scenarios are arranged and simulated to construct a historical scenario sample set. A random scene sample set is generated using a random algorithm; A set of artificial scene samples was generated using a manual arrangement method.

4. The method according to claim 1, wherein, The aforementioned multiple impact metrics include business metrics and resource metrics. The business metrics include TPS and response latency, and the resource metrics include CPU, memory, disk I / O, and network.

5. The method according to claim 1, wherein, The total probability of the impact of each fault scenario sample on the multiple impact indicators is calculated using the following formula: In the formula, n is the total number of items affecting the index; For the j-th fault scenario sample, the first... The probability of the influence of each influencing indicator; Let be the total probability of the impact of the j-th fault scenario sample on multiple influencing indicators.

6. The method according to claim 5, wherein, The j-th fault scenario sample is for the first... The probability of influence of each influencing indicator is calculated using the following formula: In the formula, The expected value of the indicator is obtained manually based on the accumulated experience of daily testing of the system under test. The actual value of the indicator is taken from the experimental results of the j-th fault scenario sample applied to the system under test.

7. The method according to claim 1, wherein, Based on the total probability of the impact of each fault scenario sample on the multiple impact indicators and a preset probability standard value, the fault scenario sample set is classified into a positive sample set and a negative sample set, including: For any fault scenario sample in the fault scenario sample set, if the total probability of the fault scenario sample's influence on the multiple influence indicators is greater than a preset probability standard value, the fault scenario sample is marked as a positive sample; otherwise, the fault scenario sample is marked as a negative sample.

8. The method according to claim 1, wherein, The minimum cost maximum flow algorithm specifically includes the following steps: Based on the position and role of each component data in the system under test, assign corresponding weights to each component data. Select a fault scenario sample corresponding to any component data as the initial fault step, calculate the product of the total probability of the impact of the fault scenario sample on the multiple impact indicators and the weight of the component data, and determine the product as the fault probability of the orchestration link of the initial fault step. Repeat the above steps for calculating the failure probability, sequentially select other failure scenario samples in the failure scenario sample set, calculate the corresponding failure probability and continuously add it to the orchestration link until the accumulated failure probability on the orchestration link reaches the preset failure probability threshold. The selected fault scenario samples on the orchestration link are determined as the target scenario sample set.

9. The method according to claim 1, wherein, The training of the pre-constructed automatic arrangement model by inputting the positive and negative sample sets into the model also includes: The training process of the automatic arrangement model goes through multiple rounds of iterative optimization. In each round of iterative optimization, the negative sample set is sampled according to a preset ratio, and the input ratio of the positive sample set and the negative sample set is adjusted.

10. The method according to claim 1, wherein, The method further includes: The trained automatic orchestration model is loaded into the target system, and various parameters of the system under test are input to generate an effective fault simulation scenario corresponding to the target system.

11. A chaotic engineering fault simulation scenario generation device, comprising: The sample extraction module is used to acquire multiple component data on the service link under test of the system under test, and extract a sample set of fault scenarios with multiple preset dimensions from the multiple component data. The probability calculation module is used to apply the fault scenario sample set to the system under test for experiments. The system under test has multiple influence indicators preset for the service under test. Based on the experimental results of the multiple influence indicators, the module calculates the total probability of the influence of each fault scenario sample on the multiple influence indicators. The sample classification module is used to classify the fault scenario sample set into a positive sample set and a negative sample set based on the total probability of the influence of each fault scenario sample on the multiple influence indicators and a preset probability standard value. The model training module is used to input the positive sample set and the negative sample set into the pre-constructed automatic orchestration model for training, and to obtain the trained automatic orchestration model, which is used to generate effective fault drill scenarios. The automatic orchestration model is pre-constructed in the following manner: An automatic orchestration model is constructed by mapping the constraints and optimization objectives involved in the orchestration methods using a minimum cost maximum flow algorithm. The constraints involved in the orchestration methods include minimum cost constraints and maximum flow constraints. The minimum cost constraints are the set of the smallest possible orchestration methods, and the maximum flow constraints are the fault scenario samples with the highest total probability of impact. The optimization objective is to find the target scenario sample set that has the highest total probability of impact on the tested system under the condition of minimizing the space of orchestration method types.

12. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images