Chaos engineering

Abstract

A chaos engineering synthesising system comprising a configuration user interface for receiving user inputs, an orchestration tool for compiling a chaos engineering test, and a chaos engineering toolkit comprising a set of chaos engineering tools is provided The orchestration tool receives, from the configuration user interface, the user inputs, the user inputs comprising application location information associated with an application to be tested and an attack. The chaos engineering test is compiled based on the user inputs and executed by the orchestration tool. When executing the chaos engineering test, the orchestration tool selects a chaos engineering tool from the set of chaos engineering tools of the chaos engineering toolkit based on the user inputs, causes the selected chaos engineering tool to generate simulation inputs to simulate the attack, and provides the simulation inputs to the application to be tested.

Description

[0001] Chaos Engineering

[0002] Technical Field

[0003] The present disclosure relates to a chaos engineering synthesising system and a method for executing a chaos engineering test.

[0004] Background

[0005] Chaos engineering is a process of testing a computer system to ensure it can withstand turbulent or unexpected conditions in production.

[0006] The main concept behind chaos engineering is to simulate these unexpected conditions in software being tested, thereby identifying any shortfalls in the system’s resilience. Controlled amounts of failure are injected into a system, and the system observed for its response.

[0007] Chaos engineering tests, or experiments, are hypothesis-driven experiments, meaning they seek to address a specific question. Chaos engineering focuses on experiments / hypotheses and then compares the results to a control, that is a steady state. An example of a chaos engineering test in a distributed system is taking down random services to see how items respond and at what detriment to the user experience.

[0008] Chaos engineering provides controlled testing of failures within a system. In other words, precise and measured amounts of harm are injected into the system to observe how the system responds for the purpose of improving the system’s resilience. By observing how a system operate under these test conditions, it can be inferred how it will operate under similar conditions after being deployed to production.

[0009] Chaos engineering experiments follow four steps:

[0010] 1. Define a “steady state” as some measurable output of a system that indicates normal behaviour. Knowing the steady-state is required for detecting deviations therefrom. The steady-state is the control group.

[0011] 2. Hypothesize that this steady state will continue in both the control group and the experimental group. Chaos engineering is designed to be run against robust and steady systems, trying to find faults such as application failures or infrastructure failures. Running Chaos engineering against unsteady systems does not provide much value, since those systems are already unreliable and instability is known.

[0012] 3. Introduce variables that reflect real world events like servers that crash, hard drives that malfunction, network connections that are severed, etc. to see how the system responds. A failure could be, for example, a hardware failure or a network interruption.

[0013] 4. Try to disprove the hypothesis by looking for a difference in steady state between the control group and the experimental group.

[0014] Based on the results of the chaos engineering experiments, the software is modified by a developer to overcome address the failures caused by the introduced chaos. Chaos engineering can be used to achieve resilience against infrastructure failures, network failures, and application failures, with the possible risks identified before they impact users.

[0015] There are a number of chaos engineering tools and platforms available. These tools typically offer specific functionalities and target specific platforms or application types.

[0016] Summary

[0017] Chaos engineering aims to test system resilience by intentionally introducing failures, but the process can be complex, time-consuming, and challenging to implement in different environments. These problems are particularly emphasised when attempting to conduct chaos engineering experiments across various target platforms and applications.

[0018] Herein is provided a chaos engineering synthesiser which allows developers to write and execute chaos experiments with ease. It offers flexibility in choosing chaos engineering tools, supports a wide range of target platforms, and simplifies the process of running experiments, promoting a proactive approach to identifying and mitigating potential system weaknesses before they lead to critical failures.

[0019] According to a first aspect, there is provided a chaos engineering synthesising system comprising: a configuration user interface for receiving user inputs; an orchestration tool for compiling a chaos engineering test; and a chaos engineering toolkit comprising a set of chaos engineering tools; wherein the orchestration tool is configured to: receive, from the configuration user interface, the user inputs, the user inputs comprising application location information associated with an application to be tested and an attack; compile the chaos engineering test based on the user inputs; and execute the chaos engineering test; wherein the orchestrion tool is configured when executing the chaos engineering test to: select a chaos engineering tool from the set of chaos engineering tools of the chaos engineering toolkit based on the user inputs; cause the selected chaos engineering tool to generate simulation inputs to simulate the attack; and provide the simulation inputs to the application to be tested.

[0020] In some embodiments, the user input may further comprise a user defined chaos engineering tool, wherein the selected chaos engineering tool is selected based on the user defined chaos engineering tool.

[0021] In some embodiments, the orchestration tool may further be configured to determine a suggested chaos engineering tool from the set of chaos engineering tools based on the user inputs.

[0022] In some embodiments, the application location information may comprise a hosting platform on which the application is hosted.

[0023] In some embodiments, the application location information may comprise a uniform resource locator, URL, for locating the application, wherein the chaos engineering test is executed at a location defined by the URL.

[0024] In some embodiments, the system may further comprise a network interface for communicating with the application hosted at a hosting platform via a network.

[0025] In some embodiments, the system may be configured to communicate with a plurality of hosting platform types, wherein the plurality of platform types comprises at least one of a public cloud, a private cloud, and an on-premises server.

[0026] In some embodiments, the orchestration tool may be configured to compile the configuration file by: obtaining, based on the user inputs, an incomplete configuration file associated with the user inputs from a configuration file database; and populating the incomplete configuration file based on the user inputs to obtain the configuration file. In some embodiments, the orchestrion tool may comprise the configuration file database.

[0027] In some embodiments, the orchestration tool may be further configured to: receive, from the application, test results generated in response to the simulation inputs; and provide the test results to the selected chaos engineering tool for analysing.

[0028] In some embodiments, the orchestrion tool may comprise a results database, wherein the orchestration tool is further configured to store the received test results at the results database.

[0029] In some embodiments, the orchestration tool may be further configured to determine that the chaos engineering test is complete and, in response, provide the test results to the selected chaos engineering tool.

[0030] In some embodiments, the orchestration tool may be further configured to: receive, from the selected chaos engineering tool, an analysis report; and provide a user report interface based on the analysis report.

[0031] In some embodiments, the orchestration tool may be further configured to: receive a plurality of sets of user inputs from the user configuration interface, wherein each set comprises application location information associated with the application to be tested and an attack; for each set of the plurality of sets of user inputs, compile a respective chaos engineering test based on the user inputs; and execute each of the respective chaos engineering tests.

[0032] In some embodiments, the orchestration tool may be further configured to: receive, from the application, test results generated in response to each of the respective chaos engineering tests; and provide the test results to the selected chaos engineering tools for analysing.

[0033] In some embodiments, the orchestration tool may be further configured to: receive, from the selected chaos engineering tools, a respective analysis reports for each of the respective chaos engineering tests; and compile a user report based on the respective analysis reports; and provide the user report at a user report interface.

[0034] According to a second aspect, there is provided a method of executing a chaos engineering test, the method comprising: providing a user interface for receiving user inputs, the user inputs comprising application location information associated with an application to be tested and an attack; receiving, at an orchestration tool, the user inputs; compiling the chaos engineering test based on the user inputs; and executing the chaos engineering test; wherein the chaos engineering test, when executed, causes the orchestration tool to: select a chaos engineering tool from a set of chaos engineering tools based on the user inputs; cause the selected chaos engineering tool to generate simulation inputs to simulate the attack; and provide the simulation inputs to the application to be tested.

[0035] In some embodiments, compiling the configuration file may comprise: obtaining, based on the user inputs, an incomplete configuration file associated with the user inputs from a configuration file database; and populating the incomplete configuration file based on the user inputs to obtain the configuration file.

[0036] In some embodiments, the method may further comprise: receiving, from the application, test results generated in response to the simulation inputs; and providing the test results to the selected chaos engineering tool for analysing.

[0037] In some embodiments, the method may further comprise storing the received test results at a results database.

[0038] In some embodiments, the method may further comprise: receiving, from the selected chaos engineering tool, an analysis report; and providing a user report interface based on the analysis report.

[0039] In some embodiments, the method may further comprise: receiving a plurality of sets of user inputs from the user configuration interface, wherein each set comprises application location information associated with the application to be tested and an attack; for each set of the plurality of sets of user inputs, compiling a respective chaos engineering test based on the user inputs; and executing each of the respective chaos engineering tests.

[0040] In some embodiments, the method may further comprise: receiving, from the application, test results generated in response to each of the respective chaos engineering tests; and providing the test results to the selected chaos engineering tools for analysing. In some embodiments, the method may further comprise: receiving, from the selected chaos engineering tools, a respective analysis reports for each of the respective chaos engineering tests; compiling a user report based on the respective analysis reports; and providing the user report at a user report interface.

[0041] According to a third embodiment, there is provided a non-transitory computer program product comprising instructions which, when implemented by a at least one processor, cause a computing device to be configured to implement the method or system provided herein.

[0042] Brief Description of the Drawings

[0043] To assist understanding of embodiment of the present disclosure and to show how such embodiment may be put into effect, reference is made, by way of example only, to the accompanying drawings in which:

[0044] Figure 1 shows a schematic diagram of a computer system for implementing a chaos engineering synthesising system;

[0045] Figure 2 shows an example configuration user interface; and

[0046] Figure 3 shows a schematic diagram of an orchestration tool.

[0047] Detailed Description of Embodiments

[0048] Figures 1 shows an example distributed system 100 in which a chaos engineering synthesising system 102 is implemented for testing applications hosted on a set of hosting platforms 104.

[0049] In the system 100, three applications are shown to be hosted on a public cloud 104a, three applications are hosted on a private cloud 104b, and three applications are shown hosted on an on-premises server 104c. It will be apricated that the number of applications shown in Figure 1 are provided by way of example only. Additional hosting platforms 104 may also be used, for example there may be two or more public clouds 104a, and / or there may be other types of hosting platforms 104 such as a remote server, or a data centre. When a new application is developed, or an existing application updated, the application is tested to ensure users are able to use the application as intended by the developer 112. The phrase "as intended” includes factors such as latencies and other user perceived quality factors, as well as functionality.

[0050] In some instances, an application is first launched on one type of hosting platform 104, and then switched to another type of hosting platform 104. Transformations of this kind have become more common with the prevalence of cloud computing, leading to applications originally launched on local or remote servers now being hosted on cloud servers. With this transformation, new sources of latencies or other previously unexperienced issues for users may be introduced. Problems may arise due to changes in input / output (I / O) conditions or the Request / Response from the user / application based on the business rule, the distribution of server or limited availability of CPU resources resulting in new latencies, and / or a state of a server may change for instance it may be shut down or some services removed. In the above, “business rules” refers to the specific instructions or conditions that dictate how the information is requested from and provided by the API, that is, the definition of requested guidelines and Response expectation.

[0051] To identify such problems before the applications are made available to users, chaos engineering is used.

[0052] The chaos engineering synthesising system 102 is provided to enable efficient implementation and management of chaos engineering experiments in applications across multiple target hosting platforms 104. The chaos engineering synthesising system 102 may be provided as part of a configuration integration / continuous development (CI / CD) pipeline.

[0053] The chaos engineering synthesising system 102 is configured to run chaos tests on the applications based on user provided inputs, and generate results based on the chaos tests which can be used by an engineer or developer 112 to modify the applications such that they meet resilience requirements.

[0054] The chaos engineering synthesising system 102 may be executed on one or more local computing devices, or on a cloud-based computing system. The chaos engineering synthesising system 102 may communicate with the hosting platforms 104 via a network such as the Internet.

[0055] The chaos engineering synthesising system 102 comprises a configuration user interface 106, an orchestration tool 108, and a chaos engineering toolkit 110 comprising a plurality of chaos engineering tools. The chaos engineering tools of the chaos engineering toolkit 110 may be open-source chaos engineering tools, chaos engineering tools which have been licensed from third-parties, and / or specifically designed in-house chaos engineering tools. Example chaos engineering tools provided in the example toolkit 110 of Figure 1 are Gremlin ©, Netflix™’ s Chaos Monkey, WireMock™, and Amazon™’ s AWS Fault Injection Simulator (FIS).

[0056] The engineering toolkit 110 comprises multiple engineering tools so that applications hosted on different platforms 104 can be tested using the same chaos engineering synthesising system 102. Engineering tools are tailored to specific hosting platforms 104 so that they can run tests which accurately identify the problems caused by, or associated with, the specific hosting platform 104.

[0057] The configuration user interface 106 is provided on a computing device of a user or developer 112. The developer 112 provides input information via the configuration user interface 106 to enable the orchestration tool 108 to select one of the chaos engineering tools of the toolkit 110 and automatically configure the chaos engineering test. The orchestration tool 108 then runs the configured chaos engineering test on the application identified by the developer 112.

[0058] The orchestration tool 108 passes results of the test from the application to the relevant chaos engineering tool, which generates a developer interpretable report. The report may be referred to as an analysis report. The report is presented to the developer 112 via a report user interface on their computer device.

[0059] The report indicates issues which have been identified using the results of the chaos engineering test. The developer 112 is then able to use the report to update the application software to overcome the identified problems. Further chaos engineering tests can be run after updating the software to check that the problems have been overcome and that no further latencies have been introduced by the changes. In some embodiments, as discussed later, the developer 112 can define multiple tests to be executed on an application. The set of multiple tests may be referred to as a project. The report provided to the developer 112 is a consolidation of the reports generated by the chaos engineering tools, compiled by the chaos engineering synthesising system 102, and more specifically by the orchestration tool 108. In this way, the final report, also referred to as a user report, provided to the developer 112 is associated with the project.

[0060] Figure 2 shows an example configuration user interface 106, which allows the developer 112 to provide information which can be used by the chaos engineering system 102 to orchestrate and execute the chaos test.

[0061] The configuration user interface 106 comprises three fields 202, 204, 206 with drop-down menus, providing options for the developer 112 to select from.

[0062] A chaos tool field 202 allows the developer 112 to select one of the chaos engineering tools from the chaos engineering toolkit 110 for executing chaos on the backend, that is for use when testing the application.

[0063] A platform field 204 provides the developer with a list of hosting platforms 104, such that the developer 112 can select the hosting platform 104 on which the application to be tested is hosted.

[0064] An attack vector field 206 allows the developer 112 to select an attack for executing, that is an area of the application and its execution to be tested. Example attacks are CPU, memory, appcrash, latency, and exception.

[0065] A text input field 208 is provided for identifying the cloud factory datacentre URL where the attack is to be executed. If the application is stored instead in a server instead of the cloud, this text field 208 may be used to defined the file path of the application.

[0066] There are four further fields shown on the configuration user interface 106. These fields may be specific to one or more of the selected tool, platform, or attack, and therefore may only be required in some instances. For example, a server host name field 212 is only required with the example system of Figure 1 in the case that the Gremlin © tool is used. Other fields which may be required, and the conditions which may require them, will be apparent to the person skilled in the art.

[0067] The inputs provided by the developer 112 define the parameters of the chaos engineering experiment to the compiled. In this way, the developer 112 designs the chaos engineering experiment.

[0068] Once the developer 112 is satisfied with the inputs they have provided, they initiate the test compilation by selecting a build button 210 on the user interface 106. This cases the inputs to the passed to the orchestration tool 108.

[0069] In some embodiments, the chaos engineering synthesising system 102, or more specifically the orchestration tool 108, may provide a suggestive functionality, in which it suggests the chaos engineering tools for use in the test. The chaos engineering system 102 may make the suggestions based on other developer provided information, such as the hosting platform, attack, or the application URL. In such an embodiment, the chaos engineering system 102 takes the developer provided information and suggests a tool using predefined or learned rules.

[0070] In the suggestive method, the developer provided information may be passed to the orchestration tool 108 as they are entered by the developer 112 in the configuration user interface 106. The orchestration tool 108 processes the inputs in real-time to suggest other parameters, such as the tool, to the developer 112 via the configuration user interface 106.

[0071] The suggested chaos engineering tool may be used by the orchestration tool 108 to configure the chaos engineering test automatically. Alternatively, the suggested chaos engineering tool may be provided to the user at the configuration user interface 106 so that the user confirms the use of the suggested tool, or select a different tool for use in the test. In such an embodiment, the suggested tool if confirmed by the user may be considered an user input.

[0072] In some embodiment, the hosting platform 104 may be inferred from other developer provided information. For example, the orchestration tool 108 may determine the hosting platform from the URL of the application. As such, the hosting platform 104 may not be provided by the developer 112. The application URL and the hosting platform 104 may each be referred to herein as application location information.

[0073] Figure 3 illustrates test compilation by the orchestration tool 108. The orchestration tool 108 comprises a compiler module 306, a results database 308, and a configuration file database 310.

[0074] The configuration file database 310 comprises a set of incomplete configuration files, also referred to as template configuration files. Each incomplete configuration files corresponds to one of the chaos engineering tools of the engineering toolkit 110. Each of the incomplete configuration files is also specific to a hosting platform 104 and the attack.

[0075] The incomplete configuration files are predefined by a developer 112 when a new chaos engineering tool is added to the chaos engineering toolkit 110, and / or when a new hosting platform 104 is used, and / or when a new attack is defined. Incomplete configuration files may also be removed from the database 310 if they are no longer required, for example if the corresponding chaos engineering tool is removed from the toolkit 110.

[0076] The orchestration tool 108 receives the developer defined inputs 302 from the configuration user interface 106. Using the defined chaos engineering tool, the defined hosting platform 104, and the defined attack, the corresponding incomplete configuration file is obtained from the configuration file database 310 and passed to the compiler module 306.

[0077] The compiler module 306 compiles a complete configuration file by modifying the incomplete configuration file as obtained from the configuration file database 310 to include data provided in the inputs 302, the location of URL of the application to be tested. By completing the configuration file, the compiler module 306 of the orchestration tool 108 compiles the chaos engineering test.

[0078] The completed configuration file is executable to initiate the chaos engineering test.

[0079] When executed, the completed configuration file causes the orchestration tool 108 to access the chaos engineering toolkit 110 and select the chaos engineering tool as defined in the inputs 302. The orchestration tool 108, based on the completed configuration file, configures the selected chaos engineering tool using the user input defining the attack, such that the selected chaos engineering tool simulates the required chaos. Chaos refers to simulated unexpected or turbulent behaviour. The tools create simulations relating to resourcing such as I / O, disk, memory, etc., and / or relating to state such as shutdown, process kill, etc., and / or relating to network such as packet loss, latency, etc.

[0080] The orchestration tool 108 performs a matchmaking process in which it matches the chaos engineering tool to the infrastructure, that is the application for testing 304, based on the completed configuration file.

[0081] During the chaos engineering experiment, or test, the selected chaos engineering tool generates the simulation data corresponding to the attack, which is passed to the application 304 via the orchestration tool 108 using the complete configuration file. Results from the experiment are passed from the application 304 to the orchestrion tool 108 and stored in the results database 308 throughout the test. The simulation data provided to the application 304 to simulate the attack may also be referred to as simulation inputs.

[0082] The results are stored at the results database 308 with an identifier associated with the test. The results may also be stored with the completed configuration file, and / or the developer provided inputs.

[0083] The chaos engineering experiment can be executed at specific API's, rather than over the whole application 304. In this case, the impact would be assessed over the full flow of the API's within application.

[0084] Once the experiment is complete, the chaos engineering tool obtains the stored results from the results database 308 and generates a report based on the results. The results are provided by the orchestration tool 108 when it determines that the test is complete. The report identifies problems which have been found during the experiment. The report is made available to the developer 112 via the orchestration tool 108.

[0085] The report allows the developer 112 to understand what aspects of the application implementation are causing the issues to arise, and modify the application 304 to overcome these issues. By using the chaos engineering synthesising system 102 described above, the developer 112 is not required to compile the chaos engineering experiment manually each time an experiment is to be executed, as is custom in the art. Instead, the developer 112 need only provide the necessary inputs via the configuration user interface 106, and the orchestration tool 108 automatically compiles the experiment based on the user inputs. This significantly improves the efficiency of implementing chaos engineering experiments, and thus too launching or updating software applications to users.

[0086] In some embodiments, when a developer 112 accesses the configuration user interface 106 to define the inputs for the chaos engineering test, the configuration user interface 106 may be pre-populated with default inputs. For example, the tool field 202 and the attack field 206 may be pre-populated with the inputs for running a database chaos test. The developer 112 can modify the inputs in these fields 202, 206 to define and execute a different chaos test.

[0087] In some embodiments, multiple pipelines, i.e. tests, are set up for a single application. For example, a developer 112 can define a test for testing the response to changes VO resourcing, and another test for testing network latencies. These tests may be grouped to define a project. This enables different chaos engineering tests to be run simultaneously. The chaos engineering tests are scheduled and run during the development of the app 304, and the frequency of the tests is set by the developer 112. The developer 112 defines each tests of the project using the configuration interface 106 as set out above. The configuration user interface 106 may comprise a further field for defining a project for the test to be associated with, for example. Once the developer 112 has defined all tests of the project, the tests are executed by the chaos synthesising system 102.

[0088] As mentioned above, the chaos engineering synthesising system 102 may be modified by adding or removing chaos engineering tools to the chaos engineering toolkit 110 as they become necessary or redundant respectively. This allows the chaos engineering synthesising system 102 to be aligned with the platforms on which applications are hosted, and keep in line with modifications and transformations of the applications.

[0089] When a tool is added or removed, the configuration user interface 106 is updated to reflect the options provided by the set of tools in the toolkit 110. For example, the tool field 202 is updated to include all of and only those chaos engineering tools in the toolkit 110. Also, the attack field 206 may be updated if the possible attacks have changed due to a change in the available tools.

[0090] The chaos engineering synthesising system 102 abstracts the complexity of running chaos experiments across different platforms 104. This allows developers 112 to focus on designing experiments and selecting the appropriate chaos tools without the need to handle intricate setup and integration tasks.

[0091] Unlike traditional chaos engineering tools that may be tied to specific platforms 104, the chaos engineering synthesising system 102 offers the freedom to select any chaos engineering tool based on the target environment or application. This flexibility provides developers with a wider range of options and accommodates various use cases.

[0092] The chaos engineering synthesising system 102 is designed to work across different target platforms 104, including cloud environments, on-premises servers, data centres, and CI / CD pipelines. This cross-platform support enables its use in diverse settings, making it suitable for a broader range of applications.

[0093] The chaos engineering synthesising system 102 allows developers 112 to create their own chaos engineering tools which can be integrated with the chaos engineering synthesising system 102, enabling developers 112 to tailor chaos engineering experiments to their specific requirements and integrate the chaos engineering synthesising system 102 with existing tools or internal systems. This extensibility enhances the system’s adaptability and versatility.

[0094] With an implemented chaos engineering synthesising system 102 as described herein, there is increased system resilience. By facilitating chaos engineering experiments, the chaos engineering synthesising system 102 helps developers build more resilient applications and services. This can lead to improved customer satisfaction, reduced downtime, and increased reliability of the products.

[0095] The chaos engineering synthesising system 102 also enhances software quality. Proactive resilience testing through chaos engineering can result in higher software quality and fewer post-release defects. This, in turn, reduces maintenance costs and customer support efforts. The use of the chaos engineering synthesising system 102 allows for faster time-to-market of application. The system’s ability to integrate with existing chaos engineering tools and platforms allows developers to adopt chaos engineering practices more efficiently. This can accelerate the development process by identifying and addressing potential issues early in the development lifecycle.

[0096] Developers that prioritize resilience testing and implement chaos engineering are better equipped to handle unexpected events and recover quickly from failures. This can give them a competitive edge by providing more reliable and stable services to customers, and being able to combat failures quickly and efficiently, thereby reducing downtime.

[0097] Various methods and devices have been described. It should be appreciated that these methods may be implemented in apparatus or devices comprising any suitable circuitry. Some embodiments may be implemented by at least one memory and at least one processor. The memory is provided by memory circuitry and the processor is provided by processor circuitry. Some embodiments may be provided by a computer program running on the at least one processor. The computer program may comprise computer implemented instructions which are stored in the at least one memory and which may be run on the at least one processor. A computer program product may be provided which comprises computer program product comprising code embodied on a computer-readable medium which is configured to be executed on a processor of the computer or user device. In some embodiments, a non-transitory computer readable storage device may be provided to store program code instructions that, when executed by at least one processor causes any of the above-described methods to be performed.

[0098] A person skilled in the art will realise that the different approaches to distributing and storing data are not exhaustive, what is described herein are certain preferred embodiments. It is possible to implement the way in a number of variations without departing from the scope of the invention as claimed. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

Claims1. A chaos engineering synthesising system comprising: a configuration user interface for receiving user inputs; an orchestration tool for compiling a chaos engineering test; and a chaos engineering toolkit comprising a set of chaos engineering tools; wherein the orchestration tool is configured to: receive, from the configuration user interface, the user inputs, the user inputs comprising application location information associated with an application to be tested and an attack; compile the chaos engineering test based on the user inputs; and execute the chaos engineering test; wherein the orchestrion tool is configured when executing the chaos engineering test to: select a chaos engineering tool from the set of chaos engineering tools of the chaos engineering toolkit based on the user inputs; cause the selected chaos engineering tool to generate simulation inputs to simulate the attack; and provide the simulation inputs to the application to be tested.

2. The system of claim 1, wherein the user input further comprises a user defined chaos engineering tool, wherein the selected chaos engineering tool is selected based on the user defined chaos engineering tool.

3. The system of claim 1 or claim 2, wherein the orchestration tool is further configured to determine a suggested chaos engineering tool from the set of chaos engineering tools based on the user inputs.

4. The system of any preceding claim, wherein the application location information comprises a hosting platform on which the application is hosted.

5. The system of any preceding claim, wherein the application location information comprises a uniform resource locator, URL, for locating the application, wherein the chaos engineering test is executed at a location defined by the URL.

6. The system of any preceding claim, wherein the system further comprises a network interface for communicating with the application hosted at a hosting platform via a network.

7. The system of claim 6, wherein the system is configured to communicate with a plurality of hosting platform types, wherein the plurality of platform types comprises at least one of a public cloud, a private cloud, and an on-premises server.

8. The system of any preceding claim, wherein the orchestration tool is configured to compile the configuration file by: obtaining, based on the user inputs, an incomplete configuration file associated with the user inputs from a configuration file database; and populating the incomplete configuration file based on the user inputs to obtain the configuration file.

9. The system of claim 8, wherein the orchestrion tool comprises the configuration file database.

10. The system of any preceding claim, wherein the orchestration tool is further configured to: receive, from the application, test results generated in response to the simulation inputs; and provide the test results to the selected chaos engineering tool for analysing.

11. The system of claim 10, wherein the orchestrion tool comprises a results database, wherein the orchestration tool is further configured to store the received test results at the results database.

12. The system of claim 11, wherein the orchestration tool is further configured to determine that the chaos engineering test is complete and, in response, provide the test results to the selected chaos engineering tool.

13. The system of any of claims 10 to 12, wherein the orchestration tool is further configured to:receive, from the selected chaos engineering tool, an analysis report; and provide a user report interface based on the analysis report.

14. The system of any preceding claim, wherein the orchestration tool is further configured to: receive a plurality of sets of user inputs from the user configuration interface, wherein each set comprises application location information associated with the application to be tested and an attack; for each set of the plurality of sets of user inputs, compile a respective chaos engineering test based on the user inputs; and execute each of the respective chaos engineering tests.

15. The system of claim 14, wherein the orchestration tool is further configured to: receive, from the application, test results generated in response to each of the respective chaos engineering tests; and provide the test results to the selected chaos engineering tools for analysing.

16. The system of claim 15, wherein the orchestration tool is further configured to: receive, from the selected chaos engineering tools, a respective analysis reports for each of the respective chaos engineering tests; compile a user report based on the respective analysis reports; and provide the user report at a user report interface.

17. A method of executing a chaos engineering test, the method comprising: providing a user interface for receiving user inputs, the user inputs comprising application location information associated with an application to be tested and an attack; receiving, at an orchestration tool, the user inputs; compiling the chaos engineering test based on the user inputs; and executing the chaos engineering test; wherein the chaos engineering test, when executed, causes the orchestration tool to: select a chaos engineering tool from a set of chaos engineering tools based on the user inputs; cause the selected chaos engineering tool to generate simulation inputs to simulate the attack; andprovide the simulation inputs to the application to be tested.

18. The method of claim 17, wherein compiling the configuration file comprises: obtaining, based on the user inputs, an incomplete configuration file associated with the user inputs from a configuration file database; and populating the incomplete configuration file based on the user inputs to obtain the configuration file.

19. The method of claim 17 or claim 18, wherein the method further comprises: receiving, from the application, test results generated in response to the simulation inputs; and providing the test results to the selected chaos engineering tool for analysing.

20. The method of claim 19, wherein the method further comprises storing the received test results at a results database.

21. The method of claim 19 or claim 20, wherein the method further comprises: receiving, from the selected chaos engineering tool, an analysis report; and providing a user report interface based on the analysis report.

22. The method of any of claims 17 to 21, wherein the method further comprises: receiving a plurality of sets of user inputs from the user configuration interface, wherein each set comprises application location information associated with the application to be tested and an attack; for each set of the plurality of sets of user inputs, compiling a respective chaos engineering test based on the user inputs; and executing each of the respective chaos engineering tests.

23. The method of claim 22, wherein the method further comprises: receiving, from the application, test results generated in response to each of the respective chaos engineering tests; and providing the test results to the selected chaos engineering tools for analysing.

24. The method of claim 23, wherein the method further comprises:receiving, from the selected chaos engineering tools, a respective analysis reports for each of the respective chaos engineering tests; compiling a user report based on the respective analysis reports; and providing the user report at a user report interface.

25. A non-transitory computer program product comprising instructions which, when implemented by a at least one processor, cause a computing device to be configured to implement the method or system of any preceding claim.

Images