Testing cloud infrastructure

Abstract

Embodiments of the present disclosure relate to testing a cloud infrastructure. In some embodiments, a method includes receiving, at a running framework controller, a test execution request including test parameters and expected test outcomes, where the test execution request includes a request for testing a micro-service architecture implemented in a cloud infrastructure; providing, by the framework controller, the test execution request and the test parameters to a framework injector running on one or more processors; providing, by the framework controller, an expected test outcome to a framework monitor running on one or more processors; performing, by the frame injector, a test corresponding to the test execution request using a test injector of a plurality of test injectors implemented by the frame injector; monitoring, by a framework monitor, the response of the micro-service architecture of the cloud infrastructure to the test; and providing, by the frame controller, test results based on the response from the frame monitor.

Description

Background Art

[0001] Information technology infrastructure and ecosystem deployments are becoming increasingly complex. As complexity increases, it can become challenging to fully test this environment as a whole and determine the impact that various parts of the environment have on other parts in the event of a failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain embodiments discussed herein will be described with reference to the drawings listed below. However, the drawings illustrate only certain aspects or implementations of the embodiments described herein and are not meant to limit the scope of the claims.

[0003] Figure 1 illustrates a block diagram of an example system for implementing a testing framework for testing a cloud infrastructure according to one or more embodiments disclosed herein;

[0004] Figure 2 illustrates a block diagram of an example framework injector 200 according to one or more embodiments disclosed herein;

[0005] Figure 3 An overview of an example method for testing a cloud infrastructure according to one or more embodiments disclosed herein is illustrated; and

[0006] Figure 4 A block diagram of a computing device according to one or more embodiments of the present disclosure is illustrated. DETAILED DESCRIPTION

[0007] Entities that provide technology and technical services are moving toward models that include microservices and other distributed technologies, which may be implemented in cloud infrastructure environments. As such models are adopted, the complexity of the systems used to implement such technologies may increase. The complexity may increase the difficulty of predicting, identifying the cause of, and / or recovering from system failures, which may be caused by failures in any of multiple locations in the system or a combination of such failures. Unexpected behavior in a complex system, sudden failure of one or more hardware or software components, unexpected delays in response times, and other problems may, for example, result in a system failure during which the entity may be unable to provide services to customers, resulting in customer dissatisfaction.

[0008] As an example, an online application or service may be implemented using any number of microservices in a cloud computing infrastructure environment where the hardware environment is distributed (e.g., in a public cloud, a private cloud, a hybrid cloud, etc.). Such microservices may be implemented at least in part using technologies such as a container platform (e.g., Kubernetes), a virtualized instance of a computing device (e.g., a virtual machine), etc. In such an environment, it may be difficult to fully test the ability of the environment to handle or otherwise respond to various faults, errors, failures, etc., because these problems occurring in the environment may have unpredictable effects on all or any part of the rest of the environment. Problems may include, but are not limited to: software crashes and / or errors (e.g., software crashes and / or errors of a container platform, microservices, containers, pods, virtual machines, operating systems, backup and recovery software, databases, public services, communication protocols, applications, processes, etc.); crashes, freezes, and / or other errors in hardware components (e.g., servers, other computing devices, network devices, storage devices, etc.); and / or errors and / or failures in communications between components. Various components or portions of a distributed computing device used to implement a microservices-based architecture may be tested, but the system as a whole may not have been fully tested and / or may not have been tested in a production-like environment, which may lead to unforeseen potential problems or an additional unknown number of potential problems when the various components, parts, portions, devices, etc. are assembled, connected, deployed, etc. and begin operating to provide services, applications, etc. in a cloud infrastructure environment.

[0009] To solve at least the above problems, embodiments disclosed herein provide a comprehensive framework that can be used to robustly test a microservice architecture implemented in a cloud infrastructure environment. In one or more embodiments, the framework includes a framework controller, a framework injector, and a framework monitor, and the framework can be operably connected to a cloud infrastructure environment in which a microservice architecture (e.g., for running an application) is implemented.

[0010] In one or more embodiments, a framework controller is a component configured to receive input from a user (e.g., a test engineer). Such input may include a definition of what the user wants to test and the expected outcome of performing the test. In one or more embodiments, the framework controller provides details about what the user expects to test to the framework injector and provides the expected outcome to the framework monitor.

[0011] In one or more embodiments, the framework injector can be configured to perform various actions that can be implemented using several testing techniques, including but not limited to interruption techniques, communication error techniques, and crash point injection techniques. The type of input required by the user via the framework controller can depend on the type of technology that the user expects to use to perform tests on the microservice architecture and cloud infrastructure. The expected results of the test can depend on the type of test specified by the user. Therefore, various categories of testing techniques, different types of errors, crashes, freezes, failures, etc. can be presented to users interacting with the framework controller, and these all fall within each category. In one or more embodiments, the user can select or otherwise specify the type of testing technology, the various actions that the user expects to perform using these technologies, and the expected results or responses of performing these actions.

[0012] One test technique that the framework injector can be configured to implement is an interruption technique. In one or more embodiments, an interruption technique is a category of test techniques that provides users with the ability to introduce crashes, freezes, restarts, etc. of various components of the tested environment. In one or more embodiments, the crashes, freezes, restarts, etc. may be caused by hardware devices (e.g., servers, network devices, etc.), platforms (e.g., Kubernetes platforms, container orchestration services, container management systems, virtual machine platforms, etc.), system parts that host microservices (e.g., containers, container pods, virtual machines), one or more microservices, authorization services, databases, workflow orchestration and management services, etc.

[0013] In one or more embodiments, the blast radius of an outage can be controlled based on what part of the microservice architecture and / or cloud infrastructure will have an outage (such as a crash, freeze, failure, etc.). As an example, an outage can be a single container, a container pod, a container platform, a specific microservice of an application, a microservice common to various other parts of the system, and / or an entire container platform. In one or more embodiments, the outages specified by the user via the framework controller are provided to the framework injector.

[0014] In one or more embodiments, the framework injector is configured to perform actions to cause the desired interruption. In one or more embodiments, for example, predefined commands such as Kubernetes commands, various application programming interface (API) commands, etc. are used to perform actions, which will depend on the part of the system to be interrupted. As an example: Kubernetes commands can be used to crash or cause the following to freeze: all or any part of the container platform (for example, causing a restart of a port); any component therein; any microservice implemented thereon, etc. As another example, various API commands can be used to cause crashes, freezes, restarts, etc. of servers, network devices, storage devices, other hardware components, data backup orchestrators, etc.

[0015] In one or more embodiments, the framework monitor observes the environment in which the interruption is introduced to determine whether the environment responds to the interruption as expected. As described above, in one or more embodiments, the expected results of a particular test are provided by a user via a framework controller. In one or more embodiments, the framework controller provides the expected results to the framework monitor. In one or more embodiments, the framework monitor is configured to monitor the environment and the components therein to determine whether the actual response to the interruption matches the expectations provided by the user. As an example, a user may expect that a crash, freeze, restart, etc. of a hardware component will result in a failover to a redundant component, the component will recover within a certain amount of time, and the rest of the environment will continue to operate while the interrupted component recovers, and so on. As another example, a user may specify the expected behavior of an application implemented using microservices when one or more microservices crash. The user may specify other expected behaviors without departing from the scope of the embodiments disclosed herein. In one or more embodiments, if the result of the interruption does not match the expected results provided by the user, the framework monitor may provide an alert to the user. Such an alert may take any form and may include any number and / or type of information. As an example, a user may receive a notification indicating that the expected results did not occur, as well as various details about the environmental response observed by the framework monitor.

[0016] Another testing technique that can be implemented by configuring the framework injector is the communication error technique. In one or more embodiments, the communication error technique includes causing an error when a communication operation is in progress. As an example, various communication protocols may include, but are not limited to, a representative state transfer (Representational State Transfer, REST) ​​protocol, a remote procedure call (remote procedure call, RPC) framework (e.g., gRPC), a Kafka protocol, etc., and other underlying protocols that can be used in communication, such as a hypertext transfer protocol (hypertext transfer protocol, HTTP), a transmission control protocol (transmission control protocol, TCP), a user datagram protocol (user datagram protocol, UDP), etc. These protocols can be used for data communication. For example, these protocols may have related APIs that allow injection of various errors (such as delayed injection, timeout errors, etc.). The communication error technique may also include causing other errors, such as resource errors (causing memory errors, causing processor errors, etc.), server busy errors (e.g., by causing heavy loads on the computing resources of one or more servers), compliance errors (e.g., causing the protocol to run in a manner that does not conform to expected behavior), and verification errors (e.g., causing errors in packet sorting). In one or more embodiments, any of the foregoing communication errors may be specified by a user via a framework controller and injected into the environment by a framework injector. In one or more embodiments, as described above, a framework monitor may monitor the environment to determine whether the behavior observed in the environment matches the expected outcomes specified by the user, and if not, an alert may be provided to the user.

[0017] Another testing technique that the framework injector can be configured to implement is the crash point injection technique. In one or more embodiments, the crash point injection technique includes strategically adding errors, faults, etc. in the microservice code to cause faults at specific points. In one or more embodiments, injecting fault points into the microservice code allows verification of the validity of mechanisms in the microservice (such as undo operations, redo operations, retry operations, etc.). As an example, certain operations can be performed by the microservice in various stages, and crash points can be inserted into the microservice code, and the crash points cause errors at specific points within these stages to verify that the microservice responds in an expected manner. As an example, the microservice can be configured to perform an operation of restoring from a previously created data backup. The restore operation can have three stages: storage (e.g., from a snapshot) restoration, attaching the restored storage to a computing instance (e.g., Amazon Elastic Cloud Compute (EC2) instance) and providing update information related to the storage attached to the instance to the manager service. In one or more embodiments, in this scenario, the crash point can be injected into any point in any of the three stages to cause a fault. In one or more embodiments, the user specifies, via the framework controller, the stage at which the user expects an error to occur and the expected outcome. Continuing with the previous example, the user can provide an expected outcome, i.e., if an error occurs in the third stage of a process being executed by a microservice, the microservice will retry the third stage after restarting, but will not redo the first two stages that have been completed. Based on input from the user, the framework controller can provide information to the framework injector, which can in turn add appropriate one or more failure points to the code of the relevant microservice so that the microservice will reach the failure point during operation. In one or more embodiments, the framework monitor observes the behavior of the microservice, determines whether the behavior matches the expected outcome provided by the user, and alerts the user if the observed behavior does not match the expected behavior.

[0018] Certain embodiments of the present disclosure may provide a framework that allows real-time testing of a microservices architecture implemented in a cloud infrastructure to discover unexpected or otherwise incorrect responses of various components of the environment when faced with various failure scenarios of varying degrees of complexity, which may improve the capabilities of such an environment to make it more robust and resilient, thereby helping to avoid system outages, downtime, and the like.

[0019] Figure 1 A block diagram of an example system for implementing a test framework for testing cloud infrastructure according to one or more embodiments disclosed herein is illustrated. Figure 1As shown, the system may include a cloud infrastructure 100. The cloud infrastructure 100 may include any number of microservices (e.g., microservice A 102, microservice B 104, microservice C 108, microservice N 110), a user interface microservice 106, one or more public microservices 114, and a communication component 112. The system may also include a test framework 116. The test framework 116 may include a framework controller 118, a framework monitor 120, and a framework injector 122. Each of these components is described below.

[0020] In one or more embodiments, the cloud infrastructure 100 is a collection of any number of computing devices and other components in which a cloud computing environment is deployed, and which can host any number of applications, as-a-service (aaS) products (e.g., software as a service (SaaS), infrastructure as a service (IaaS), platform as a service (PaaS), storage as a service (STaaS), etc.). In one or more embodiments, as used herein, a computing device can be any single computing device, a collection of computing devices, a portion of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. In one or more embodiments, a computing device is any device, part of a device, or any collection of devices capable of electronically processing instructions, and may include, but is not limited to, any of the following: one or more processors (e.g., components comprising circuitry), memory (e.g., random access memory (RAM)), (multiple) input devices and output devices, non-volatile storage hardware (e.g., solid-state drive (SSD), persistent memory (Pmem) device, hard disk drive (HDD) (not shown)), one or more physical interfaces (e.g., network ports, storage ports), any number of other hardware components (not shown), and / or any combination thereof.

[0021] Examples of computing devices include, but are not limited to, servers (e.g., blade servers in blade server chassis, rack servers in racks, etc.), desktop computers, mobile devices (e.g., laptop computers, smart phones, personal digital assistants, tablet computers, automotive computing systems, and / or any other mobile computing devices), storage devices (e.g., disk drive arrays, Fibre Channel storage devices, Internet Small Computer Systems Interface (iSCSI) storage devices, tape storage devices, flash arrays, network attached storage devices, etc.), network devices (e.g., switches, routers, multilayer switches, etc.), virtual machines, virtualized computing environments, logical containers (e.g., for one or more applications), Internet of Things (IoT) devices, arrays of computing resource nodes, supercomputing devices, data centers or any portion thereof, and / or any other type of computing device having the foregoing requirements. In one or more embodiments, any or all of the foregoing examples may be combined to create a system of such devices, or may be segmented into separate logical devices, which may be collectively referred to as computing devices. For example, other types of computing devices may be used without departing from the scope of the embodiments described herein, such as Figure 4 Computing devices shown and described below. Cloud infrastructure 100 may include any number and / or type of these computing devices in any arrangement and / or configuration without departing from the scope of the embodiments disclosed herein.

[0022] In one or more embodiments, the storage device and / or memory of a computing device or computing device system may be and / or include one or more data repositories for storing any number of data structures storing any number of data (e.g., information). In one or more embodiments, a data repository is any type of storage unit and / or device (e.g., a file system, a database, a collection of tables, RAM, and / or any other storage mechanism or medium) for storing data. In addition, a data repository may include multiple different storage units and / or devices. Multiple different storage units and / or devices may or may not be of the same type or located in the same physical location.

[0023] In one or more embodiments, any storage and / or memory of a computing device or computing device system may be considered, in whole or in part, to be a non-transitory computer-readable medium that stores software and / or firmware.

[0024] Such software and / or firmware may include instructions that, when executed by one or more processors (not shown) and / or other hardware (e.g., circuits) of a computing device and / or computing device system, cause the one or more processors and / or other hardware components to perform operations according to one or more embodiments described herein.

[0025] The software instructions may be in the form of computer-readable program code to perform the methods, processes, etc. of the embodiments as described herein, and may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium, such as a compact disc (CD), a digital versatile disc (DVD), a storage device, a disk, a tape storage, a flash memory, a physical memory, or any other non-transitory computer-readable medium, as examples.

[0026] Cloud infrastructure 100 may include any number of computing devices, which may be collectively referred to as computing devices as used herein. All or any portion of the computing devices may be computing devices of the same type or of different types.

[0027] In one or more embodiments, for example, all or any portion of the cloud infrastructure 100 may be managed at least in part by a container platform (not shown) such as Kubernetes. In one or more embodiments, the container platform includes a control plane (not shown). The control plane of the container platform may include various components for managing the cloud infrastructure 100 (e.g., one or more API servers, controllers, schedulers, etc.). For example, the control plane may be implemented on one or more master nodes, each of which may be a computing device (as described above).

[0028] In one or more embodiments, the cloud infrastructure 100 includes an API server (not shown). In one or more embodiments, the API server is a computing device of the cloud infrastructure 100 and runs in the control plane of the container platform. In one or more embodiments, the API server is configured to interact with entities that seek to configure various aspects of the container ecosystem and / or otherwise interact with various aspects of the container ecosystem (e.g., the testing framework 116 described below). As an example, the API server can be configured to receive commands (e.g., to configure containers, deploy container instances, terminate containers, configure storage for containers, configure networking for containers, close ports, restart pods, disconnect various components from other components, cause crashes, restart components, etc.).

[0029] In one or more embodiments, the cloud infrastructure 100 can host any number of applications that can be implemented using a microservices architecture, which includes any number of microservices (e.g., microservice A 102, microservice B 104, microservice C 108, microservice N 110), as well as various other microservices (e.g., user interface microservice 106, any number of public microservices 114) and various communication components that allow the various microservices to interact with other microservices, other parts of the cloud infrastructure 100, and / or entities and / or devices external to the cloud infrastructure 100.

[0030] In one or more embodiments, microservices (e.g., 102, 104, 106, 108, 110) may be implemented using any number of containers, container pods, etc. Each microservice (e.g., 102, 104, 106, 108, 110) may implement a specific service component that constitutes part of an overall application deployed in a cloud infrastructure.

[0031] In one or more embodiments, user interface microservice 106 may be a microservice that provides any type of user interface (eg, graphical, command line, etc.) to users of applications deployed as microservices of a microservices architecture.

[0032] In one or more embodiments, the public microservice 114 may be a collection of services common to all or any portion of other microservices (e.g., 102, 104, 106, 108, 110) of an application deployed using a microservice architecture in the cloud infrastructure 100. Inspectors of the public microservices include, but are not limited to, database services, authorization services, workflow orchestration microservices, auditing microservices, management microservices, and the like.

[0033] In one or more embodiments, the cloud infrastructure 100 includes a communication component 112. In one or more embodiments, as used herein, the communication component 112 includes physical devices and / or logical devices and / or physical components and / or logical components that implement any number of communication protocols that allow microservices to communicate. As an example, various communication protocols may include, but are not limited to, a representative state transfer (REST) ​​protocol, a remote procedure call (RPC) framework (e.g., gRPC), a Kafka protocol, and the like, as well as other underlying protocols that may be used in communication, such as a hypertext transfer protocol (HTTP), a transmission control protocol (TCP), a user datagram protocol (UDP), and the like. These protocols may be used to transmit data, commands, and the like.

[0034] although Figure 1A particular set of microservices arranged in a particular manner is shown, but the cloud infrastructure 100 may include any microservice architecture with any number of microservices arranged and configured in any manner without departing from the scope of the embodiments disclosed herein. These microservice architectures can be used to implement any type of application, any number of applications, and / or any other service (e.g., aaS products). In addition, all or any part of any of the aforementioned devices, components, etc. (e.g., computing devices, servers, network devices, storage devices, container platforms, containers, container pods, microservices, virtual machines, etc.) in any combination may be referred to herein as one or more components of the cloud infrastructure.

[0035] In one or more embodiments, Figure 1 The illustrated system includes a test framework 116 operatively connected to the cloud infrastructure 100. In one or more embodiments, the test framework 116 is implemented using any number of computing devices (as described above). In one or more embodiments, the test framework 116 includes a collection of components that can be used to perform tests on the microservice architecture in the cloud infrastructure 100, monitor the results of the tests, and provide the results to relevant entities conducting the tests.

[0036] In one or more embodiments, the test framework 116 includes a framework controller 118. The framework controller 118 can be any hardware, software, firmware, or any combination thereof that is configured to receive input from a user related to test components of the cloud infrastructure 100 and provide information related to such input to other components of the test framework 116. As an example, the framework controller 118 can be implemented using one or more computing devices (as described above). In one or more embodiments, the framework controller 118 is a component configured to receive input from a user (e.g., a test engineer). Such input can include a definition of what the user wants to test and the expected results of performing the test. In one or more embodiments, the framework controller provides details related to what the user expects to test to the framework injector 122 and provides the expected results to the framework monitor 120.

[0037] The type of input requested by the user via the framework controller 118 may depend on the type of testing techniques that the user desires to use to perform testing of the microservice architecture and cloud infrastructure. The expected outcome of the testing may depend on the type of testing specified by the user. Thus, various categories of testing techniques, different types of errors, crashes, freezes, failures, etc., may be presented to a user interacting with the framework controller 118, and these all fall within each category. In one or more embodiments, the user may select or otherwise specify the type of testing techniques, the various actions that the user desires to perform using these testing techniques, and the expected outcomes or responses of performing these actions.

[0038] In one or more embodiments, the test framework 116 includes a framework injector 122. The framework injector 122 can be any hardware, software, firmware, or any combination thereof configured to receive information based on user input from the framework controller 118 and use the information to perform any number of tests by interacting with various components of the cloud infrastructure 100. The framework injector 122 can include various components for performing such tests, including an interrupt injector, a communication error injector, and a crash point injector (which can be collectively referred to as a test injector), which will be described below. Figure 2 Each of them is further described in the description of .

[0039] In one or more embodiments, the test framework 116 includes a framework monitor 120. The framework monitor can be any hardware, software, firmware, or any combination thereof configured to monitor the cloud infrastructure while the framework injector 122 executes the test to observe the results, effects, etc. of the tests on the various components of the cloud infrastructure 100 and record the results and provide them to the user who initiated the test.

[0040] In one or more embodiments, the framework monitor 120 observes the cloud infrastructure 100 to which outages, errors, and / or crash points are introduced (e.g., injected by the framework injector 122) to determine whether the cloud infrastructure 100 responds to the outages, errors, and / or crash points as expected. As described above, in one or more embodiments, the expected outcome of a particular test is provided by a user via the framework controller 118. In one or more embodiments, the framework controller 118 provides the expected outcome to the framework monitor 120. In one or more embodiments, the framework monitor 120 is configured to monitor the cloud infrastructure 100 and the components therein to determine whether the actual response to the outages, errors, and / or crash points matches the expected outcome provided by the user.

[0041] As an example, a user may expect that a crash, freeze, reboot, etc. of a hardware component will result in a failover to a redundant component, that the component will recover within a certain amount of time, that the rest of the environment will continue to operate while the interrupted component recovers, and so on. As another example, a user may specify the expected behavior of an application implemented using microservices when one or more microservices crash. The user may specify other expected behaviors without departing from the scope of the embodiments disclosed herein. In one or more embodiments, the framework monitor may provide an alert to the user if the results of the interruption, error, and / or crash point do not match the expected outcome provided by the user. Such an alert may take any form and may include any number and / or type of information. As an example, a user may receive a notification indicating that the expected outcome did not occur, as well as various details about the response of the cloud infrastructure 100 observed by the framework monitor 120.

[0042] In one or more embodiments, the framework controller 118 , the framework monitor 120 , and the framework injector 122 of the test framework 116 may be implemented as part of a single computing device, or may be implemented as any number of separate and operably connected computing devices.

[0043] Although Figure 1 A particular configuration of components is shown, but other configurations may be used without departing from the scope of the embodiments described herein. Figure 1 Some components are shown as part of the same device, but any component may be grouped in a collection of one or more components that may exist and function as part of any number of separate and operably connected devices. As another example, a single component may be configured to perform operations performed by Figure 1 All or any part of the functions performed by all or any part of the components shown. Accordingly, the embodiments disclosed herein should not be limited to Figure 1 Component configuration shown.

[0044] Figure 2 2 illustrates a block diagram of an example framework injector 200 according to one or more embodiments disclosed herein. Figure 2 As shown, framework injector 200 includes interrupt injector 202, communication error injector 204, and crash point injector 206. Each of these components is described below.

[0045] In one or more embodiments, the framework injector 200 and Figure 1 The framework injector 200 may be the same or substantially similar to the framework injector shown and described above. Thus, the framework injector 200 may be a component of a test framework (e.g., Figure 1 The framework injector 200 may be operably connected to a framework controller (e.g., Figure 1 Frame controller 118 as shown and described above), frame monitor (e.g., Figure 1 The framework monitor 120 as shown and described above) and the cloud infrastructure (e.g., Figure 1 The framework injector 200 may be any hardware, software, firmware, or any combination thereof configured to perform the functions described herein. In one or more embodiments, the framework injector 200 is implemented using any number of computing devices (as described above).

[0046] In one or more embodiments, the framework injector 200 includes an interrupt injector 202. The interrupt injector 202 can be any hardware, software, firmware, or any combination thereof configured to inject an interrupt into any component, element, portion, etc. of the cloud infrastructure. In one or more embodiments, the interrupt injector 202 is configured to implement a method for providing a user with a method for causing an environment under test (e.g., Figure 1 The present invention is a testing technology for the ability to introduce crashes, freezes, restarts, etc. (e.g., interruptions) of various components of the cloud infrastructure 100. In one or more embodiments, crashes, freezes, restarts, etc. may be caused on hardware devices (e.g., servers, network devices, storage devices, etc.), platforms (e.g., Kubernetes platforms, container orchestration services, container management systems, virtual machine platforms, etc.), system parts hosting microservices (e.g., containers, container pods, virtual machines, etc.), one or more microservices, authorization services, databases, workflow orchestration and management services, etc.

[0047] In one or more embodiments, depending on what part of the microservice architecture and / or cloud infrastructure will have an outage (such as a crash, freeze, failure, etc.), the scope of the outage can be controlled. As an example, an outage can be a single container, a container pod, a container platform, a specific microservice of an application, a microservice common to various other parts of the system, and / or the entire container platform. In one or more embodiments, the user can control the scope of the outage via a framework controller (e.g., Figure 1 The interrupt specified by the framework controller 118) is provided to the framework injector 200.

[0048] In one or more embodiments, the framework injector 200 is configured to perform actions to cause a desired interruption to a component, device, part, etc. of the cloud infrastructure (e.g., via the interruption injector 202). In one or more embodiments, for example, the actions are performed using predefined commands such as Kubernetes commands, various application programming interface (API) commands, etc., which will depend on the part of the cloud infrastructure to be interrupted. As an example: Kubernetes commands can be used to crash or cause the following to freeze: all or any part of the container platform (e.g., causing a restart of a port); any component therein; any microservice implemented thereon, etc. As another example, various API commands can be used to cause a crash, freeze, restart, etc. of a server, a network device, a storage device, other hardware components, a data backup orchestrator, etc.

[0049] In one or more embodiments, the framework injector 200 includes a communication error injector 204. In one or more embodiments, the communication error injector 204 is any hardware, software, firmware, or any combination thereof configured to induce errors in the cloud infrastructure when communication operations are ongoing. As an example, the various communication protocols may include, but are not limited to, the REST protocol, the RPC framework (e.g., gRPC), the Kafka protocol, and the like, as well as other underlying protocols that may be used in communication, such as HTTP, TCP, UDP, and the like. For example, these protocols may be used to transmit data or any other information (e.g., commands). For example, these protocols may have associated APIs that allow injection of various errors (such as delay injection, timeout errors, and the like). Communication error techniques may also include inducing other errors, such as resource errors (inducing memory errors, inducing processor errors, and the like), server busy errors (e.g., by inducing heavy loads on the computing resources of one or more servers), compliance errors (e.g., causing the protocol to run in a manner that does not conform to expected behavior), and validation errors (e.g., causing errors in packet ordering). In one or more embodiments, any of the aforementioned communication errors may be detected by a user via a framework controller (e.g., Figure 1 118) and is injected into the cloud infrastructure by the framework injector 200. In one or more embodiments, as described above, the framework monitor (e.g., Figure 1 The framework monitor 120) can monitor the environment to determine whether the behavior observed in the cloud infrastructure matches the expected outcomes specified by the user, and if not, an alert can be provided to the user.

[0050] In one or more embodiments, the framework injector 200 includes a crash point injector 206. In one or more embodiments, the crash point injector 206 is any hardware, software, firmware, or any combination thereof configured to add crash points to code that can run in a cloud infrastructure environment. In one or more embodiments, crash point injection includes strategically adding errors, faults, and the like in microservice code, application code, and the like to intentionally cause failures at specific points. In one or more embodiments, injecting failure points into microservices or other code allows for verification of the validity of mechanisms in microservices (such as undo operations, redo operations, retry operations, and the like).

[0051] As an example, certain operations may be performed by a microservice in various phases, and crash points may be inserted into the microservice code that cause errors at specific points within these phases in order to verify that the microservice responds in an expected manner. For example, a microservice may be configured to perform an operation to restore from a previously created data backup. The restore operation may have three phases: restoration of storage (e.g., from a snapshot), attaching the restored storage to a compute instance (e.g., an Amazon Elastic Cloud Computing (EC2) instance), and providing updated information about the storage attached to the instance to a manager service.

[0052] In one or more embodiments, in this scenario, a crash point can be injected into any point in any of the three stages to cause a failure. In one or more embodiments, a user can inject a crash point into any of the three stages via a framework controller (e.g., Figure 1 The framework controller 118 of the microservice specifies the stage at which the user expects the error to occur and the expected outcome. Continuing with the previous example, the user can provide an expected outcome, i.e., if an error occurs in the second stage of the process being executed by the microservice, the microservice will retry the second stage after restarting (if the error is injected after a certain point), but will not redo the completed first stage, and then continue with the third stage. Based on the input from the user, the framework controller (e.g., Figure 1 The framework controller 118 of the framework monitor 118 can provide information to the framework injector 200, which can then add one or more appropriate failure points to the code of the relevant microservice so that the microservice will reach the failure point during operation. In one or more embodiments, the framework monitor (e.g., Figure 1 The framework monitor 120) observes the behavior of the microservice, determines whether the behavior matches the expected results provided by the user, and warns the user if the observed behavior does not match the expected behavior.

[0053] Although Figure 2 A particular configuration of components is shown, but other configurations may be used without departing from the scope of the embodiments described herein. Figure 2 Some components are shown as part of the same device, but any component may be grouped in a collection of one or more components that may exist and function as part of any number of separate and operably connected devices. As another example, a single component may be configured to perform operations performed by Figure 2 All or any part of the functions performed by all or any part of the components shown. Accordingly, the embodiments disclosed herein should not be limited to Figure 2 Component configuration shown.

[0054] Figure 3An overview of an example method for testing a cloud infrastructure according to one or more embodiments disclosed herein is illustrated. The method may be at least partially implemented by a testing framework (e.g., Figure 1 The test framework 116 shown) or any component of the test framework (e.g., Figure 1 The frame controller 118, Figure 1 Framework Monitor 120, Figure 1 Framework injector 122, Figure 2 Framework injector 200, Figure 2 The interrupt injector 202, Figure 2 Communication error injector 204, Figure 2 Crash point injector) to execute.

[0055] Although Figure 3 The various steps in the flowcharts shown are presented and described sequentially, but some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, and some or all of the steps may be combined or omitted. Figure 3 The other steps are run in parallel.

[0056] In step 300, the method includes receiving a test execution request including test parameters and expected test results. As an example, the test execution request can be sent by a test framework (e.g., Figure 1 The test framework 116) of the framework controller (e.g., Figure 1 The test parameters are received from the user by the framework controller 118 shown. In one or more embodiments, the test parameters are any information related to which one or more tests or test scenarios the user desires to execute in the cloud infrastructure.

[0057] As an example, a user may want to test a Figure 1 The microservices architecture implemented in the cloud infrastructure 100 of the framework. The test parameters may depend on the one or more tests to be performed. In one or more embodiments, the framework controller includes a user interface that presents to the user options for different testing techniques that the test framework can perform. As an example, the framework controller may present to the user or otherwise allow the user to input, select, etc., one or more testing techniques (e.g., interrupt injection, communication error injection, crash point injection, etc.). In one or more embodiments, the framework controller may further guide or otherwise receive input from the user based on the type of test to be performed.

[0058] As an example, if a user indicates that an interrupt injection test is to be performed, the user may provide as test parameters an identification of what components (e.g., servers, network devices, storage devices, specific subcomponents, virtual machines, containers, container pods, microservices, applications, container platforms, etc.) will be interrupted by the test and how these components will be interrupted by the test (e.g., crashes, freezes, failures, etc.). The test parameters may also include information related to the expected scope of impact of the interruption (e.g., a single container, a single container pod, an entire microservice, an entire application, an entire container platform, a specified percentage or portion of one or more types of devices in a cloud infrastructure, etc.).

[0059] As another example, if a user indicates that a communication error injection test is desired to be performed, the framework controller can direct or otherwise receive information from the user regarding one or more specific communication protocols (e.g., gRPC, Kafka, etc.) into which errors are to be injected, the length of the communication delay to be injected, the devices and / or components into which errors are to be caused (e.g., memory errors, server busy errors due to overload, etc.), compliance errors (e.g., causing behavior that does not comply with the protocol specification), validation errors (e.g., causing intended packet reordering on a particular data path or paths), and the like.

[0060] As another example, if a user indicates that a crash point injection test is desired, the framework controller may direct or otherwise receive details from the user about what code, process, phase, etc. of an operation performed in the cloud infrastructure into which the user desires to inject a crash point or failure point. The user may specify generally (e.g., inject into a crash point in phase two of a backup and restore process), or may specify more specifically (e.g., inject into a crash point in the microservice code of a particular microservice when conditions in the cloud environment meet certain specified parameters).

[0061] In one or more embodiments, the expected test outcome is what the user expects to happen when running one or more tests in the cloud infrastructure corresponding to the aforementioned test parameters. The expected test outcome can be any number of information at any level of detail related to the response of all or any part of the cloud infrastructure to any number of tests executed by the test framework. Examples include, but are not limited to, expected timely failover and service recovery, retries within at least a specific time period before the failure, and expected services or applications to continue to operate after the failure of one or more microservices, containers, pods, etc.

[0062] In step 302, the method includes providing a test execution request and test parameters to a framework injector. As an example, the test execution request and test parameters may be provided by a framework controller (e.g., Figure 1 The frame controller 118) provides Figure 1 Framework injector 122 or Figure 2 The framework injector 200 of the present invention. The test execution request and test parameters may be provided to the framework injector using any technique for communicating information, such as passing inter-process messages, sending one or more packets over a network, etc. The test execution request and test parameters may match what the user provides to the framework controller, or may have been altered in any way by the framework controller into a form expected by the framework injector.

[0063] In step 304, the method includes providing the expected test results to the framework monitor. As an example, the framework controller (e.g., Figure 1 The frame controller 118 of the embodiment may send a signal to the frame monitor (e.g., Figure 1 The expected test results may be provided to the framework injector using any technique for communicating information, such as passing inter-process messages, sending one or more packets over a network, etc. The expected test results may match what the user provides to the framework controller, or may have been modified in any way by the framework controller into a form expected by the framework injector.

[0064] In step 306, the method includes running a test corresponding to the test execution request using one of a plurality of testing techniques implemented by the framework injector and based at least in part on the test parameters. As an example, the framework injector (e.g., Figure 2 The framework injector 200 of the present invention may determine whether the test will use an interrupt injector (e.g., Figure 2 interrupt injector 202), a communication error injector (e.g., Figure 2 communication error injector 204), a crash point injector (e.g., Figure 2 In one or more embodiments, running the test includes configuring the associated one or more of the aforementioned injectors using the test parameters to initiate, execute, or otherwise conduct the test requested by the test execution request. Running the test may also include transmitting information about the test to the framework monitor (e.g., what test is being run, when the test started or has started, the cloud infrastructure part in which the test will occur, etc.).

[0065] In step 308, the method includes monitoring the cloud infrastructure's response to the test performed in step 306. As an example, the cloud infrastructure may be monitored by a framework monitor (e.g., Figure 1The framework monitor 120) performs monitoring while the framework injector executes the test. In one or more embodiments, monitoring the cloud infrastructure includes observing all or any part of the cloud infrastructure using any appropriate technology for monitoring computing devices, software running thereon, etc. As an example, monitoring may include, but is not limited to: observing network traffic patterns; tracking the status of components, devices, applications, containers, pods, virtual machines, etc.; monitoring the results of operations (e.g., backup operations, restore operations, failover operations, retry operations, recovery operations, etc.). In one or more embodiments, monitoring may include recording or otherwise generating or saving information related to the behavior of the cloud infrastructure observed during the execution of the test.

[0066] In step 310, the method includes providing results including the behavior of the cloud infrastructure in response to the test. As an example, the results can be provided from the framework monitor to the framework controller for presentation to the user who requested the test. In one or more embodiments, the results include whether the behavior observed in the cloud infrastructure during the test matches the expected test results provided by the user. The results can be in any suitable form for conveying information. For example, the results can be formatted into a report that details the results of the test, the behavior of various parts of the cloud infrastructure during the test, and whether the behavior matches the expected test results.

[0067] Figure 4 A block diagram of a computing device according to one or more embodiments of the present disclosure is illustrated. As described above, the embodiments described herein may be implemented using a computing device. For example, Figure 1 All or any part of the components shown may be implemented at least in part using one or more computing devices. Computing device 400 may include one or more computer processors 402, non-permanent storage 404 (e.g., volatile memory, such as random access memory (RAM), cache memory, etc.), permanent storage 406 (e.g., hard disk, optical drive (such as compact disk (CD) drive or digital versatile disk (DVD) drive), flash memory, etc.), communication interface 412 (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), input device 410, output device 408, and many other elements (not shown) and functions. Each of these components is described below.

[0068] In one or more embodiments, computer processor(s) 402 may be integrated circuits for processing instructions. For example, computer processor(s) may be one or more cores or microkernels of a processor. Processor 402 may be a general purpose processor configured to run program code included in software running on computing device 400. Processor 402 may be a special purpose processor in which certain instructions are incorporated into the processor design. Although Figure 3 Only one processor 402 is shown in FIG. 4 , but computing device 400 may include any number of processors without departing from the scope of the embodiments disclosed herein.

[0069] The computing device 400 may also include one or more input devices 410, such as a touch screen, a keyboard, a mouse, a microphone, a touchpad, an electronic pen, a motion sensor, or any other type of input device. The input device 410 may allow a user to interact with the computing device 400. In one or more embodiments, the computing device 400 may include one or more output devices 408, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, an external storage device, or any other output device. One or more output devices may be the same or different from the (multiple) input devices. The (multiple) input devices and output devices may be locally or remotely connected to the (multiple) computer processors 402, the non-permanent storage device 404, and the permanent storage device 406. There are many different types of computing devices, and the aforementioned (multiple) input devices and output devices may take other forms. In some cases, a multimodal system may allow a user to provide multiple types of input / output to communicate with the computing device 400.

[0070] In addition, the communication interface 412 can facilitate connecting the computing device 400 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) (such as the Internet), a mobile network, or any other type of network) and / or another device (such as another computing device). The communication interface 412 can use a wired and / or wireless transceiver to perform or facilitate the reception and / or transmission of wired or wireless communications, including reception and / or transmission using the following: an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Wireless signal transmission, BLE wireless signal transmission, Wireless signal transmission, RFID wireless signal transmission, near-field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11WiFi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signal transmission, Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, self-organizing network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or a combination thereof. The communication interface 412 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing device 400 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, GPS based in the United States, Global Navigation Satellite System (GLONASS) based in Russia, BeiDou Navigation Satellite System (BDS) based in China, and Galileo GNSS based in Europe. There is no restriction on operating on any particular hardware configuration, and therefore, as hardware or firmware arrangements develop, the basic features here can be easily replaced with improved hardware or firmware arrangements.

[0071] The term computer-readable medium includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying (multiple) instructions and / or data. Computer-readable media may include non-transient media in which data can be stored, and does not include carrier waves and / or temporary electronic signals that are propagated wirelessly or through wired connections. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as CDs or DVDs), flash memory, memory, or memory devices. Computer-readable media may store codes and / or machine-executable instructions thereon, and codes and / or machine-executable instructions may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software groupings, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent via any suitable means (including memory sharing, message passing, token passing, network transmission, etc.).

[0072] All or any portion of the components of computing device 400 may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals including bitstreams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0073] In the above description, many details are set forth as examples of the embodiments described herein. Those skilled in the art (who also benefit from this disclosure) will appreciate that one or more embodiments described herein may be practiced without these specific details, and that many variations or modifications are possible without departing from the scope of the embodiments described herein. Certain details known to those of ordinary skill in the art may be omitted to avoid obscuring the description.

[0074] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be appreciated by those of ordinary skill in the art that these aspects can be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including functional blocks, which may include devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, additional components may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams to avoid obscuring these aspects with unnecessary details. In other cases, in order to avoid obscuring these aspects, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details.

[0075] Various aspects may be described above as a process or method shown in a flow chart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flow chart may describe an operation as a sequential process, many operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process is terminated when the operation of the process is completed, but may have additional steps not included in the figure. A process may correspond to a method, function, program, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0076] The processes and methods according to the above examples can be implemented using computer executable instructions stored or otherwise available from a computer readable medium. These instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. The computer resources used may be partially accessible via a network. The computer executable instructions may be, for example, binary intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information, and / or information created during the methods according to the examples include disks or optical disks, flash memory, USB devices equipped with non-volatile memory, networked storage devices, etc.

[0077] In the above description of the drawings, in the various embodiments described herein, any component described with respect to a drawing may be equivalent to one or more identically or similarly named and / or numbered components described with respect to any other drawing. For the sake of brevity, the description of these components may not be repeated with respect to each drawing. Therefore, each embodiment of the components of each drawing is incorporated by reference and is assumed to be optionally presented in each other drawing with one or more identically or similarly named and / or numbered components. In addition, according to the various embodiments described herein, any description of the components of the drawings should be interpreted as an optional embodiment, which can be implemented in addition to, in combination with, or in place of the embodiments described with respect to the corresponding one or more identically or similarly named and / or numbered components in any other drawing.

[0078] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in the application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to just a single element, unless explicitly disclosed, such as by the use of the terms "before," "after," "single," and other such terms. Instead, the use of ordinal numbers is to distinguish between elements. For example, a first element is different from a second element, and a first element may encompass more than one element and be located after (or before) a second element in the ordering of elements.

[0079] As used herein, the phrase "operably connected" or "operably connected" means that there is a direct or indirect connection between elements / components / devices that allows the elements to interact with each other in some manner. For example, the phrase "operably connected" can refer to any direct connection (e.g., a wired connection directly between two devices or components) or indirect connection (e.g., a wired and / or wireless connection between any number of devices or components that connect operably connected devices). Thus, any path that information can take can be considered an operative connection.

[0080] Although the embodiments discussed herein have been described with respect to a limited number of embodiments, those skilled in the art having benefit of this disclosure will appreciate that other embodiments may be designed without departing from the scope of the embodiments disclosed herein. Accordingly, the scope of the embodiments described herein should be limited only by the appended claims.

Claims

1. A test framework, comprising: one or more processors; a framework controller executing on the one or more processors; a framework monitor executing on the one or more processors; a framework injector executing on the one or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving, at the framework controller, a test execution request including test parameters and expected test outcomes, wherein the test execution request includes a request for testing a microservices architecture implemented in a cloud infrastructure; The framework controller provides the test execution request and the test parameters to the framework injector; Providing the expected test results to the frame monitor by the frame controller; executing, by the framework injector, a test injector from among a plurality of test injectors implemented by the framework injector, a test corresponding to the test execution request; Monitoring, by the framework monitor, a response of the microservice architecture of the cloud infrastructure to the test; as well as A test result based on the response is provided by the frame controller from the frame monitor.

2. The test framework of claim 1, wherein the test injector comprises an interrupt injector.

3. The testing framework of claim 2, wherein the testing is implemented using the interruption injector and comprises inducing an interruption in one or more components of the cloud infrastructure. The testing framework of claim 1 , wherein the test injector comprises a communication error injector. 5 . The testing framework of claim 4 , wherein the testing is performed using the communication error injector and comprises injecting communication errors into the cloud infrastructure. The testing framework of claim 1 , wherein the test injector comprises a crash point injector.

7. The testing framework of claim 6, wherein the testing is implemented using a crash point injector and comprises inserting crash points into code corresponding to one or more components of the cloud infrastructure.

8. The testing framework of claim 1, wherein the test result comprises an indication of whether the response matches the expected test outcome.

9. A computer-implemented method comprising: receiving, at a framework controller executing on one or more processors, a test execution request including test parameters and expected test outcomes, wherein the test execution request includes a request to test a microservices architecture implemented in a cloud infrastructure; providing, by the framework controller, the test execution request and the test parameters to a framework injector executing on the one or more processors; providing, by the framework controller, the expected test outcome to a framework monitor executing on the one or more processors; executing, by the framework injector, a test injector from among a plurality of test injectors implemented by the framework injector, a test corresponding to the test execution request; Monitoring, by the framework monitor, a response of the microservice architecture of the cloud infrastructure to the test; as well as A test result based on the response is provided by the framework controller from the framework monitoring provider.

10. The computer implemented method of claim 9, wherein the test injector comprises an interrupt injector.

11. The computer-implemented method of claim 10, wherein the testing is performed using the interruption injector and comprises inducing an interruption in one or more components of the cloud infrastructure.

12. The computer implemented method of claim 9, wherein the test injector comprises a communication error injector.

13. The computer-implemented method of claim 12, wherein the testing is performed using the communication error injector and comprises injecting communication errors into the cloud infrastructure.

14. The computer-implemented method of claim 9, wherein the test injector comprises a crash point injector.

15. The computer-implemented method of claim 14, wherein the testing is performed using the crash point injector and comprises inserting crash points into code corresponding to one or more components of the cloud infrastructure.

16. The computer-implemented method of claim 9, wherein the test result comprises an indication of whether the response matches the expected test outcome.

17. A non-transitory computer-readable medium storing a program for execution by one or more processors, the program comprising instructions for: receiving, at a framework controller, a test execution request including test parameters and expected test outcomes, wherein the test execution request includes a request for testing a microservices architecture implemented in a cloud infrastructure; The framework controller provides the test execution request and the test parameters to the framework injector; The frame controller provides the expected test result to the frame monitor; executing, by the framework injector, a test injector from among a plurality of test injectors implemented by the framework injector, a test corresponding to the test execution request; Monitoring, by the framework monitor, a response of the microservice architecture of the cloud infrastructure to the test; as well as A test result based on the response is provided by the frame controller from the frame monitor.

18. The non-transitory computer readable medium of claim 17, wherein: The test injector comprises an interrupt injector, and The testing is performed using the interruption injector and includes inducing an interruption in one or more components of the cloud infrastructure.

19. The non-transitory computer readable medium of claim 17, wherein: The test injector comprises a communication error injector, and The testing is performed using the communication error injector and includes injecting communication errors into the cloud infrastructure.

20. The non-transitory computer readable medium of claim 17, wherein: The test injector includes a crash point injector, and The testing is performed using the crash point injector and includes inserting crash points into code corresponding to one or more components of the cloud infrastructure.

Images