TESTING A CLOUD INFRASTRUCTURE

Abstract

In certain embodiments, a method comprises receiving a test execution request with test parameters and an expected test result at a frame controller, wherein the test execution request includes a request to test a microservice architecture implemented in a cloud infrastructure; the frame controller providing the test execution request and the test parameters to a frame injector running on one or more processors; the frame controller providing the expected test result to a frame monitor running on one or more processors; and the frame injector executing a test that complies with the test execution request using a test injector from a plurality of test injectors implemented by the frame injector.Monitoring the response of the cloud infrastructure's microservice architecture to the test by the frame monitor; and providing a test result based on the response by the frame controller and the frame monitor.

Description

BACKGROUND

[0001] Information technology infrastructures and ecosystems are becoming increasingly complex. As this complexity increases, it can become challenging to comprehensively test such environments as a whole and determine the impact of different parts of the environment on others when failures occur. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain embodiments discussed herein are described with reference to the accompanying drawings below. However, the accompanying drawings only illustrate certain aspects or embodiments of the embodiments described herein by way of example and are not intended to limit the scope of the claims. Fig. 1 shows a block diagram of an example system for implementing a test framework for testing cloud infrastructures in accordance with one or more embodiments disclosed herein; Fig. 2 shows a block diagram of an exemplary frame injector 200 according to one or more embodiments disclosed herein; Fig. 3 shows an overview of an example method for testing a cloud infrastructure according to one or more embodiments disclosed herein; and Fig. 4 shows a block diagram of a computing device according to one or more embodiments of this disclosure. DETAILED DESCRIPTION

[0003] Companies that provide technology and technology services are moving to models that include microservices and other distributed technologies that can be implemented in cloud infrastructure environments. With the adoption of such models, the complexity of the systems used to implement such technologies can increase. This complexity can make it difficult to predict, determine the causes, and / or resolve system outages, which can be caused by failures at any number of points in the system or by a combination of such failures. For example, unexpected behavior in a complex system, sudden failures of one or more hardware or software components, unexpected delays in response times, and other problems can lead to system outages during which a company is unable to provide a service to customers, resulting in customer dissatisfaction.

[0004] For example, an online application or service may be implemented using any number of microservices in a cloud computing infrastructure environment, where the hardware environment is distributed (e.g., in a public cloud, a private cloud, a hybrid cloud, etc.). Such microservices may be implemented, at least in part, using technologies such as container platforms (e.g., Kubernetes), virtualized instances of computing devices (e.g., virtual machines), etc. In such environments, it may be difficult to fully test the environment's ability to handle or otherwise respond to various failures, errors, outages, etc., because such problems occurring in the environment may have unpredictable effects on all or part of the rest of the environment. Such problems include, but are not limited to, software crashes and / or errors (e.g.,of container platforms, microservices, containers, pods, virtual machines, operating systems, backup and recovery software, databases, shared services, communication protocols, applications, processes, etc.), crashes, panics, and / or other errors in hardware components (e.g., servers, other computing devices, network devices, storage devices, etc.) and / or errors and / or failures in communication between components. Various components or parts of distributed computing devices for implementing microservice-based architectures may be tested, but the system as a whole may not be fully tested and / or not tested in a production-like environment, which could lead to an unforeseen or otherwise unknown number of potential problems when the various components, parts, sections, devices, etc., are assembled, connected, deployed, etc.and begin executing to deliver a service, application, etc. in a cloud infrastructure environment.

[0005] To address at least the above-mentioned problems, the embodiments disclosed herein provide a comprehensive framework that can be used for robust testing of microservice architectures implemented in cloud infrastructure environments. In one or more embodiments, the framework comprises a framework controller, a framework injector, and a framework monitor, and the framework can be operatively connected to a cloud infrastructure environment in which a microservice architecture is implemented (e.g., for running an application).

[0006] In one or more embodiments, a framework controller is a component configured to receive input from a user (e.g., a test engineer). Such input may include a definition of what the user wants to test and the expected result of the test execution. In one or more embodiments, the framework controller provides details about what the user wants to test to the framework injector and provides the expected results to the framework monitor.

[0007] In one or more embodiments, the framework injector may be configured to perform a variety of actions that can be implemented using various testing technologies, including, but not limited to, interrupt technology, communication fault technology, and crash point injection technology. The type of input required from the user via the framework controller may depend on the type of technology the user intends to use to perform tests of the microservice architecture and cloud infrastructure. The expected results of the tests may depend on the test types specified by the user. Thus, a user interacting with the framework controller may be presented with the various categories of testing technologies, the various types of errors, crashes, panics, faults, etc., that fall into each of the categories.In one or more embodiments, the user may select or otherwise specify the type of testing technology, the various actions the user wishes to perform with those technologies, and the expected result or response to performing those actions.

[0008] One testing technology that the Framework Injector can implement is disruption technology. In one or more embodiments, disruption technology is a category of testing techniques that provides the user with the ability to trigger crashes, panics, reboots, etc., of various components of the environment under test. In one or more embodiments, the crashes, panics, reboots, etc., can be caused on hardware devices (e.g., servers, network devices, etc.), platforms (e.g., Kubernetes platforms, container orchestration services, container management systems, virtual machine platforms, etc.), parts of systems hosting microservices (e.g., containers, container pods, virtual machines), one or more microservices, authorization services, databases, workflow orchestration and management services, etc.

[0009] In one or more embodiments, the radius of the disruption can be controlled depending on which part of the microservice architecture and / or cloud infrastructure is experiencing disruptions such as crashes, panics, errors, etc. For example, the disruption can relate to a single container, a container pod, a container platform, a specific microservice of an application, microservices common to a variety of other parts of the system, and / or an entire container platform. In one or more embodiments, the interruptions specified by the user via the framework controller are provided to the framework injector.

[0010] In one or more embodiments, the framework injector is configured to perform actions to cause the desired disruptions. In one or more embodiments, the actions are performed using predefined commands, such as Kubernetes commands, various application programming interface (API) commands, etc., depending on the part of the system that is to be disrupted. For example, Kubernetes commands can be used to crash or panic (e.g., trigger a port restart) all or part of the container platform; all components within it; all microservices deployed on it, etc. As another example, various API commands can be used to trigger crashes, panics, reboots, etc. of servers, network devices, storage devices, other hardware components, data backup orchestrators, etc.

[0011] In one or more embodiments, the framework monitor observes the environment into which the disturbances are introduced to determine whether the environment responds to the disturbance as expected. As previously mentioned, in one or more embodiments, the expected outcome of certain tests is provided by the user via the framework controller. In one or more embodiments, the framework controller provides the expected results to the framework monitor. In one or more embodiments, the framework monitor is configured to monitor the environment and the components therein to determine whether the actual response to the disturbance matches the user-specified expectation. For example, the user may expect a crash, panic, reboot, etc.a hardware component results in a failover to a redundant component, that the component recovers within a certain period of time, that other parts of the environment continue to operate while the failed component recovers, etc. As another example, the user may specify the expected behavior of an application implemented with microservices when one or more of the microservices fails. Other expected behaviors may be specified by the user without exceeding the scope of the embodiments disclosed herein. In one or more embodiments, the framework monitor may issue a warning to the user if the result of the failure does not match the result expected by the user. Such a warning may take any form and may contain any amount and / or type of information.For example, the user may receive a notification indicating that the expected result did not occur, as well as various details about the environment's response observed by the framework monitor.

[0012] Another testing technique that the framework injector can be configured for is the communication error technique. In one or more embodiments, the communication error technique involves throwing errors when communication operations are in progress. Examples of various communication protocols include the Representational State Transfer (REST) ​​protocol, RPC frameworks (e.g., gRPC), the Kafka protocol, etc., as well as other underlying protocols that may be used in communication, such as the Hypertext Transfer Protocol (HTTP), the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), etc. Such protocols may be used for communicating data. Such protocols may have associated APIs that allow the injection of various errors, such as delay errors, timeouts, etc.Communication error technologies may also include causing other errors, such as resource errors (causing memory errors, causing processor errors, etc.), server utilization errors (e.g., causing a heavy load on the computing resources of one or more servers), conformance errors (e.g., causing protocols to behave inconsistently with expected behavior), and validation errors (e.g., causing packet ordering errors). In one or more embodiments, any of the above communication errors may be specified by the user via the framework controller and injected into the environment by the framework injector.In one or more embodiments, as described above, the framework monitor may monitor the environment to determine whether the behavior observed in the environment matches the expected results specified by the user, and if it does not, a warning may be issued to the user.

[0013] Another testing technology that the framework injector can implement is crash point injection technology. In one or more embodiments, crash point injection technology involves strategically adding errors, faults, etc., to the microservice code to cause failures at specific points. In one or more embodiments, inserting failure points into the microservice code enables validation of the effectiveness of mechanisms in the microservice, such as undo operations, redo operations, retry operations, etc. For example, certain operations of microservices can be performed in different phases, and failure points can be inserted into the microservice code to cause failures at specific points within those phases to verify that the microservice is responding in the expected manner.For example, a microservice may be configured to perform a restore operation from a previously created backup. The restore operation may involve three stages: restoring the storage (e.g., from a snapshot), attaching the restored storage to a compute instance (e.g., an Amazon Elastic Cloud Compute (EC2) instance), and providing the updated information regarding the storage attached to the instance to a manager service. In one or more embodiments, in such a scenario, a crash point may be injected at any point in any of the three stages to cause an outage. In one or more embodiments, a user specifies, via the framework controller, the stage at which the failure should occur and the expected outcome.Continuing the previous example, the user can specify an expected outcome stating that if an error occurs in the third phase of a process performed by a microservice, the microservice will retry the third phase after restarting, but will not repeat the two previous phases that have been completed. Based on the user's input, the framework controller can pass information to the framework injector, which in turn can insert the appropriate one or more failure points into the code of the microservice in question, so that the microservice will reach the failure point during execution. In one or more embodiments, the framework monitor observes the behavior of the microservice, determines whether the behavior matches the result expected by the user, and warns the user if the observed behavior does not match the expected behavior.

[0014] Certain embodiments of this disclosure may provide a framework that enables real-time testing of microservice architectures implemented in cloud infrastructures to uncover unexpected or otherwise inappropriate responses of various components of the environment when faced with a variety of failure scenarios of varying complexity, which may improve the ability of such environments to be more robust and resilient, thus helping to avoid system failures, downtime, etc.

[0015] Fig. 1 shows a block diagram of an example system for implementing a test framework for testing cloud infrastructures according to one or more embodiments disclosed herein. As in Fig. 1, the system may include a cloud infrastructure 100. The cloud infrastructure 100 may include any number of microservices (e.g., microservice A 102, microservice B 104, microservice C 108, microservice N 110), a user interface microservice 106, one or more shared microservices 114, and communication components 112. The system may also include a test framework 116. The test framework 116 may include a framework controller 118, a framework monitor 120, and a framework injector 122. Each of these components is described below.

[0016] In one or more embodiments, cloud infrastructure 100 is a collection of any number of computing devices and other components in which a cloud computing environment is deployed and which can host any number of applications, as-a-service (aaS) products (e.g., software aaS (SaaS), infrastructure aaS (laaS), platform aaS (PaaS), storage aaS (STaaS), etc.). In one or more embodiments, a computing device, as used herein, can be a single computing device, a set of computing devices, a portion of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. In one or more embodiments, a computing device is any device, portion of a device, or group of devices capable of electronically processing instructions and can include, among other things, one or more processors (e.g.,Components with circuitry), memory (e.g., random access memory (RAM)), input and output device(s), non-volatile storage hardware (e.g., solid-state drives (SSDs), persistent memory (PMEM) devices, hard disk drives (HDDs) (not shown)), one or more physical interfaces (e.g., network ports, storage ports), any number of other hardware components (not shown), and / or any combination thereof.

[0017] Examples of computing devices include, but are not limited to, a server (e.g., a blade server in a blade server enclosure, a rack server in a rack, etc.), a desktop computer, a mobile device (e.g., a laptop computer, a smartphone, a personal digital assistant, a tablet computer, a vehicle computing system, and / or other mobile computing device), a storage device (e.g., a hard disk drive array, a Fibre Channel storage device, an Internet Small Computer Systems Interface (iSCSI) storage device, a tape storage device, a flash memory array, a network-attached storage device, etc.), a network device (e.g., a switch, router, multi-layer switch, etc.), a virtual machine, a virtualized computing environment, a logical container (e.g.,for one or more applications), an Internet of Things (IoT) device, an array of nodes with computing resources, a supercomputing device, a data center or a portion thereof, and / or any other type of computing device having the aforementioned requirements. In one or more embodiments, some or all of the aforementioned examples may be combined to create a system of such devices, or they may be separated into separate logical devices that may collectively be referred to as a computing device. Other types of computing devices may also be used without departing from the scope of the embodiments described herein, such as the computing device described in . Fig. 4 and described below. Cloud infrastructure 100 may include any number and / or type of such computing devices in any arrangement and / or configuration without departing from the scope of the embodiments disclosed herein.

[0018] In one or more embodiments, the storage and / or memory of a computing device or system of computing devices may be and / or include one or more data stores for storing any number of data structures that store any amount of data (e.g., information). In one or more embodiments, a data store is any type of storage unit and / or device (e.g., a file system, a database, a spreadsheet, RAM, and / or other storage mechanism or storage medium) for storing data. Furthermore, the data repository may include multiple different storage units and / or devices. The different storage units and / or devices may be of the same type or located in the same physical location.

[0019] In one or more embodiments, any memory and / or storage of a computing device or system of computing devices may be considered, in whole or in part, a non-transitory computer-readable medium that stores software and / or firmware.

[0020] Such software and / or firmware may include instructions that, when executed by one or more processors (not shown) and / or other hardware (e.g., circuitry) of a computing device and / or system of computing devices, cause the one or more processors and / or other hardware components to perform operations in accordance with one or more embodiments described herein.

[0021] The software instructions may be in the form of computer-readable program code to perform methods, processes, etc. of the embodiments described herein, and may, for example, be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium such as a compact disc (CD), a digital versatile disc (DVD), a storage device, a floppy disk, tape storage, flash memory, physical storage, or other non-transitory computer-readable medium.

[0022] Cloud infrastructure 100 may include any number of computing devices, any number of which may collectively be considered a computing device as used herein. All or a portion of the computing devices may be of the same type or different types of computing devices.

[0023] In one or more embodiments, all or any portion of cloud infrastructure 100 may be managed, at least in part, by a container platform (not shown), such as Kubernetes. In one or more embodiments, a container platform includes a control plane (not shown). A control plane of a container platform may include various components for managing cloud infrastructure 100 (e.g., one or more API servers, controllers, schedulers, etc.). The control plane may, for example, be implemented on one or more master nodes, each of which may be a computing device (see above).

[0024] In one or more embodiments, the cloud infrastructure 100 includes the API server (not shown). In one or more embodiments, the API server is a computing device of the cloud infrastructure 100 and executes in a control plane of a container platform. In one or more embodiments, the API server is configured to interact with entities (e.g., a test framework 116, described further below) that attempt to configure and / or otherwise interact with various aspects of the container ecosystem. For example, the API server may be configured to receive commands (e.g.,to configure a container, to deploy a container instance, to terminate a container, to configure storage for a container, to configure networking for a container, to shut down a port, to restart pods, to detach components from other components, to trigger crashes, to restart components, etc.).

[0025] In one or more embodiments, the cloud infrastructure 100 may host any number of applications that may be implemented using a microservice architecture that includes any number of microservices (e.g., Microservice A 102, Microservice B 104, Microservice C 108, Microservice N 110) as well as various other microservices (e.g., a user interface microservice 106, any number of shared microservices 114) and various communication components that enable the various microservices to interact with other microservices, other parts of the cloud infrastructure 100, and / or entities and / or devices external to the cloud infrastructure 100.

[0026] In one or more embodiments, a microservice (e.g., 102, 104, 106, 108, 110) may be implemented with any number of containers, container pods, etc. Each microservice (e.g., 102, 104, 106, 108, 110) may implement a particular service component that constitutes a portion of an overall application deployed on the cloud infrastructure.

[0027] In one or more embodiments, the user interface microservice 106 may be a microservice that provides a user interface of any type (e.g., graphical, command line, etc.) to users of an application deployed as microservices of a microservice architecture.

[0028] In one or more embodiments, shared microservices 114 may be a set of services common to all or a portion of the other microservices (e.g., 102, 104, 106, 108, 110) of an application deployed using a microservice architecture in cloud infrastructure 100. Examples of shared microservices include database services, authorization services, workflow orchestration microservices, auditing microservices, management microservices, etc.

[0029] In one or more embodiments, cloud infrastructure 100 includes communication components 112. In one or more embodiments, as used herein, communication components 112 include physical and / or logical devices and / or components implementing any number of communication protocols that enable microservice communication. Examples of various communication protocols include the Representational State Transfer (REST) ​​protocol, Remote Procedure Call (RPC) frameworks (e.g., gRPC), the Kafka protocol, etc., as well as other underlying protocols that may be used in communication, such as the Hypertext Transfer Protocol (HTTP), the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), etc. Such protocols may be used for conveying data, commands, etc.

[0030] Although Fig. 1 depicts a particular set of microservices arranged in a particular manner, the cloud infrastructure 100 may comprise any microservice architecture having any number of microservices arranged and configured in any manner without departing from the scope of the embodiments disclosed herein. Such microservice architectures may be used to implement any type of application, any number of applications, and / or any other services (e.g., aaS products). Furthermore, all or a portion of the aforementioned devices, components, etc. (e.g., computing devices, servers, network devices, storage devices, container platforms, containers, container pods, microservices, virtual machines, etc.) in any combination may be referred to herein as one or more components of a cloud infrastructure.

[0031] In one or more embodiments, the Fig. 1 includes a test framework 116 operatively connected to the cloud infrastructure 100. In one or more embodiments, the test framework 116 is implemented with any number of computing devices (as described above). In one or more embodiments, the test framework 116 includes a set of components that can be used to perform tests on a microservices architecture in the cloud infrastructure 100, monitor the test results, and provide the results to relevant entities performing the tests.

[0032] In one or more embodiments, the test framework 116 includes a framework controller 118. The framework controller 118 may be any hardware, software, firmware, or a combination thereof that is configured to receive input from users regarding test components of the cloud infrastructure 100 and to forward information regarding such input to other components of the test framework 116. For example, the framework controller 118 may be implemented with one or more computing devices (described above). In one or more embodiments, the framework controller 118 is a component configured to receive input from a user (e.g., a test engineer). Such input may include a definition of what the user wants to test and the expected outcome of the test execution.In one or more embodiments, the framework controller provides details of what the user wishes to test to the framework injector 122 and provides the expected results to the framework monitor 120.

[0033] The type of input requested from the user via the framework controller 118 may depend on the type of testing technology the user intends to use to test the microservice architecture and cloud infrastructure. The expected results of the tests may depend on the test types specified by the user. Thus, a user interacting with the framework controller 118 may be presented with the various categories of testing technologies, the various types of errors, crashes, panics, failures, etc., that fall into each of the categories. In one or more embodiments, the user may select or otherwise specify the type of testing technology, the various actions they wish to perform with those testing technologies, and the expected outcome or response to performing those actions.

[0034] In one or more embodiments, the test framework 116 includes the framework injector 122. The framework injector 122 may be any hardware, software, firmware, or any combination thereof that is configured to receive information based on user input from the framework controller 118 and use the information to perform any number of tests by interacting with various components of the cloud infrastructure 100. The framework injector 122 may include various components for performing such tests, including an interrupt injector, a communication fault injector, and a crash point injector (which may be collectively referred to as test injectors), each of which is described in the description of Fig. 2 described below.

[0035] In one or more embodiments, the test framework 116 includes the framework monitor 120. The framework monitor may be any hardware, software, firmware, or a combination thereof configured to monitor the cloud infrastructure as tests are performed by the framework injector 122, to observe the results, impacts, etc., of the tests on the various components of the cloud infrastructure 100, to record the results, and to make those results available to the user initiating the tests.

[0036] In one or more embodiments, the framework monitor 120 monitors the cloud infrastructure 100 into which the faults, errors, and / or crash points are introduced (e.g., by the framework injector 122) to determine whether the response of the cloud infrastructure 100 to the faults, errors, and / or crash points is as expected. As described above, in one or more embodiments, the expected result of certain tests is provided by the user via the framework controller 118. In one or more embodiments, the framework controller 118 provides the expected results to the framework monitor 120.In one or more embodiments, the framework monitor 120 is configured to monitor the cloud infrastructure 100 and the components therein to determine whether the actual response to the interruptions, errors, and / or crash points matches the expected results specified by the user.

[0037] For example, the user may expect that a crash, panic, reboot, etc., of a hardware component will result in a failover to a redundant component, that the component will recover within a certain period of time, that other parts of the environment will continue to operate while the failed component recovers, etc. As another example, the user may specify the expected behavior of an application implemented with microservices when one or more of the microservices fails. Other expected behaviors may be specified by the user without exceeding the scope of the embodiments disclosed herein. If the outcome of the interruption, error, and / or crash does not match the result expected by the user, the framework monitor may, in one or more embodiments, issue a warning to the user.Such a warning may take any form and contain any amount and / or type of information. For example, the user may receive a notification indicating that the expected result did not occur, as well as various details about the response of the cloud infrastructure 100 observed by the framework monitor 120.

[0038] In one or more embodiments, the framework controller 118, the framework monitor 120, and the framework injector 122 of the test framework 116 may be implemented as part of a single computing device or as any number of separate and operatively connected computing devices.

[0039] Although Fig. 1 shows a particular configuration of components, other configurations may be used without departing from the scope of the embodiments described herein. Although in Fig. 1 certain components are shown as part of one and the same device, each of the components may be grouped into sets of one or more components that may exist and be executed as part of any number of separate and operatively connected devices. As a further example, a single component may be configured to perform all or part of the functions required by all or part of the devices shown in Fig. 1. Accordingly, the embodiments disclosed herein should not be limited to the components shown in Fig. 1 shown configuration of the components may be limited.

[0040] Fig. 2 shows a block diagram of an example framework injector 200 in accordance with one or more embodiments disclosed herein. As in Fig. 2, the Framework Injector 200 includes a Disruption Injector 202, a Communication Error Injector 204, and a Crash Point Injector 206. Each of these components is described below.

[0041] In one or more embodiments, the frame injector 200 is the same as or substantially similar to the frame injector 200 shown in Fig. 1 and described above. As such, the framework injector 200 may be a component of a test framework (e.g., the one described in Fig. 1 and described above). The framework injector 200 may be operatively connected to a framework controller (e.g., the one shown in Fig. 1 and described above), a framework monitor (e.g. the one shown in Fig. 1 and described above) and with a cloud infrastructure (e.g. the one in Fig. 1 and described above). The framework injector 200 may be any hardware, software, firmware, or a combination thereof configured to perform the functions described herein. In one or more embodiments, the framework injector 200 is implemented with any number of computing devices (described above).

[0042] In one or more embodiments, the framework injector 200 includes the disruption injector 202. The disruption injector 202 may be any hardware, software, firmware, or a combination thereof that is configured to introduce disruptions into any component, element, part, etc., of a cloud infrastructure. In one or more embodiments, the disruption injector 202 is configured to implement testing techniques that provide the user with the ability to prevent the introduction of crashes, panics, reboots, etc. (e.g., disruptions) of various components of the environment under test (e.g., the cloud infrastructure 100 of Fig. 1). In one or more embodiments, the crashes, panics, reboots, etc., may be caused on hardware devices (e.g., servers, network devices, storage devices, etc.), platforms (e.g., Kubernetes platforms, container orchestration services, container management systems, virtual machine platforms, etc.), portions of systems hosting microservices (e.g., containers, container pods, virtual machines, etc.), one or more microservices, authorization services, databases, workflow orchestration and management services, etc.

[0043] In one or more embodiments, the radius of the disruption may be controlled depending on which part of the microservice architecture and / or cloud infrastructure is experiencing disruptions, such as crashes, panics, errors, etc. For example, the disruption may relate to a single container, a container pod, a container platform, a specific microservice of an application, microservices common to a variety of other parts of the system, and / or an entire container platform. In one or more embodiments, the disruptions initiated by the user via a framework controller (e.g., the framework controller 118 of Fig. 1) specified faults are made available to the framework injector 200.

[0044] In one or more embodiments, the framework injector 200 is configured to perform actions to cause the desired disruptions (e.g., via the disruption injector 202) to components, devices, parts, etc., of a cloud infrastructure. In one or more embodiments, the actions are performed using predefined commands, such as Kubernetes commands, various application programming interface (API) commands, etc., depending on the part of the cloud infrastructure that is to be disrupted. For example, Kubernetes commands can be used to crash or panic all or part of a container platform (e.g., cause a port to restart); all components within it; all microservices deployed on it; etc. As another example, various API commands can be used to cause crashes, panics, restarts, etc.of servers, network devices, storage devices, other hardware components, data backup orchestrators, etc.

[0045] In one or more embodiments, the framework injector 200 includes the communication fault injector 204. In one or more embodiments, the communication fault injector 204 is any hardware, software, firmware, or any combination thereof configured to throw faults in a cloud infrastructure when communications are in progress. Various communication protocols include, for example, the REST protocol, RPC frameworks (e.g., gRPC), the Kafka protocol, etc., as well as other underlying protocols that may be used in communication, such as, but are not limited to, HTTP, TCP, UDP, etc. Such protocols may be used, for example, for the transmission of data or other information (e.g., commands). Such protocols may have associated APIs through which various faults can be injected, such as delay errors, timeouts, etc.Communication error technologies may also include causing other errors, such as resource errors (causing memory errors, causing processor errors, etc.), server utilization errors (e.g., causing a heavy load on the computing resources of one or more servers), conformance errors (e.g., causing protocols that do not conform to expected behavior), and validation errors (e.g., causing packet ordering errors). In one or more embodiments, any of the aforementioned communication errors may be triggered by a user via a framework controller (e.g., framework controller 118 of FIG. Fig. 1) and injected into a cloud infrastructure by the framework injector 200. In one or more embodiments, as described above, a framework monitor (e.g., the framework monitor 120 in Fig. 1) Monitor the environment to determine whether the behavior observed in the cloud infrastructure matches the expected results specified by the user, and if not, an alert can be delivered to the user.

[0046] In one or more embodiments, the framework injector 200 includes the crash point injector 206. In one or more embodiments, the crash point injector 206 is any hardware, software, firmware, or any combination thereof configured to inject crash points into code executable in a cloud infrastructure environment. In one or more embodiments, crash point injection includes the strategic addition of bugs, errors, etc., into microservice code, application code, etc., to intentionally cause failures at specific points. In one or more embodiments, the injection of fault points into microservice or other code enables validation of the effectiveness of mechanisms in a microservice, such as undo operations, redo operations, retry operations, etc.

[0047] For example, certain microservice operations may be performed in different phases, and crash points may be inserted into the microservice code to cause errors at specific points within those phases to verify that the microservice is responding as expected. For example, a microservice may be configured to perform a restore operation from a previously created backup. The restore operation may involve three phases: restoring the storage (e.g., from a snapshot), attaching the restored storage to a compute instance (e.g., an Amazon Elastic Cloud Compute (EC2) instance), and providing the updated information about the storage attached to the instance to a management service.

[0048] In one or more embodiments, in such a scenario, a crash point may be inserted at any point in any of the three phases to cause an error. In one or more embodiments, a user issues a command via a frame control (e.g., frame control 118 of Fig. 1) the stage at which the error should occur, as well as the expected result. Continuing the previous example, the user can specify an expected result, which states that if an error occurs in the second stage of a process executed by a microservice, the microservice, after restarting, retries the second stage if the error occurs after a certain point, but does not retry the first stage, which has been completed, and then proceeds to the third stage. Based on the user's input, a framework controller (e.g., the framework controller 118 of Fig. 1) Pass information to the framework injector 200, which in turn can insert the corresponding one or more failure points into the code of the respective microservice so that the microservice reaches the failure point during execution. In one or more embodiments, a framework monitor (e.g., the framework monitor 120 of Fig. 1) the behavior of the microservice, determines whether the behavior matches the result expected by the user, and warns the user if the observed behavior does not match the expected behavior.

[0049] Although Fig. 2 shows a particular configuration of components, other configurations may be used without departing from the scope of the embodiments described herein. Although in Fig. 2 certain components are shown as part of the same device, each of the components may be grouped into sets of one or more components that may exist and be executed as part of any number of separate and operatively connected devices. As a further example, a single component may be configured to perform all or part of the functions required by all or part of the devices shown in Fig. 2. Accordingly, the embodiments disclosed herein should not be limited to the components shown in Fig. 2 shown configuration of the components may be limited.

[0050] Fig. 3 shows an overview of an example method for testing a cloud infrastructure according to one or more embodiments disclosed herein. The method may be implemented at least in part by a test framework (e.g., the one described in Fig. 1) or any component of a test framework (e.g., the framework controller 118 of Fig. 1, the Framework Monitor 120 from Fig. 1, the framework injector 122 of Fig. 1, the framework injector 200 of Fig. 2, the interrupt injector 202 of Fig. 2, the communication error injector 204 of Fig. 2, the crash point injector from Fig. 2) are carried out.

[0051] While the various steps in the Fig. 3 are shown and described sequentially, some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, and some or all of the steps may be performed in parallel with other steps of Fig. 3 are executed.

[0052] In step 300, the method includes receiving a test execution request with test parameters and an expected test result. For example, the test execution request may be initiated by a user through a framework controller (e.g., the Fig. 1) of a test frame (e.g. the test frame 116 of Fig. 1). In one or more embodiments, the test parameters are information about one or more tests or test scenarios that the user wishes to perform in a cloud infrastructure.

[0053] For example, a user may want to test a microservice architecture implemented in a cloud infrastructure (e.g., the cloud infrastructure 100 of Fig. 1). The test parameters may depend on which test(s) are to be performed. In one or more embodiments, the framework controller includes a user interface that provides the user with options for various test technologies that the test framework can perform. For example, the framework controller may present one or more test technologies (e.g., interrupt injection, communication fault injection, crash point injection, etc.) to the user or otherwise allow the user to input, select, etc. In one or more embodiments, the framework controller may also request or otherwise receive user input based on the type of tests to be performed.

[0054] For example, if the user specifies that a disruption injection test should be performed, they can specify as test parameters which components (e.g., servers, network devices, storage devices, specific subcomponents, virtual machines, containers, container pods, microservices, applications, container platforms, etc.) and how these components should be disrupted by the test (e.g., a single container, a single container pod, an entire microservice, an entire application, an entire container platform, a specific percentage or subset of devices of one or more types in the cloud infrastructure, etc.).

[0055] Another example: If the user specifies that they want to perform communication fault injection tests, the frame controller can request information from the user about one or more specific communication protocols (e.g., gRPC, Kafka, etc.) into which faults should be injected, the length of communication delays to be injected, devices and / or components where faults should be caused (e.g., memory errors, server busy errors due to congestion, etc.), conformance errors (e.g., causing behavior that is inconsistent with the protocol specifications), validation errors (e.g., causing intentional reordering of packets on one or more specific data paths), etc.

[0056] Another example: If a user specifies that they want to perform a crash point injection test, the framework controller can request or otherwise obtain details from the user about which code, processes, phases, etc. of operations in a cloud infrastructure the user wants a crash or failure point injected into. A user can specify general details (e.g., a crash point injected into the second stage of a backup and restore process) or they can specify more precisely (e.g., a crash point injected into the microservice code for a specific microservice when conditions in a cloud environment meet certain specified parameters).

[0057] In one or more embodiments, an expected test result is what the user expects when one or more tests are executed on the cloud infrastructure according to the aforementioned test parameters. An expected test result can be any amount of information at any level of detail related to the response of all or part of the cloud infrastructure to any number of tests performed by the test framework. Examples include, but are not limited to, the expectation of timely failover and resumption of service, retry for at least a certain period of time prior to the failure, the expectation that a service or application will continue to run after a failure of one or more microservices, containers, pods, etc.

[0058] In step 302, the method includes providing the test execution request and the test parameters to a framework injector. For example, the test execution request and the test parameters may be provided to the framework injector 122 of Fig. 1 or the frame injector 200 from Fig. 2 from the frame control (e.g. the frame control 118 of Fig. 1). The test execution request and test parameters can be communicated to the framework injector using any information-passing technique, such as passing a cross-process message, sending one or more packets over a network, etc. The test execution request and test parameters can be what the user provided to the framework controller, or they can have been modified in some way by the framework controller to be in the form expected by the framework injector.

[0059] In step 304, the method includes providing the expected test result to the framework monitor. For example, a framework controller (e.g., the framework controller 118 of Fig. 1) the expected test result to a framework monitor (e.g. the framework monitor 120 of Fig. 1) The expected test result can be communicated to the framework injector using any information-passing technique, such as passing a cross-process message, sending one or more packets over a network, etc. The expected test result may match what the user provided to the framework controller, or it may have been modified in some way by the framework controller to be in a form expected by the framework injector.

[0060] In step 306, the method includes executing a test that conforms to the test execution request using one of several testing technologies implemented by the framework injector and based at least in part on the test parameters. For example, a framework injector (e.g., the framework injector 200 of Fig. 2) determine, based on the test execution requirement, whether the test uses an interrupt injector (e.g., the interrupt injector 202 of Fig. 2), a communication error injector (e.g. the communication error injector 204 of Fig. 2), a crash point injector (e.g. the crash point injector 206 of Fig. 2) or any combination thereof. In one or more embodiments, executing the test comprises using the test parameters to configure the relevant one or more of the above-mentioned injectors to initiate, perform, or otherwise execute the tests requested by the test execution request. Executing the test may also comprise communicating information about the test to the framework monitor (e.g., which tests are being executed, when the test is scheduled to start or has started, portions of a cloud infrastructure where the test is scheduled to take place, etc.).

[0061] In step 308, the method includes monitoring the response of a cloud infrastructure to the test performed in step 306. For example, the cloud infrastructure may be monitored by a framework monitor (e.g., the framework monitor 120 of Fig. 1) be monitored while the test is being performed by the framework injector. In one or more embodiments, monitoring the cloud infrastructure comprises observing all or part of the cloud infrastructure using a suitable technique for monitoring computing devices, software running thereon, etc. Monitoring may include, for example, but is not limited to, observing network traffic patterns, tracking the state of components, devices, applications, containers, pods, virtual machines, etc.; monitoring the results of operations (e.g., backup operations, restore operations, failover operations, retry operations, recovery operations, etc.).In one or more embodiments, monitoring may include recording or otherwise generating or storing information about the observed behavior of the cloud infrastructure during the conduct of the test.

[0062] In step 310, the method includes providing a result comprising the behavior of the cloud infrastructure in response to the test. For example, the result may be communicated from the framework monitor to the framework controller for presentation to the user who requested the test. In one or more embodiments, the result includes whether or not the behavior observed in the cloud infrastructure during the test matches the expected test result specified by the user. The result may be in any suitable form for conveying information. For example, the result may be formatted into a report detailing the result of the test, the behavior of various parts of the cloud infrastructure during the test, and the conformity of the behavior to the expected test result.

[0063] Fig. Figure 4 shows a block diagram of a computing device according to one or more embodiments of this disclosure. As described above, the embodiments described herein may be implemented using computing devices. For example, all or a portion of the Fig. 1 may be implemented at least in part with one or more computing devices. Computing device 400 may include one or more computer processors 402, non-persistent memory 404 (e.g., volatile memory such as RAM, cache memory, etc.), persistent storage 406 (e.g., a hard drive, an optical drive such as a CD drive or a DVD drive, flash memory, etc.), a communications interface 412 (e.g., a Bluetooth interface, an infrared interface, a network interface, an optical interface, etc.), input devices 410, output devices 408, and numerous other elements (not shown) and functions. Each of these components is described below.

[0064] In one or more embodiments, the computer processor(s) 402 may be an integrated circuit for processing instructions. The computer processor(s) may be, for example, one or more cores or micro-cores of a processor. The processor 402 may be a general-purpose processor configured to execute program code included in the software executing on the computing device 400. The processor 402 may be a special-purpose processor in which certain instructions are integrated into the processor design. Although in Fig. 3 only one processor 402 is shown, the computing device 400 may include any number of processors without departing from the scope of the embodiments disclosed herein.

[0065] Computing device 400 may also include one or more input devices 410, such as a touchscreen, a keyboard, a mouse, a microphone, a touchpad, an electronic pen, a motion sensor, or another type of input device. Input devices 410 may enable a user to interact with computing device 400. In one or more embodiments, computing device 400 may include one or more output devices 408, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touchscreen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, external storage, or other output device. One or more of the output devices may be the same as or different from the input device(s).The input and output device(s) may be connected locally or remotely to the computer processor(s) 402, the non-persistent memory 404, and the persistent memory 406. There are many different types of computing devices, and the above-mentioned input and output device(s) may take other forms. In some cases, multimodal systems may allow a user to provide multiple types of input / output to communicate with the computing device 400.

[0066] Additionally, the communications interface 412 may facilitate connection of the computing device 400 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or to another device, such as another computing device.The communication interface 412 can perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceiver devices, including those having an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, Bluetooth® wireless signaling, BLE wireless signaling, IBEACONO wireless signaling, RFID wireless signaling, Near Field Communication (NFC) wireless signaling, Dedicated Short Range Communication (DSRC) wireless signaling, an 802.11 WiFi radio signal transmission, WLAN signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication radio signal transmission, PSTN (Public Switched Telephone Network) signal transmission, ISDN (Integrated Services Digital Network) signal transmission, 3G / 4G / 5G / LTE radio signal transmission for cellular data networks, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, radio signal transmission along the electromagnetic spectrum, or a combination thereof.The communications interface 412 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing device 400 based on the reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, among others, the U.S. GPS, the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS. There is no restriction on operation on any particular hardware arrangement, and therefore the basic functions here can be easily replaced with improved hardware or firmware arrangements as they are developed.

[0067] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transmitting instructions and / or data. A computer-readable medium may be a non-transitory medium capable of storing data and which does not contain carrier waves and / or transitory electronic signals that propagate wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as CDs or DVDs, flash memory, storage, or memory devices.A computer-readable medium may store code and / or machine-executable instructions that may represent a procedure, function, subroutine, program, routine, subprogram, module, software package, class, or any combination of instructions, data structures, or program instructions. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, relayed, or transmitted by any means, such as memory sharing, message passing, token passing, network transmission, or the like.

[0068] All or a portion of the components of computing device 400 may be implemented in circuitry. For example, the components may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits) and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some aspects, the computer-readable storage devices, media, and memories may include a wired or wireless signal containing a bitstream and the like.However, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves and signals as such.

[0069] In the above description, numerous details are given as examples of embodiments described herein. Those skilled in the art (who will also benefit from this disclosure) will understand that one or more embodiments described herein may be practiced without these specific details, and that numerous variations or modifications are possible without departing from the scope of the embodiments described herein. Certain details known to those skilled in the art may be omitted in order not to obscure the description.

[0070] In the above description, specific details are set forth to provide a thorough understanding of the aspects and examples presented herein. However, one skilled in the art will appreciate that the aspects may be practiced without these specific details. For clarity, in some cases, the present technology may be represented as functional blocks, which may include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be represented as components in block diagram form so as not to obscure the aspects with unnecessary detail.In other cases, known circuits, processes, algorithms, structures and techniques may be presented without unnecessary details in order not to obscure the aspects.

[0071] Individual aspects can be described above as a process or method, which are represented in the form of a flowchart, a sequence diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are complete, but may contain additional steps not included in a diagram. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. If a process corresponds to a function, its termination can correspond to the function returning to the calling function or to the main function.

[0072] Methods and techniques according to the examples described above may be implemented using computer-executable instructions stored on computer-readable media or otherwise available. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or set of functions. Portions of the computing resources used may be accessible over a network. The computer-executable instructions may be, for example, binary files, instructions in intermediate formats such as assembly language, firmware, source code, etc.Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0073] In the above description of the figures, each component described in a figure may, in various embodiments described herein, correspond to one or more equally or similarly named and / or numbered components described in another figure. For brevity, the descriptions of these components are not repeated for each figure. Therefore, each individual embodiment of the components of each figure is incorporated by reference and is understood to be optionally present in any other figure having one or more equally or similarly named and / or numbered components.Moreover, in accordance with various embodiments described herein, any description of the components of a figure is to be understood as an optional embodiment that may be implemented in addition to, in conjunction with, or in place of the embodiments described with respect to one or more like or similarly named and / or numbered components in another figure.

[0074] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not intended to imply a particular order of the elements, nor is it intended to limit an element to a single element, unless explicitly stated, e.g., by the use of the terms "before," "after," "single," and other such terms. Rather, the use of ordinal numbers serves to distinguish between the elements. For example, a first element is to be distinguished from a second element, and the first element may comprise more than one element and may follow (or precede) the second element in the order of the elements.

[0075] As used herein, the term "operatively connected" or "operative link" means that a direct or indirect connection exists between elements / components / devices that enables the elements to interact with each other in some way. The term "operatively connected" can refer, for example, to any direct (e.g., a direct wired connection between two devices or components) or indirect (e.g., wired and / or wireless connections between any number of devices or components connecting the operatively connected devices) connection. Thus, any path over which information can be transmitted can be considered an operative link.

[0076] While the embodiments discussed herein have been described with respect to a limited number of embodiments, those skilled in the art, given the benefit of this disclosure, will recognize that other embodiments may be developed that do not depart from the scope of the embodiments disclosed herein. Accordingly, the scope of the embodiments described herein should be limited only by the appended claims.

Claims

[1] A test framework consisting of: one or more processors; a frame controller executing on the one or more processors; a framework monitor executing on the one or more processors; a framework injector executing on the one or more processors; and one or more non-transferable computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: Receiving, at the framework controller, a test execution request comprising test parameters and an expected test result, wherein the test execution request comprises a request to test a microservice architecture implemented in a cloud infrastructure; Transmission of the test execution request and the test parameters by the frame controller to the frame injector; to transmit the expected test result to the framework monitor by the framework controller; Executing a test corresponding to the test execution request by the framework injector using one test injector from a plurality of test injectors implemented by the framework injector; Monitoring a response of the cloud infrastructure microservice architecture to the test by the framework monitor; and Providing a test result based on the response through the frame control and frame monitor. [2] The test frame of claim 1, wherein the test injector comprises an interrupt injector. [3] The test framework of claim 2, wherein the test is performed using the disturbance injector and comprises causing a disturbance in one or more components of the cloud infrastructure. [4] The test framework of claim 1, wherein the test injector comprises a communication fault injector. [5] The test framework of claim 4, wherein the test is performed using the communication fault injector and comprises injecting a communication fault into the cloud infrastructure. [6] The test frame of claim 1, wherein the test injector comprises a crash point injector. [7] The test framework of claim 6, wherein the test is performed using the crash point injector and comprises inserting a crash point into the code corresponding to one or more components of the cloud infrastructure. [8] The test framework of claim 1, wherein the test result includes an indication of whether the response matches the expected test result. [9] A computer-implemented method comprising: Receiving a test execution request comprising test parameters and an expected test result at a framework controller executing on one or more processors, the test execution request comprising a request to test a microservice architecture implemented in a cloud infrastructure; Providing the test execution request and the test parameters by the frame controller to a frame injector executing on the one or more processors; providing the expected test result by the frame controller to a frame monitor executing on the one or more processors; Executing a test corresponding to the test execution request by the framework injector using one test injector from a plurality of test injectors implemented by the framework injector; Monitoring a response of the cloud infrastructure microservice architecture to the test by the framework monitor; and Providing a test result based on the response through the frame control and frame monitor. [10] The computer-implemented method of claim 9, wherein the test injector comprises an interrupt injector. [11] The computer-implemented method of claim 10, wherein the test is performed using the disturbance injector and comprises causing a disturbance in one or more components of the cloud infrastructure. [12] The computer-implemented method of claim 9, wherein the test injector comprises a communication fault injector. [13] The computer-implemented method of claim 12, wherein the test is performed using the communication fault injector and comprises injecting a communication fault into the cloud infrastructure. [14] The computer-implemented method of claim 9, wherein the test injector comprises a crash point injector. [15] The computer-implemented method of claim 14, wherein the test is performed using the crash point injector and comprises inserting a crash point into the code corresponding to one or more components of the cloud infrastructure. [16] A computer-assisted method according to claim 9, wherein the test result includes an indication of whether the response matches the expected test result. [17] A non-transitory computer-readable medium storing programming for execution by one or more processors, the programming comprising instructions to: Receiving, at a framework controller, a test execution request comprising test parameters and an expected test result, wherein the test execution request comprises a request to test a microservice architecture implemented in a cloud infrastructure; Provision of the test execution request and test parameters by the framework controller to a framework injector; Transmission of the expected test result by the framework controller to a framework monitor; Executing a test corresponding to the test execution request by the framework injector using one test injector from a plurality of test injectors implemented by the framework injector; Monitoring a response of the cloud infrastructure microservice architecture to the test by the framework monitor; and Providing a test result based on the response through the frame control and frame monitor. [18] The non-transitory computer-readable medium of claim 17, wherein: the test injector comprises an interruption injector, and the test is performed using the interrupt injector and involves causing an interrupt in one or more components of the cloud infrastructure. [19] The non-transitory computer-readable medium of claim 17, wherein: the test injector comprises a communication fault injector, and the test is performed using the communication fault injector and involves injecting a communication fault into the cloud infrastructure. [20] The non-transitory computer-readable medium of claim 17, wherein: the test injector comprises a crash point injector, and the test is performed using the Crash Point Injector and involves inserting a crash point into code corresponding to one or more components of the cloud infrastructure.

Images