Non-intrusive, dynamically scalable network load generation

Abstract

Embodiments of the present disclosure relate to non-intrusive dynamically scalable network load generation. A method for non-intrusive network load generation may include determining available resources in a distributed computing system, wherein the distributed computing system includes multiple computing devices and a target deployment. Based on the amount of available resources between the target deployment and the multiple source computing devices, multiple source computing devices may be selected to generate a network load directed from the multiple source computing devices to the target deployment. The multiple source computing devices may be a subset of the multiple computing devices in the distributed computing system. A network traffic generator service may be provided to the multiple source computing devices to generate a network load directed from the multiple source computing devices to the target deployment. The performance of the distributed computing system in response to the generated network load may be monitored.

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2015 / 048747, international application date September 7, 2015, entering the Chinese national phase on May 3, 2017, Chinese national application number 201580059805.6, and invention name “Non-intrusive dynamically scalable network load generation”. Background Art

[0002] In a distributed computing system, a new or modified deployment of a compute or storage cluster may require specific capacity or throughput. Before releasing the deployment to customers, testing is required to ensure that the network connectivity provided to the deployment is sufficient to meet the specified service levels. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used alone as an aid in determining the scope of the claimed subject matter.

[0004] Embodiments of the present invention provide methods, systems, and computer storage media for performing non-intrusive, dynamically scalable network load generation. This can be used to verify network connectivity for new and / or existing cloud infrastructure, such as deployment of storage or computing clusters (e.g., multiple computing devices), in a distributed computing system. Network connectivity can include, for example, network bandwidth and network latency (e.g., bandwidth and latency specified in a service level agreement with a customer). Load generation can also be performed to simulate a denial of service ("DoS") attack and ensure that various network components gracefully manage the simulated DoS attack.

[0005] In order to generate a network load targeting the desired cloud infrastructure, a network traffic generator service can be executed at multiple source computing devices across different source sites (such as different data centers). The computing device selected to generate the network load can be selected based on the resources available at the computing device (such as bandwidth and computing power). Specifically, available resources can be determined (e.g., there are no resources for supporting real-time operations), and the computing device can be selected as a source computing device based on the association of the computing device with a certain amount of available resources. In this way, the network load for testing can be generated using unused or "spare" resources in the network, and the network load can be generated in parallel with the real-time network business. This use of available resources enables the generation of network load in a non-intrusive manner, so that the generation does not interrupt the real-time network business that supports other customers' real-time operations.

[0006] Initially, the generated network load may be relatively small. If the distributed computing system performs satisfactorily in response to the load, the size of the load can be increased. The network load can then be increased incrementally until the load is sufficient to verify the desired level of network connectivity and / or simulate a DoS attack at the target infrastructure. As the load increases, the source computing device used to generate the load can be changed and / or additional source computing devices can be adopted. In addition, existing source computing devices can be released back to reduce resource usage. In this way, the size of the load can be dynamically scalable, and the starting point of the network load can be dynamically selected. The size of the network load can be dynamically scaled within each specific source site and across multiple source sites.

[0007] The distributed computing system may be monitored at each incremental network load generated. The monitoring may include generating reference points indicating the performance of various system components at different network loads. The monitoring may also include monitoring performance metrics during load generation to determine whether the distributed computing system performs satisfactorily in response to the generated network load.

[0008] If performance issues occur during testing, such as network failures or outages at the target infrastructure, the network load can be reduced and / or eliminated. Additionally, if the computing power of existing source computing devices is needed, the existing source can be reduced and / or eliminated. Thus, the size of the load can be dynamically scalable based on various performance metrics. Automatic throttling can prevent real-time business incidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present invention are described in detail below with reference to the accompanying drawings, in which:

[0010] Figure 1 is a block diagram of an exemplary computing environment suitable for use in implementing embodiments of the present invention;

[0011] Figure 2 is a block diagram of an exemplary network environment in which embodiments of the present invention may be employed;

[0012] Figure 3 is a block diagram of an exemplary network environment in which embodiments of the present invention may be employed;

[0013] Figure 4 is a block diagram of an exemplary geographic region including a plurality of source locations for generating network loads according to an embodiment of the present invention;

[0014] Figure 5 is a flow chart illustrating an exemplary method for non-intrusive network load generation in a distributed computing system according to an embodiment of the present invention;

[0015] Figure 6is a flow chart illustrating an exemplary method for non-intrusive network load generation according to an embodiment of the present invention; and

[0016] Figure 7 is a flow chart illustrating an exemplary method for non-intrusive network load generation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The subject matter of the embodiments of the present invention is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. On the contrary, it is expected that the claimed subject matter may also be implemented in other ways to include different steps or combinations of steps similar to the steps described in this document in combination with other existing or future technologies. In addition, although the terms "step" and / or "frame" may be used herein to indicate different elements of the method employed, these terms should not be interpreted as implying any particular order between or among the various steps disclosed herein, unless and except when the order of the individual steps is explicitly described.

[0018] For purposes of this disclosure, the word "including" has the same broad meaning as the word "comprising." Furthermore, unless otherwise indicated, words such as "a" and "an" include the plural as well as the singular. Thus, for example, where there are one or more features, the constraint of "one feature" is satisfied. Furthermore, the term "or" includes conjunctions, transitions, and both (a or b therefore includes a or b, and a and b).

[0019] Embodiments of the present invention provide methods, systems, and computer storage media for performing non-intrusive dynamic scalable network load generation. The generated load can be used to verify network connectivity for new and / or existing cloud infrastructures (such as deployments of computing and / or storage infrastructures). For example, network connectivity can be verified at a storage infrastructure such as a "storage cluster" or "storage stamp". A specific level of network connectivity may be required at a storage cluster in order to meet specific service levels that need to be provided under service level agreements ("SLAs") with various customers. In some cases, a storage cluster is a dedicated storage cluster to which a customer has exclusive access. In this case, a specific level of network connectivity may be required at a storage cluster in order to meet specific customer requirements under the SLA with the customer. Therefore, before releasing a storage cluster to one or more customers (e.g., using a storage cluster "golive"), end-to-end testing that simulates customer behavior can be performed to ensure that the storage cluster is correctly configured within the network infrastructure. End-to-end testing can also ensure that the network connectivity at the storage cluster is sufficient to meet the standards set forth in the SLA. For example, testing may confirm that a storage cluster is associated with a connection that provides sufficient bandwidth, latency, and / or throughput and also maintains latency and / or failures within acceptable levels (e.g., those set forth in an SLA). End-to-end testing in a live environment in which live traffic flows around the storage cluster shows how the storage cluster will behave when it is rolled out. It should be noted that a storage cluster is only one example of a deployment that may be tested according to embodiments of the present invention. Deployments of other infrastructure, such as those of a compute cluster, may be tested for adequate network connectivity.

[0020] Network load generation can also be used to subject various components within the network infrastructure to simulated DoS attacks. A true DoS attack can include making specific network components and / or resources unavailable for user efforts by filling target components and / or resources with so many external requests that components and / or resources cannot reasonably support legitimate business (e.g., the response time of components and / or resources becomes very slow, so that it cannot support legitimate business). For example, a DoS attack can flood network connections and / or fill storage capacity. A simulated DoS attack as described herein can be performed to evaluate the response of various network components to the attack. This can be done to determine whether a component is configured to gracefully manage a true DoS attack. For example, a target deployment can be subjected to simulated attacks before the target deployment is launched. The simulation can ensure that the target deployment is configured to gracefully manage a true attack. Additionally or alternatively, the target deployment can be an existing infrastructure, and the simulation can ensure that the existing infrastructure is configured to gracefully manage a true attack. Elegant management can include implementing a defensive response. This defensive response can include detecting attacks, responding to attacks with appropriate tools, and / or preventing future attacks. Appropriate tools can include firewalls, switches, and routers. These tools can help block illegal traffic while allowing legitimate traffic.

[0021] In order to verify the network connection within the network infrastructure and / or simulate a DoS attack, a large-scale network load can be generated. For example, a load of 40 gigabytes per second may be required to test the network connection at the target storage cluster. Generating this large load may require the bandwidth and / or computing resources of multiple computing devices to minimize the impact on the source of large network use. For example, the sum of the resources available at multiple computing devices located in multiple data centers may be sufficient to generate the large load required for the test. Therefore, multiple computing devices across multiple source sites can be selected as source computing devices for generating loads.

[0022] Advantageously, the network load generation described herein enables full scale testing (e.g., testing a load of 40 gigabits per second) while minimizing interference with real-time operations. This is important for several reasons. First, full scale testing ensures that the network infrastructure in the entire network pipeline is correctly set up and / or configured for full scale operation. In other words, the test is not limited to the local network infrastructure because such small-scale testing will not reveal problems outside the local infrastructure. The network device manages the real-time traffic and the network traffic generated for the purpose of testing, which means that the network device is tested for full scale operation. Secondly, current network load generation provides the advantage of full scale testing in a real-time network environment without interfering with real-time network operations. Therefore, load generation is "non-intrusive" and minimizes failures in real-time operations.

[0023] Specifically, non-intrusive network load generation aims to minimize the damage or intrusion to the real-time business on the network. This can be achieved by dynamically selecting the most suitable source location set for generating load. The suitability of a specific source location and / or a specific source computing device can be based on the current network bandwidth between the source and the target deployment. The utilization of the network bandwidth (e.g., the current utilization for real-time operation) and the remaining available capacity can also be considered. For example, the computing device selected to initiate the network load generated for the test can be different from the computing device currently supporting real-time operation. The use of available resources enables the generation of network load in a non-intrusive manner, so that it generates real-time network business that does not interrupt the real-time operation of other customers. Based on the availability of network capabilities and computing power, the network business generator service can be deployed to geographically distributed source computing devices. The network business generator service can be used to generate network load.

[0024] Initially, the network load may be relatively small. The network load may then be increased incrementally until the load is sufficient to verify the desired level of network connectivity at the target service and / or simulate a DoS attack of the desired size. As the load increases, the source computing device used to generate the load may be changed and / or additional source computing devices may be employed. In this way, the size of the load may be dynamically scalable, and the starting point of the network load may be dynamically selected.

[0025] When performing load generation, the distributed computing system can be monitored. For example, various metrics for network traffic generator services, health metrics for the network, and metrics for target deployments can be monitored. These monitored metrics can be used as reference points for the performance of the distributed computing system at a specific network load. For example, a reference point can provide a snapshot of the performance of the distributed computing system at any number of network loads. These reference points can be used to understand how the distributed computing system behaves at various loads. Reference points can be used to assess whether additional network loads can be generated while continuing to meet the current SLA with the customer, and to help set expectations for the future performance of the distributed computing system.

[0026] In addition, based on monitoring the performance of the distributed computing system in response to the generated network load, the network load can be dynamically scaled across the source computing devices. For example, the load can be expanded and / or reduced at various source computing devices. Such monitoring and associated scaling can help minimize disruptions and costs.

[0027] If a problem occurs during testing, such as a network failure or outage at the target deployment, the network load can be reduced and / or eliminated. Also, in this manner, the size of the load can be dynamically scalable based on various performance metrics. Dynamic scaling can prevent resource exhaustion. Additionally, automatic throttling can adapt to reductions in available resources (e.g., network bandwidth and / or computing power) at the source computing device and / or prevent real-time business incidents. In some cases, Azure is used for scalability and geographic distribution, which allows tests to take advantage of unused or "spare" bandwidth and computing power.

[0028] Advantageously, this end-to-end testing in a live network environment allows for error detection across the entire distributed computing system. Additionally, end-to-end testing in a live environment combines the real-world behavior of components in the distributed computing system with simulated network loads. For example, network loads generated using the network traffic generator service can be generated from different locations and arrive at target deployments at different times, and the network loads can also be processed through the distributed computing system in parallel with live network traffic.

[0029] Thus, an exemplary embodiment relates to a computer-implemented method for non-intrusive network load generation. The method may include determining available resources in a distributed computing system. The distributed computing system may include multiple computing devices and a target deployment. The deployment may include, for example, a cluster of computing devices, such as a storage and / or computing cluster. Based on the target deployment and the amount of available resources between multiple source computing devices, multiple source computing devices may be selected to generate a network load directed from the multiple source computing devices to the target deployment. The multiple source computing devices may include a subset of the multiple computing devices in the distributed computing system. The method may also include providing a network traffic generator service to the multiple source computing devices to generate a network load directed from the multiple source computing devices to the target deployment. The performance of the distributed computing system may be monitored, wherein the performance is responsive to the generated network load.

[0030] Another exemplary embodiment relates to one or more computer storage media having computer executable instructions implemented thereon, which when executed perform a method for non-intrusive network load generation. The method may include determining a portion of the total resources in a distributed computing system as available resources. The available resources may include computing power and network bandwidth capabilities associated with computing devices in the distributed computing system. In addition, the method may include selecting multiple source computing devices associated with the amount of available resources, wherein the amount of available resources meets a predetermined capacity threshold. A network traffic generator service may be provided to multiple source computing devices. The network traffic generator service may then be executed to generate a first network load directed to a target deployment from multiple source computing devices. The first network load may be transmitted simultaneously with ongoing network activities associated with real-time operations. Finally, the method may include monitoring the distributed computing system to evaluate the performance of the distributed computing system in response to the first network load.

[0031] In yet another embodiment, a system for non-intrusive dynamic scalable network load generation is provided. The system may include multiple source computing devices associated with computing capabilities and network bandwidth capabilities. The system may also include a target deployment. In addition, the system may include a verification server. The verification server may be configured to select multiple source computing devices to generate a network load directed to the target deployment from the multiple source computing devices. The selection of multiple source computing devices may be based on the amount of available resources between the multiple source computing devices and the target deployment. The verification server may also be configured to provide a network traffic generator service to the multiple source computing devices to generate a network load, and then monitor the performance of the system in response to the generated network load.

[0032] Having briefly described an overview of embodiments of the present invention, an exemplary operating environment in which embodiments of the present invention may be implemented is described below to provide a general context for various aspects. Figure 1 , an exemplary operating environment for implementing embodiments of the invention is shown and generally designated as computing device 100. Computing device 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention. Neither should computing device 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

[0033] Embodiments of the present invention may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions (such as program modules) executed by a computer or other machine (such as a personal digital assistant or other handheld device). Typically, a program module refers to (including routines, programs, objects, components, data structures, etc.) code that performs a specific task or implements a specific abstract data type. Embodiments of the present invention may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. Embodiments of the present invention may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including memory storage devices.

[0034] refer to Figure 1 , computing device 100 includes bus 110 that directly or indirectly couples memory 112, one or more processors 114, one or more presentation components 116, input / output ("I / O") ports 118, I / O components 120, and an illustrative power supply 122. Bus 110 represents something that may be one or more buses, such as an address bus, a data bus, or a combination thereof. Although Figure 1 The various boxes of FIG. 1 are shown with lines for clarity, but in reality, delineating the various components is not so clear, and metaphorically, the lines would more accurately be gray and fuzzy. For example, a presentation component such as a display device may be considered an I / O component. Furthermore, a processor has memory. Recognizing that this is the nature of the art, and reiterating that Figure 1 The figures are merely illustrations of exemplary computing devices that can be used in conjunction with one or more embodiments of the present invention. No distinction is made between types such as "workstation," "server," "laptop," "handheld device," as all of these types are in the same Figure 1 within the scope of and reference to a “computing device”.

[0035] The computing device 100 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 100, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0036] Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 100. Computer storage media excludes the signals themselves.

[0037] Communication media typically implements computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" refers to a signal having one or more of its characteristics set or changed in a manner that enables information to be encoded in the signal. As an example and not limitation, communication media include wired media such as a wired network or a direct wired connection and wireless media such as acoustic, RF, infrared and other wireless media. Any combination of the above should also be included in the scope of computer readable media.

[0038] Memory 112 includes computer storage media in the form of volatile and / or non-volatile memory. Memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. Computing device 100 includes one or more processors that read data from various entities such as memory 112 or I / O components 120. One or more presentation components 116 present data indications to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, etc.

[0039] I / O ports 118 allow computing device 100 to be logically coupled to other devices including I / O components 120, some of which may be built-in. Illustrative components include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless devices, and the like.

[0040] Now go to Figure 2, describes a block diagram depicting an exemplary distributed system 200 suitable for use in implementing embodiments of the present invention. Generally, system 200 shows an environment that can be used to, among other things, generate non-intrusive dynamically scalable network loads in order to verify network connections for new and / or existing cloud services. System 200 can also be used to simulate DoS attacks and evaluate system responses. Among other components not shown, system 200 can generally include a target deployment 212, a plurality of computing devices 214, a verification server 222 that can be used to verify network connections and / or simulate DoS attacks while monitoring network performance, and a data repository 232, all of which communicate with each other via network 210. Network 210 can include, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). Network 210 can also include a cloud computing network, such as a public cloud, a private cloud, and / or a dedicated cloud. Such a network environment is common in offices, enterprise-wide computer networks, intranets, and the Internet.

[0041] In some embodiments, one or more of the components and / or modules shown may be implemented as standalone applications. In other embodiments, one or more of the components and / or modules shown may be implemented as an Internet-based service and / or a module within the authentication server 222 via a computing device. Those of ordinary skill in the art will appreciate that Figure 2 The components and modules shown in are exemplary in nature and number and should not be construed as limiting. Any number of components and / or modules can be used to implement the functions described herein. For example, within the scope of an embodiment of the present invention, any number of computing devices, target deployments, verification servers, data repositories, and networks can be used in computing system 200. Each can include a single device / interface or multiple devices / interfaces that collaborate in a distributed environment. For example, multiple devices and / or modules arranged in a distributed environment can jointly provide the verification server functions described herein. As used herein, the phrase "application" or "service" can broadly refer to any software or part of a storage location that runs or accesses a computing device and / or multiple computing devices (such as multiple computing devices in a data center) on a computing device and / or multiple computing devices.

[0042] It should be understood that this and other arrangements described herein are only set forth as examples. In addition to or in place of the arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, sequences and / or functional groupings) may be used, and some elements may be omitted together. In addition, many elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components and implemented in any suitable combination and position. The various functions described herein as being performed by one or more entities may be performed by hardware, firmware and / or software. For example, various functions (including the functions described below about distributed computing system 200) may be performed by a processor executing instructions stored in a memory.

[0043] Turning now to the various components that may be included in the system 200, the target deployment 212 may include any deployment of a network infrastructure, such as a deployment of a computing cluster and / or a storage cluster. A computing cluster may include a collection of linked computing devices that provide computing capabilities and can share computing workloads. A storage cluster (which may also be referred to as a "storage stamp") may include any collection of physical drives and / or other computer-readable memories that may be coupled to one or more processors. For example, a storage cluster may include a group of nodes, a cluster or collection of racks that provide storage, or one or more data centers. The deployment of computing and / or storage capabilities may be located in a particular data center along with other devices and services. It should be noted that the various features discussed below with respect to the target deployment 212 may apply to any new and / or modified deployment, such as the deployment of computing and / or storage capabilities.

[0044] The target deployment 212 can be a new and / or existing deployment. For example, the target deployment 212 can be a storage cluster that is the target of deployment for new capabilities, network reconfiguration and / or updates, equipment replacements, software upgrades, DoS simulation testing, and any combination of these and other activities. Before rolling out a new deployment or a modification to an existing deployment (e.g., a new storage cluster or a modification to an existing storage cluster), testing can be performed to verify network connectivity for the new and / or existing deployment. For example, the testing can confirm that the target storage cluster is associated with a connection that provides sufficient bandwidth and / or throughput, and also maintains latency and / or failures within acceptable levels. The testing can also confirm that providing the connection to the target storage cluster will not interrupt real-time operations occurring on the network 210. These aspects of the testing are discussed in more detail below.

[0045] In one example, the target deployment 212 may be an existing storage cluster in a data center. In this case, the target deployment 212 may need to increase capacity, which may require additional hardware and / or software at the cluster. In order to ensure that the additional components are running correctly and the network connection provided to the storage cluster meets the increased capacity requirements, a network load test can be performed. In another case, the existing storage cluster may need to modify its hardware and / or software components, such as hardware replacement and / or software upgrade. After completing this modification, a network load test can be performed to confirm that the hardware and / or software components are running correctly. Additionally or alternatively, the target deployment 212 can be a new storage cluster. For example, a new storage cluster can be implemented at a data center to provide new capabilities. A network load test can be performed to verify the network connection for the new cluster and ensure that all components are running correctly. In addition, a new and / or existing target storage cluster can be subjected to simulated DoS attacks, and the performance of the target storage cluster can be evaluated to determine whether the target storage cluster is configured to gracefully manage a real DoS attack. As described, the above discussion also applies to other deployments within the network, such as the deployment of a computing cluster.

[0046] In an embodiment, the target deployment 212 is a dedicated storage cluster. For example, a single entity (such as a single customer) may have exclusive access to the target storage cluster, such as read and write access. In this case, all data services for this particular customer may go to one or more dedicated storage clusters. A specific level of network connection may be required to support the service. In addition, the SLA with the customer who is the end user of the storage cluster may include the terms of establishing the minimum service level to be provided at the storage cluster. For example, the SLA may provide a specific number of gigabytes per second, I / O per second, read / write per second, or any other measurement of service performance. Sufficient network connections may be required to meet the specified service level. Therefore, a network load test may be performed to verify the network connection for the dedicated storage cluster. If the target deployment 212 is a shared and / or public storage cluster, different SLAs may be applied. Tests may also be performed to verify the network connection for these arrangements.

[0047] The source of the network load for load generation described above may include any number of computing devices, such as a plurality of computing devices 214. It will be appreciated that although a plurality of computing devices 214 are shown in a cluster, in reality, these computing devices may be geographically distributed over a large area. For example, computing device 216 may be located in Chicago, IL, while computing device 218 may be located in Quincy, WA. It may be desirable to employ computing devices in various geographic locations in order to generate sufficient load for verifying network connectivity at target deployment 212. Figure 4Discuss this geographic distribution. It should also be understood that although multiple computing devices 214 are depicted as individual items and described as "one computing device" (e.g., computing device 216 and computing device 218), in reality, each depicted item may actually represent multiple computing devices. For example, computing device 216 may represent multiple computing devices. In addition, computing devices may not include separate physical devices, but may instead include an array of virtual machines. A combination of physical machines and virtual machines may be included in multiple computing devices 214.

[0048] A plurality of computing devices 214 may be associated with resources including network bandwidth capabilities and / or computing capabilities. The computing devices and associated resources may be used to generate network loads in order to test the network connections provided to the target deployment 212, and / or to test the response of the target deployment 212 to simulated DoS attacks. The network load (also referred to as network traffic load) may include network traffic based on network sessions such as I / O requests, web, email, etc. generated through multiple connections in order to utilize a predetermined amount of computing and / or bandwidth. The network load generated at the plurality of computing devices may in turn result in additional loads generated from the target deployment 212 during the test. For example, a request generated at the computing device for generating a network load may trigger a confirmation from the target deployment 212 in response to the request, such as a confirmation from a target storage cluster. As described, in order to generate sufficient network load, a plurality of computing devices may be employed. For example, a group of source computing devices 220 as a subset of the plurality of computing devices 214 may be employed. Additional details on this functionality are as follows.

[0049] The verification server 222 may include various components for implementing non-intrusive network traffic load generation to test various end-to-end scenarios. For example, the verification server 222 may include a resource confirmation component 224, a network traffic generator component 226, a monitoring component 228, and a cleaning component 230. It should be understood that Figure 2 The components 224, 226, 228, and 230 shown are exemplary in nature and number and should not be construed as limiting. The verification server 222 may implement services that support the execution of the functions of the verification server 222. As an example, a service application may provide a processing interface to send and receive service requests to internal and external components and modules. Any number of components and verification services may be used to implement the desired functionality within the scope of embodiments of the present invention.

[0050] At a high level, the validation server 222 can implement an automated service that enables multiple source computing devices, such as one or more of the multiple computing devices 214, to generate network loads targeted to the target deployment 212 or any other deployment within the network infrastructure. The validation server 222 can dynamically scale the load directed to the target deployment 212 until sufficient network connectivity at the target deployment 212 has been confirmed. Additionally or alternatively, the network load can be generated and scaled to simulate a DoS attack of a desired severity. During these various situations, the validation server 222 can monitor the performance of the distributed computing system 200 to generate performance reference points for the system 200 at various levels of network load, as well as to minimize interference and costs at various components within the system 200.

[0051] This test of network connection and / or this simulation of DoS attack can occur simultaneously with the real-time operation occurring on network 210. As used herein, phrases "real-time operation" and / or "real-time business" can refer to ongoing operations and / or business that support real-time activities (such as customer activities) on the network. In an embodiment, phrases "real-time operation" and / or "real-time business" can refer to business that is unrelated to the network traffic load generated for test purposes. "Real-time operation" and / or "real-time business" can occur simultaneously with the network traffic load generated for test purposes. Test business and real-time business can point to similar locations at the same time, and can share network pipelines. Therefore, these two types of business can occur in parallel. For example, at least one source computing device can be associated with real-time operation when the source computing device generates network load for test purposes. In addition, when generating network load for test, the network infrastructure between the source computing device and the target deployment 212 can support real-time operation. Real-time business can therefore be referred to as flowing around the target deployment 212. In this way, the test can verify the performance of the target deployment 212 and the network and other network components in an environment including real-time traffic, thereby ensuring satisfactory performance when the target deployment 212 is rolled out. At the same time, the verification server 222 can ensure that the test does not interfere with real-time operations. In an embodiment, the test is performed after the target deployment 212 has been integrated into the production environment (e.g., connected to the production system), but before it is "rolled out" (e.g., before it is released for customer use). Therefore, the test for the target deployment 212 can be described as a "pre-rollout" test.

[0052] The functionality provided by exemplary components included in authentication server 222 will now be described in greater detail.

[0053] The resource confirmation component 224 can confirm the settings and / or parameters used for the test. For example, the resource confirmation component 224 can confirm the required network connection at the target deployment 212 (for example, if the target deployment 212 is a storage cluster, the required network connection can be defined in the SLA). The total bandwidth in the target location (such as a data center) associated with the target deployment 212 can also be determined. The resource confirmation component 224 can use the network monitoring infrastructure to determine other settings and / or parameters associated with the system 200. For example, the load that a network router can handle can be determined.

[0054] In addition, resource confirmation component 224 can determine a network link associated with the location of the target deployment. For example, a network link associated with the target location where target deployment 212 is located can be identified. As shown in system 200, multiple computing devices 214 can be linked to target deployment 212 via network 210, and resource confirmation component 224 can identify them as such.

[0055] Resource verification component 224 can then detect characteristics associated with these links, such as resources. For example, bandwidth capabilities associated with the connection between the target location and each of the plurality of computing devices 214 can be determined. Computing capabilities associated with the plurality of computing devices 214 linked to the target location can also be determined.

[0056] In this manner, resource confirmation component 224 can determine the total amount of resources associated with computing devices that can be linked to target deployment 212. Resource confirmation component 224 can also determine the portion of the total amount of resources that are currently being used for real-time operations within system 200. The remainder of the resources can be considered available resources, such as resources that can be used to test a network connection at target deployment 212.

[0057] For example, resource confirmation component 224 can determine a specific bandwidth associated with a connection between computing device 216 and a target location for target deployment 212 (e.g., a data center where target deployment 212 is located). Resource confirmation component can also determine that a portion of this bandwidth is being used for real-time operations between computing device 216 and the target location. For example, 40% of the bandwidth can be used for ongoing real-time operations, such as video streaming. Resource confirmation component 224 can then determine that 60% of the total bandwidth between computing device 216 and the target location is an available resource. Resource confirmation component 224 can make similar determinations for other computing devices that have connections (such as network connections) to the location of target deployment 212.

[0058] Based on the determination of available resources, resource confirmation component 224 can select one or more computing devices to be used as source computing devices for generating network load during the test. For example, a subset of multiple computing devices 214 can be selected. In exemplary system 200, source computing device 220 is selected. This selection can be based on the discovery that source computing device 220 is associated with sufficient available resources to run the test. For example, it is assumed that target deployment 212 needs to support a specific service level (for example, as specified in the SLA), and a specific network load must be generated to test the service level. As an additional example, it is assumed that a specific network load is required to simulate a DoS attack. In either example, source computing device 220 can be selected based on the determination that the combined available resources of source computing device 220 are sufficient to generate the specific network load. For example, the selection can be based on the amount of available resources that meet a predetermined threshold (for example, based on the amount of resources required to provide the service level specified in the SLA and / or the predetermined threshold of the amount of resources required to simulate the DoS attack). In one example, a computing device associated with the maximum available bandwidth amount to the location of target deployment 212 is identified and considered for selection. In this manner, selection may be based on the absolute amount of available resources associated with the computing device. For example, available resources may need to be sufficient to meet a service level of a specific number of gigabytes per second, I / O per second, writes / reads per second, and / or any other service measurement at the target deployment 212.

[0059] Selection can also be based on the relative amount of available resources associated with the computing device. For example, a specific source computing device such as source computing device 218 can be selected based on the determination that the available portion of the total resources associated with the computing device meets a predetermined individual threshold. For example, if the predetermined threshold is set to 80%, then if the portion of the bandwidth resources available at the device is greater than or equal to 80% (or in other words, the ongoing real-time operation consumes less than or equal to 20% of the total bandwidth resources associated with the device), the computing device can only be selected as a source computing device. The resource confirmation component 224 can determine whether the computing device meets the predetermined threshold before selecting the computing device as the source computing device. In an embodiment, meeting the predetermined threshold can be a prerequisite for selection. The predetermined threshold can be set anywhere between 0% and 100% (or any similar measurement indicating a relative portion). For example, a source computing device can be selected based on a determination that no real-time operation is supported. In this way, a source computing device that generates a network load for test purposes (e.g., a network connection test or a simulated DoS test) can be separated from a computing device that supports real-time operations.

[0060] The selection of the source computing device 220 may also be based on other factors. For example, the selection may be based on a determination that a predefined portion of the available resources associated with the computing device (such as the available resources associated with all source computing devices and / or the available resources associated with a specific source computing device) can be used to generate the required network load. For example, assume that it is determined that 40% of the bandwidth resources associated with the source computing device 218 are being used to support real-time operations, and the remaining 60% of the bandwidth resources are available. In this case, it may be determined that only 50% of the available bandwidth resources (or 30% of the total bandwidth resources associated with the source computing device 218) will be used for testing purposes. The amount of available resources for testing may be limited to a predetermined threshold (e.g., 50% in the aforementioned example) to ensure that the test does not interfere with real-time operations. For example, even if there is a spike in the amount of resources consumed to support real-time operations, utilizing only a portion of the available resources may prevent such interference. Therefore, the source computing device may be selected based on the ability of the source computing device to generate a specific network load using a portion of its available resources (where the portion falls below the specified limit). As described above, the limitation can be implemented at the level of each individual device and at the level of the source location associated with multiple source computing devices. For example, the portion of the resources used for testing at a single source location can be limited at a predetermined threshold. Predefined thresholds for individual computing devices and / or source locations can be set anywhere between 0% and 100% (or any similar measurement indicating a relative portion). In an embodiment, the threshold can be an absolute measurement of the resource, such as a limit of a specific number of gigabytes per second. In another example, the threshold can correspond to the cost of the resource (e.g., the cost of network bandwidth and / or the cost of computing resources). This cost can be linear or nonlinear. For example, the cost can be determined by demand-based pricing.

[0061] Resource confirmation component 224 may also determine whether sufficient computing resources are available at a computing device that is considered to be used as a source. For example, if a computing device meets criteria related to available bandwidth, the computing device may still be rejected as a source computing device if it lacks sufficient computing resources to generate the required network load. Thus, resource confirmation component 224 may consider bandwidth capabilities, computing capabilities, and other resources associated with a computing device when selecting a source computing device.

[0062] The selection of source computing device 220 may be based on any number of other factors. For example, location constraints may be considered. If it is determined that the load test should not cross geographic region boundaries, this factor may influence the selection of source computing device 220 (e.g., in this case, only computing devices in the same geographic region as target deployment 212 may be selected as source computing devices).

[0063] In response to selecting the source computing device 220, the network traffic generator component 226 can deploy the network traffic generator service to the source computing device 220. For example, each of the source computing devices 220 can be provided with a network traffic generator service. The network traffic generator service can then be executed to generate a network load directed from each source computing device 220 (e.g., from the source computing device 218) to the target deployment 212. For example, a batch service can be called to deploy computing resources. Load generation can also include initiating data writes at the target deployment 212, such as data writes at the target storage cluster, or otherwise generating traffic in the direction of the target deployment. The dashboard can show the active jobs included in the test, and can also display information about the source computing devices selected to generate the load associated with a particular job.

[0064] In an embodiment, the network traffic generator component 226 configures a relatively small initial network load to be generated. A small initial load can minimize the impact of the test on the source computing device 220, the target deployment 212, the network 210, and any other components included in the system 200. For example, during normal operation, some source computing devices and / or target deployment components may be in sleep mode. When the initial network load causes an increase in business, these computing devices and / or target storage cluster components may wake up. During this initial time frame, some faults may occur at these devices. Starting with a minimum initial load can minimize such faults.

[0065] After the initial network load has been generated, the network traffic generator component 226 can wait for the load to stabilize. As described above, all source computing devices, target storage cluster components, and / or other components within the network infrastructure may take time to become active. One indication of load stability can be that the source computing device 220 provides consistent data over a given time period. Another indication can be the consistency of the number of observed failures. For example, as described above, if one or more source computing devices are initially in sleep mode, a certain period of time can pass before those devices become active and provide consistent data, and before the number of observed failures falls back below a predefined limit (such as a limit established in an SLA).

[0066] Throughout the test process, monitoring component 228 can monitor the performance of system 200. For example, performance can be monitored to determine whether the load has stabilized. In addition, monitoring component 228 can evaluate various performance metrics for system 200 to determine whether system 200 performs satisfactorily in response to the generated network load. Satisfactory performance can correspond to the performance of the standard specified in the SLA and / or the performance of the real-time operation that does not interfere with the ongoing operation. The SLA performance metric can also include availability, response time, channel capacity, waiting time, completion time, service time, bandwidth, throughput, relative efficiency, scalability, performance per watt, compression ratio, instruction path length and acceleration. Other variations and combinations of expected performance metrics fall within the scope of embodiments of the present invention. Performance metrics can also explain the response of the system to simulated DoS attacks. For example, performance metrics can indicate whether the network infrastructure component manages DoS attacks gracefully.

[0067] The metrics considered and interpreted by the monitoring component 228 may include metrics for the network infrastructure, metrics for the network traffic generator services (including metrics for the source computing device 220), and metrics for the target deployment 212. For example, health data for network devices (such as access routers, DC routers, and other network components) may be transmitted to the monitoring component 228 or retrieved by the monitoring component 228 to be monitored and interpreted. In addition, if a problem at a component is detected, the monitoring component 228 may receive an alert from one or more of these components. For example, if the network monitoring infrastructure detects a problem at the network (such as a network failure), the network monitoring infrastructure may generate an alert that is received by the monitoring component 228.

[0068] If it is determined that the performance metric for one or more of these components fails to meet the predetermined performance threshold and / or if the monitoring component 228 receives an alert regarding one or more of these components, a course of action can be selected based on the specific metric and / or alert that failed to be met. For example, if it is determined that there is a problem at the network infrastructure (e.g., full capacity, higher cost, insufficient throughput, and / or other network constraints), the load can be redistributed across existing source computing devices and / or across additional source computing devices (this aspect will be discussed in more detail below). If this is unsuccessful, the test can be terminated.

[0069] In another example, if an unexpected error and / or failure occurs at the target deployment 212, the test can be terminated, as will be described with respect to the cleanup component 230. The problem causing the error and / or failure can be resolved before the test is re-initiated. Specifically, the monitoring component 228 can ensure that the target deployment 212 is able to perform certain functions. For example, if the target deployment 212 is a storage cluster, the monitoring component can ensure that the storage cluster is able to write data, load balance, and perform any other functions that may be required at the storage cluster. A storage diagnostic service (not shown) can provide relevant information to the monitoring component 228.

[0070] If errors and / or failures are observed at the source computing device 220 (e.g., latency and / or throughput metrics are failure criteria specified in the SLA), one of several actions may be taken. These errors and / or failures may cause warnings. Based on monitoring, it may be determined whether additional attempts should be made to meet the performance metrics. For example, if the load is not actually stable yet (e.g., the stability determination was incorrectly made), allowing additional time to pass may produce improved performance metrics and / or the resolution of the conditions that caused the warning. If the monitoring component 228 determines that subsequent attempts should not be made and / or if repeated attempts are made and the performance does not improve, the network load may be reduced. Additionally or alternatively, the test may be terminated completely. Such termination may avoid failures in real-time operations. The termination of the test will be discussed in more detail below with respect to the cleaning component 230.

[0071] On the other hand, if it is determined that the performance of the system is satisfactory, such as when there are no warnings and / or the monitored performance metrics meet a predetermined performance threshold, it can be determined whether more load is needed. For example, if only a minimum initial load has been generated, an increased load greater than the minimum initial load may be needed. In this case, the network traffic generator component 226 can initiate the generation of additional load. This can be referred to as "expanding" the load and / or "expanding" the network traffic generator service in order to increase the network load.

[0072] Scaling the load may take various forms. In some cases, this may involve increasing the load generated from existing source computing devices. For example, the load generated from source computing device 220 may be increased. Additionally or alternatively, this may include employing additional devices as source computing devices. If it is determined that the increased load cannot be generated from the existing source computing devices without compromising the performance of system 200, additional devices may be employed. In this case, the load generated by the existing source computing devices 220 may remain unchanged, but the increased load may be generated by adding more source computing devices. This may be done by adding more computing devices to the system 200. Figure 3, where the number of source computing devices 310 is greater than the previously selected source computing devices 220. (It should be noted that throughout this set of figures, the same reference numerals are used to indicate similarly configured components in the various views; however, some aspects of these components sharing the same reference numerals may vary according to embodiments of the invention.) In this case, the network traffic generator service can be deployed to additional source computing devices to generate additional network load from the additional source computing devices. It should be noted that the additional source computing devices can be physical machines and / or virtual machines. The additional virtual machines can be existing virtual machines and / or newly provided virtual machines. A combination of these activities (e.g., increasing the load generated by existing devices and / or adding new devices, such as adding existing physical devices, adding existing virtual machines, and / or providing new virtual machines) can be used to increase the network load generated for testing purposes.

[0073] After the load increases, the network traffic generator component 226 can wait for a period of time to pass again to allow the increased load to stabilize. Then, the monitoring component 228 can monitor and / or evaluate various performance metrics to determine whether the system 200 meets the predetermined performance threshold, and the above process can be repeated. Specifically, the process can be repeated in the case of increasing the network load until the generated network load is large enough to verify the network connection required at the target deployment 212, and / or simulate a DoS attack of the desired scale. For example, the process can be repeated until it has been determined that the network connection provided to the target storage cluster is sufficient to meet the predefined service level, such as the standard specified in the SLA. The network load can be increased incrementally. These increments can be based on the performance of the distributed computing system in response to the generated network load.

[0074] After having verified the expected level of network connectivity and / or having simulated a DoS attack, having found that the performance of the system 200 meets the performance threshold, and the test has been successfully completed, the test can be terminated. Termination will be discussed in more detail below with respect to the cleanup component 230.

[0075] When scaling the load according to one or more of the above-mentioned processes, the monitoring component 228 can continuously monitor the performance of the system 200. Based on the monitored performance, the load generated from a specific source computing device can be adjusted. For example, it can be determined that a specific source computing device is not performing well in response to an increased load. Additionally or alternatively, it can be determined that network load generation is consuming too many resources (e.g., the amount of resources exceeding the resource threshold discussed previously). Based on this determination, the load generated from the source computing device can be reduced and / or eliminated, and the required load can be generated at another existing and / or new source computing device instead. Therefore, the source computing device for the purpose of testing can be dynamically selected throughout the test process. In addition, although the overall goal can be to continuously increase the total network load generated for the test, the load can be reduced between some test iterations. For example, when a failure and / or a reduction in the amount of available resources at the source computing device is observed, the total network load can be reduced before increasing the total network load again. In this way, the generated load can be dynamically scaled based on real-time monitoring of the performance of the system 200, including dynamic scaling that can be based on the performance of a specific system component.

[0076] For example, it is assumed that 90% of the total bandwidth resources of a specific computing device are available at the beginning of the test. Based on the available resources at this level, a computing device can be selected as a source computing device. But it is assumed that after the test begins, the real-time traffic supported by the specific source computing device increases, so that only 50% of its bandwidth resources can be used for testing. Therefore, continuing to utilize the resources associated with the computing device for testing may interfere with the real-time operations supported by the computing device. Therefore, the network load generated by the source computing device can be reduced or even eliminated to avoid interfering with real-time operations. Similarly, in response to errors and / or failures associated with the device, the network load at a specific source computing device can be reduced and / or eliminated. The reduction and / or elimination of the generated load can be compensated by increasing the load generated by another source computing device and by adding a new source computing device. These examples illustrate the aspects of the dynamics associated with the network load test provided by an embodiment of the present invention.

[0077] As described above, termination may be triggered by a performance metric that fails to meet a predetermined performance threshold as well as successful verification of network connectivity at an expected level. In either case, such termination may include terminating the network traffic generator service and triggering the actions of cleanup component 230. Cleanup component 230 may be configured, such as by cleaning up Figure 2 The source computing device 220 (and Figure 3The cleanup component 230 may clean up the environment by removing the virtual machine setup and generally undoing the actions taken to initiate the test. Batch jobs may be deleted, and test artifacts at the target deployment 212 may also be deleted. The cleanup component 230 may also collect all logs associated with the test, create a report of the results, and email any documents associated with the test as needed. The cleanup component 230 may take other actions associated with ending the test process.

[0078] As previously described, throughout the above-described testing process, the monitoring component 228 can monitor the performance of the system 200. In addition to monitoring the system to evaluate system performance and dynamically scaling the generated network load, the monitoring component 228 can monitor the system 200 to capture reference points at the various network loads generated. The reference point may correspond to a snapshot of the system 200 at a specific network load. For example, when a network load of 20 gigabits per second is generated, the reference point for the network load of 20 gigabits per second may include performance metrics for various components within the system 200. If the network load increases to 25 gigabits per second, a reference point corresponding to a load of 25 gigabits per second may be generated to provide a snapshot of the performance of the system 200 at a load of 25 gigabits per second. In this way, the reference point may correspond to a specific network load, and the reference point may be generated and / or captured at multiple different network loads. In an embodiment, the reference point provides an overview of the performance of the system 200. In another embodiment, a reference point is generated for each component in the system 200 at a specific network load so that the performance of each component is captured. The performance metrics captured by the reference points may include availability, response time, channel capacity, latency, completion time, service time, bandwidth, throughput, relative efficiency, scalability, performance per watt, compression ratio, instruction path length, and acceleration. Thus, the reference points may provide an overall picture of what is happening within the system 200 at a particular load.

[0079] These reference points can be beneficial for a number of reasons. For example, at a particular network load, the performance of the system 200 may meet a particular performance threshold, such as an SLA-based threshold, but in order to achieve this, various components within the system may be pushed to undesirable limits. For example, if 90% of the available resources are used to meet a performance threshold for a network connection, this may be undesirable and corrective action may be taken. Additionally or alternatively, in such a situation, it may be determined that current SLAs should be re-evaluated and / or that a certain SLA cannot be offered to new customers because a failure may occur. Reference points may also be useful because they may indicate the performance of the system 200 at various network loads. This may be used for planning purposes, as well as for predicting future system behavior.

[0080] Turning now to the data repository 232, as described, the plurality of computing devices 214, the target deployment 212, and the verification server 222 can all communicate with the data repository 232 via the network 210. The data repository 232 can be configured to store information for use by, for example, the verification server 222. The information stored in association with the data repository 232 can be configured to be searchable for information stored in association therewith. The information stored in association with the data repository 232 can include general information used by the verification server 222, such as information about the network infrastructure (e.g., settings and / or parameters associated with the system 200, such as settings and / or parameters verified by the resource verification component 224), information from an SLA associated with the target deployment 212, reference points generated at various network loads, performance thresholds to which the monitoring component 228 can compare performance metrics for the system 200, test results and / or logs, reports (e.g., reports generated by the cleanup component 230), and any other information related to the system 200, verification of network connectivity at the target deployment 212, and / or simulation of DoS attacks. The content and amount of such information in data repository 232 is not intended to limit the scope of embodiments of the present invention in any way.In addition, although shown as a single independent component, data repository 232 may in fact be multiple storage devices, such as a database cluster.

[0081] Now go to Figure 4 , an illustrative geographic region 400 including a collection of geographic locations is shown. Figure 4 Discuss distributed computing systems such as Figure 2 As shown in geographic region 400, target deployment 410 is associated with a geographic location. Target deployment 410 can be any network infrastructure deployment, such as Figure 2 4. The target deployment 212 of FIG. 400 is a diagram of a deployment of computing and / or storage capabilities as described in FIG. 412. Multiple computing devices that may represent potential source computing devices may be associated with multiple other locations such as source locations 412, 414, 428, 430, 432, and 434. For example, computing device 436 is associated with source location 434. In an embodiment, the source location represents a data center. Geographic area 400 is exemplary only, and it will be understood that other configurations including other source locations and / or other arrangements of computing devices are included within the scope of the present invention.

[0082] The arrows between each source location and target deployment 410 represent resources associated with the link between the two. For example, an arrow may represent a bandwidth capability associated with a link between a source location and a target deployment 410. Consider, for example, an arrow between source location 414 and target deployment 410. The arrow may represent a bandwidth capability 416. The width 418 of the arrow may indicate the total amount of bandwidth capability associated with the link between source location 414 and target deployment 410. As indicated by the arrows having varying widths, the bandwidth capabilities associated with the links between various source locations and target deployments 410 may vary. For example, the width of the arrow between source location 414 and target deployment 410 may indicate that the bandwidth capability 416 associated with source location 414 is greater than the bandwidth capability 438 associated with source location 412, compared to the width of the arrow between source location 412 and target deployment 410. Bandwidth capabilities 440, 442, 444, and 446 associated with source locations 428, 430, 432, and 434, respectively, may also be compared in this manner. It should be understood that this configuration is merely exemplary, and is intended merely to illustrate the manner in which varying resource capabilities may be associated with different source locations.

[0083] The shadow of the arrow associated with the source location can indicate the allocation of the total resources associated with these locations. For example, the total width 418 of the arrow between the source location 414 and the target deployment 410 can represent the total amount of bandwidth capabilities associated with the link. The solid area can represent the bandwidth used to support the real-time operation 420. Therefore, the remaining width 426 (for example, the difference between the total width 418 and the width of the solid area corresponding to the real-time operation 420) can represent the relative amount of available resources, such as available bandwidth. A portion of these available resources can be used for network connection verification at the target deployment 410, as described above. The shaded area can represent a portion of the resources used as network connection and / or DoS simulation test resources 422, such as bandwidth capabilities. The remaining white area can represent the available resources 424 after considering the bandwidth consumed by both the real-time operation 420 and the test 422.

[0084] Any number of computing devices at any number of source locations may be used to perform network connectivity verification. Generating sufficient network load may include employing computing devices associated with a plurality of different geographic locations, such as one or more of the source locations included in geographic area 400. Network load may be initiated dynamically from the source location based on network links and available resources, thereby utilizing spare resource capacity for verification. This may minimize overall cost in a distributed computing system.

[0085] Steering Figure 5, exemplary flow chart 500 illustrates various steps and decision points included in an exemplary process for generating network traffic load in a distributed computing system. The process may begin at block 510. At block 512, test parameters may be verified and available resources within the distributed computing system may be determined. For example, as described with respect to Figure 2 These steps are performed as described in resource confirmation component 224 of the distributed computing system. Based on this determination, a source computing device can be selected in block 514. For example, the source computing device can be selected based on the amount of available resources between the target deployment and the source computing device. The source computing device can be a subset of the total number of computing devices included in the distributed computing system.

[0086] At block 516, a network traffic generator service may be provided to the source computing device. The network traffic generator service may be used to generate a network load at block 518, wherein the network load is directed from the source computing device to the target deployment. At block 520, a period of time may be allowed to pass so that the network load stabilizes. For example, time may be allowed to pass until the source computing device provides consistent data and errors are observed at a consistent rate. These steps may be as described with respect to Figure 2 The network traffic generator component 226 is executed as described above.

[0087] The distributed computing system may be monitored during the above steps, and in block 522, in particular, its performance may be monitored to capture reference points and / or determine how the distributed computing system responds to the generated network load. This may be as described with respect to Figure 2 The method may be implemented as described above with respect to the monitoring component 228 of the distributed computing system, and may include receiving an alert regarding the performance of the network components. At block 524, a determination is made as to whether the performance of the distributed computing system meets a predetermined performance threshold. This may be accomplished by comparing the performance metrics for the various components of the distributed computing system to the performance threshold for each respective component. Additionally or alternatively, this may be accomplished based on whether an alert is received indicating a problem at one or more of the components.

[0088] If it is found at block 524 that the performance does not meet the performance threshold, then at block 526, it may be determined whether another attempt to meet the performance threshold should be made. For example, if an error is observed within the network, corrective action may be taken at block 528 (e.g., the network load may be redistributed), and the process may return to block 520, where time is allowed for the redistributed load to stabilize. The corrective action at block 528 may also include reducing the generated network load, de-provisioning and / or eliminating the source computing device so that it no longer generates network traffic for testing purposes, and any other corrective actions that may be required. In some cases, positive corrective measures may not be required. For example, if an error is observed at the network traffic generator service and / or the source computing device, the process may simply return to block 520 because there may not be enough time allowed for the load to initially stabilize. Additional time may allow the load to stabilize, and the observed errors may be reduced and / or eliminated. If the load redistribution and / or repeated attempts are unsuccessful, then at block 526, it may be determined that no additional attempts are made to meet the performance threshold, and the network load may be reduced, and / or the test may be terminated completely at block 534. Similarly, the decision of block 526 may be made to terminate the test if an unexpected error is observed at the target storage cluster. In this case, load redistribution and / or additional time may not be able to resolve the observed problem.

[0089] If, at box 524, it is found that the performance meets the performance threshold, the process can proceed to box 530, where it can be determined whether more network load is required. If the current network load is insufficient to verify the required network connectivity at the target deployment and / or simulate a DoS attack of the desired severity, the query can be answered in the affirmative. In this case, it can also be determined at box 532 whether the current source computing device is associated with resources sufficient to generate the increased network load. If so, the process can return to box 518 to generate the increased network load using existing source computing devices. This can include redistributing the load among existing source computing devices. If it is determined at box 532 that the existing source computing devices are not associated with resources sufficient to generate the increased network load, additional source computing devices can be selected and / or provisioned at box 514. The network traffic generator service can be deployed to these additional devices. Therefore, as described in reference Figure 2 As explained, generating increased network load may include any combination of increasing the load generated by existing source computing devices, adding new source computing devices to generate additional network load, redistributing load among source computing devices, and any other manner of dynamically initiating and / or scaling load.

[0090] At block 530 , it may be determined that no increased network load is required. This determination may be based on successful verification of network connectivity at the target deployment, and distributed computing system performance meeting a performance threshold. In the event of a successful test, the test may be terminated at block 534 .

[0091] Terminating the test and cleaning the environment at block 534 may include Figure 2 The activities described by the cleanup component 230 of the embodiment of the present invention can be performed. For example, the batch job can be deleted, the test artifacts at the target storage cluster can be cleaned up, a report can be generated, and the documents associated with the test can be distributed, such as by email. The process can end at box 536.

[0092] Now go to Figure 6 , a flowchart illustrating an exemplary method 600 for performing non-intrusive network load generation is shown. The method may be performed in a distributed computing system including multiple computing devices and a target deployment such as a storage cluster and / or a computing cluster. In box 610, available resources in the distributed computing system may be determined. The available resources may include computing power and network bandwidth capabilities associated with multiple computing devices in the distributed computing system. In box 612, multiple source computing devices may be selected to generate a network load directed to the target deployment from multiple source computing devices. The multiple source computing devices may be a subset of multiple computing devices included in the distributed computing system. The selection of the source computing device may be based on the amount of available resources between the target deployment and the multiple source computing devices. In one example, the selection includes determining that the amount of available resources between the multiple source computing devices and the target deployment meets a predetermined threshold. In an embodiment, the target deployment is a target storage cluster, and the predetermined threshold may be based on a service level included in an SLA with an end user of the target storage cluster.

[0093] At block 614, a network traffic generator service may be provided to a plurality of source computing devices to generate a network load directed from the plurality of source computing devices to a target deployment. In some cases, an individual network load directed from one of the plurality of source computing devices to the target deployment uses a predefined portion of an individual amount of available resources between one of the plurality of source computing devices and the target deployment. In another example, it is determined that an available portion of individual resources between each of the plurality of source computing devices and the target deployment satisfies a predetermined individual threshold. At least one of the plurality of source computing devices may be associated with real-time operation when the plurality of source computing devices generate the network load.

[0094] The performance of the distributed computing system may be monitored at block 616. Specifically, the performance of the distributed computing system in response to the generated network load may be monitored.

[0095] Exemplary method 600 may also include, for example, dynamically scaling a network load directed to a target deployment from multiple source computing devices. In some cases, dynamically scaling a network load includes configuring an initial minimum load to be generated by multiple source computing devices, wherein the initial minimum load is directed to the target deployment from multiple source computing devices. It can then be determined that the initial minimum load is stable. Performance metrics for a distributed computing system can be monitored, and based on those performance metrics meeting predetermined performance criteria, a network traffic generator service can be expanded to generate an increased network load directed to the target deployment from multiple source computing devices. The increased network load can be greater than the initial minimum load. In some cases, a network traffic generator service can be expanded until the network load directed to the target deployment from multiple source computing devices meets a predetermined threshold associated with a predefined service level. On the other hand, if the performance metric fails to meet the predetermined performance criteria, the network traffic generator service can be terminated based on the failure to meet.

[0096] In another example, dynamically scaling the network load may include configuring an initial minimum load to be generated by a plurality of source computing devices, wherein the initial minimum load is directed from the plurality of source computing devices to a target deployment. The initial minimum load may be determined to be stable. Performance metrics for the distributed computing system may be monitored, and based on the performance metrics satisfying predetermined performance criteria, network traffic generator services may be provided to additional source computing devices to generate additional network load directed from the additional source computing devices to the target deployment. In some cases, providing the network traffic generator services to the additional source computing devices may include provisioning virtual machines and providing the network traffic generator services to the provisioned virtual machines.

[0097] In another example, dynamically scaling the network load may include determining that at least one of the plurality of source computing devices will generate a reduced network load directed to the target deployment. The determination may be based on the fact that resources associated with at least one of the plurality of source computing devices are needed to support real-time operations.

[0098] Figure 7 A flow chart illustrating an exemplary method 700 for performing non-intrusive network load generation is shown in FIG. At block 710, a portion of total resources in a distributed computing system as available resources may be determined. Available resources may include computing power and network bandwidth capabilities associated with computing devices in the distributed computing system. The determination of available resources may be based on a difference between total resources associated with the computing device and an amount of resources used for real-time operations.

[0099] At block 712, a plurality of source computing devices associated with an amount of available resources that satisfies a predetermined capacity threshold may be selected. At block 714, a network traffic generator service may be provided to the plurality of source computing devices. At block 716, the network traffic generator service may be executed to generate a network load, such as a first network load, directed from the plurality of source computing devices to a target deployment. Specifically, the target deployment may be a target storage cluster and / or a target computing cluster. The first network load may be generated and / or transmitted simultaneously with ongoing network activity associated with real-time operations.

[0100] At block 718, the distributed computing system may be monitored. Monitoring may include evaluating the performance of the distributed computing system in response to the network load. Monitoring the distributed computing system may include monitoring network infrastructure metrics, network traffic generator service metrics, and target deployment metrics. Specifically, monitoring the network infrastructure metrics may include receiving a warning that the first network load should be reallocated. Monitoring may also include generating reference points at various network loads, wherein the reference points correspond to performance metrics for the distributed computing system.

[0101] In an embodiment, method 700 may also include determining whether the performance of the distributed computing system meets a predetermined performance threshold. Based on determining that the performance meets the threshold, the network traffic generator service may be expanded to generate a second network load directed from the plurality of source computing devices to the target deployment. The second network load may be greater than the first network load. Conversely, based on determining that the performance fails to meet the predetermined performance threshold, the network traffic generator service may be terminated.

[0102] As can be seen from the foregoing, embodiments of the present invention are well suited to achieve all of the intentions and purposes set forth above as well as other advantages that are obvious and inherent to the structure. It should be understood that certain features and sub-combinations are useful and can be used without reference to other features or sub-combinations. This is encompassed by and within the scope of the claims.

Claims

1. A computerized system comprising: one or more computer processors; as well as A computer memory storing computer usable instructions that, when used by the one or more computer processors, cause the one or more computer processors to perform operations comprising: generating, using a network traffic generator service in a source computing device, a first network load directed from the source computing device to a target deployment, wherein the source computing device is included in a group of source computing devices, wherein the group of source computing devices is identified based on network bandwidth capabilities from the group of source computing devices to the target deployment, wherein the set of source computing devices is associated with an amount of unused resources that satisfies a predetermined capacity threshold for the target deployment, wherein the predetermined capacity threshold for the amount of unused resources is defined based on a network bandwidth capability and a computing capability used to determine that the amount of unused resources satisfies the predetermined capacity threshold; and transferring the first network load from the source computing device to the target deployment; The target deployment includes a cluster of computing devices.

2. The system of claim 1 , wherein the identified set of source computing devices is also identified based on resources in a distributed computing system, the distributed computing system comprising the set of source computing devices and the target deployment, the resources comprising used resources operating on real-time services and unused resources that can be used to generate test services.

3. The system of claim 1, wherein the operations further comprise: The generation of network load is dynamically terminated via the network traffic generator service based on the predetermined capacity threshold, the predetermined capacity threshold being defined based on network bandwidth and computing power.

4. A system according to claim 1, wherein the first network load satisfies a predetermined threshold to test a service level specified in a service level agreement (SLA), wherein the target deployment is a target storage cluster, and the predetermined threshold is based on the service level associated with an end user of the target storage cluster included in the SLA.

5. The system of claim 1, the operations further comprising transmitting a performance metric, wherein the performance metric includes an indication of whether the performance metric is met.

6. The system of claim 5, wherein the performance metric is transmitted to cause monitoring of the performance metric associated with the first network load, wherein the performance metric is associated with an expanded or contracted network load generated via the network traffic generator.

7. The system of claim 1, the operations further comprising dynamically scaling the first network load from the source computing device.

8. One or more computer storage media having computer executable instructions embodied thereon, which when executed by a computing system having a processor and a memory, cause the processor to perform operations comprising: generating, using a network traffic generator service in a source computing device, a first network load directed from the source computing device to a target deployment, wherein the source computing device is included in a group of source computing devices, wherein the group of source computing devices is identified based on network bandwidth capabilities from the group of source computing devices to the target deployment, wherein the set of source computing devices is associated with an amount of unused resources that satisfies a predetermined capacity threshold for the target deployment, wherein the predetermined capacity threshold for the amount of unused resources is defined based on a network bandwidth capability and a computing capability used to determine that the amount of unused resources satisfies the predetermined capacity threshold; as well as transferring the first network load from the source computing device to the target deployment, The target deployment includes a cluster of computing devices.

9. The medium of claim 8, wherein the identified set of source computing devices is also identified based on resources in a distributed computing system, the distributed computing system comprising the set of source computing devices and the target deployment, the resources comprising used resources operating on real-time services and unused resources that can be used to generate test services.

10. The medium of claim 8, the instructions further comprising: The generation of the network load is dynamically terminated via the network traffic generator service based on the predetermined capacity threshold, the predetermined capacity threshold being defined based on network bandwidth and computing power.

11. The medium of claim 8, wherein the first network load satisfies a predetermined threshold to test a service level specified in a service level agreement (SLA), wherein the target deployment is a target storage cluster, and the predetermined threshold is based on the service level associated with an end user of the target storage cluster included in the SLA.

12. The medium of claim 8, the instructions further comprising transmitting a performance metric, wherein the performance metric includes an indication of whether the performance metric is met.

13. The medium of claim 12, wherein the performance metric is transmitted to cause monitoring of the performance metric associated with the first network load, wherein the performance metric is associated with an expanded or contracted network load generated via the network traffic generator.

14. The medium of claim 8, the operations further comprising dynamically scaling the first network load from the source computing device.

15. A computer-implemented method comprising: generating, using a network traffic generator service in a source computing device, a first network load directed from the source computing device to a target deployment, wherein the source computing device is included in a group of source computing devices, wherein the group of source computing devices is identified based on network bandwidth capabilities from the group of source computing devices to the target deployment, wherein the set of source computing devices is associated with an amount of unused resources that satisfies a predetermined capacity threshold for the target deployment, wherein the predetermined capacity threshold for the amount of unused resources is defined based on a network bandwidth capability and a computing capability used to determine that the amount of unused resources satisfies the predetermined capacity threshold; as well as transferring the first network load from the source computing device to the target deployment, The target deployment includes a cluster of computing devices.

16. A method according to claim 15, wherein the identified set of source computing devices is also identified based on resources in a distributed computing system, the distributed computing system includes the set of source computing devices and the target deployment, and the resources include used resources operating on real-time business and unused resources that can be used to generate test business.

17. The method according to claim 15, further comprising: The generation of network load is dynamically terminated via the network traffic generator service based on the predetermined capacity threshold, the predetermined capacity threshold being defined based on network bandwidth and computing power.

18. A method according to claim 15, wherein the first network load meets a predetermined threshold to test a service level specified in a service level agreement (SLA), wherein the target deployment is a target storage cluster, and the predetermined threshold is based on the service level associated with an end user of the target storage cluster included in the SLA.

19. The method of claim 15, further comprising transmitting a performance metric, wherein the performance metric includes an indication of whether the performance metric is met.

20. The method of claim 19, wherein the performance metric is transmitted to cause monitoring of the performance metric associated with the first network load, wherein the performance metric is associated with an expanded or contracted network load generated via the network traffic generator.

Images