A provision, deployment, performance-test, and retirement system and method therefor

Abstract

In some embodiments, a system includes a processor; and a non-transitory computer readable medium coupled to the processor, the non-transitory computer readable medium including code that: provisions a quality assurance environment for use in execution of a performance test on an application under test, the quality assurance environment being configured to be provisioned on demand; provisions quality assurance environment servers for use by the quality assurance environment to execute the performance test; deploys the application under test to the quality assurance environment servers of the quality assurance environment; executes the performance test on the application under test; and retires the quality assurance environment servers utilized by the quality assurance environment upon completion of the performance test.

Description

A PROVISION, DEPLOYMENT, PERFORMANCE-TEST, AND RETIREMENT SYSTEM AND METHOD THEREFOR BACKGROUND

[0001] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventor(s), to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0002] Performance testing of software applications is an indispensable part of the software development process. The software development process generally requires the non-functional testing of software applications to ascertain the speed, stability, and scalability of the software applications under extreme testing conditions in a production-like environment. In traditional software development processes, performance testing is generally performed at the end of the software development cycle and utilizes a vast number of servers and resources. Utilization of a vast number of servers and resources late in the software development lifecycle is generally cost prohibitive and time consuming and may even delay the release of software. Therefore, reducing the server footprint associated with performance testing in the software development process is of upmost importance in providing quality software to a client.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] FIG. 1 illustrates a block diagram of a network infrastructure in accordance with some embodiments.

[0004] FIG. 2 illustrates a block diagram of a continuous integration / continuous deployment(CI / CD) management system in accordance with some embodiments.

[0005] FIG 3 illustrates a block diagram of a provisioning, deployment, performance testing, and retirement (PDPR) framework unit of the CI / CD management system in FIG. 2 in accordance with some embodiments.

[0006] FIG. 4 is a flow diagram illustrating a PDPR method in accordance with some embodiments.DETAILED DESCRIPTION

[0007] FIG. 1 illustrates a network infrastructure 100 in accordance with some embodiments. In some embodiments, network infrastructure 100 includes a network 192, a container management system 160, a continuous integration / continuous deployment (CT / CD) management system 170, a source code management system 180, servers 150, and a client device 190. In some embodiments, the network infrastructure 100 is configured to utilize a provisioning, deployment, performance testing, and retirement (PDPR) framework unit (described further herein with reference to FIG. 1 - FIG. 4) at CI / CD management system 170 to perform and / or manage provisioning operations, deployment operations, performance testing operations, and retirement operations for network infrastructure 100.

[0008] In some embodiments, source code management system 180 is a server configured to operate as a version control system and provide a platform for the maintenance, collaborative development, and modifications of software applications. In some embodiments, source code management system 180 is configured to serve as a storage and management platform for source code repositories, e g., repository / s 181 . Tn some embodiments, the source code management system 180 is configured to manage the storage and retrieval of code content and interact with client device 190 and the CT / CD management system 170. In some embodiments, source code management system 180 may utilize, for example, Stash, GIT or BITBUCKET, which arecommercially available version control software applications. For example, in GTT, a working directory may be utilized to represent a repository of repositories 181 that maintains a history of code updates and tracks revision metadata. In the context of source code management system 180, a stash is a temporary storage area where changes may be saved to a working directory without committing modifications to a main branch or repository. Stashing modifications allow developers to switch to a different branch or work on a different task without losing previous versions. In some embodiments, a "stash location path" refers to a specific location or directory where the stashed performance test script (e.g., Gatling scripts) are stored within the version control system (e.g., source code management system 180). In some embodiments, a stash location path may vary depending on the configuration and setup of the CI / CD management system 170 and / or source code management system 180 being utilized.

[0009] In some embodiments, CI / CD management system 170 is a server that includes executable code that, in addition to being configured to automate the process of integrating code updates in software applications and delivering or deploying the software applications, is configured to perform PDPR operations described herein with reference to FIG. 1 - FIG. 4. For example, in some embodiments, CI / CD management system 170 may be configured to utilize a quality assurance environment server retirement pipeline job to provision a quality assurance environment and retire a quality assurance environment infrastructure associated with the quality assurance environment after execution of a performance test on an application under test (described further in detail herein).

[0010] In some embodiments, a quality assurance environment may be a target environment generated by, for example, CI / CD management system 170, that is specifically configured to perform PDPR operations described herein, such as, for example, retiring a quality assuranceenvironment infrastructure or portions thereof after execution of a performance test on an application under test. In some embodiments, the quality assurance environment may include the quality assurance environment infrastructure and the PDPR applications and configurations used to implement the PDPR operations. For example, in some embodiments, the quality assurance environment includes, in addition to a provision, deployment, performance test, and server retirement configuration, the servers utilized to perform a performance test, a server configuration, an application deployment configuration, a container pod and container configuration, and an automated test configuration for an application under test. In some embodiments, CI / CD management system 170 may include a CI / CD management system such as, for example, Jenkins or TravisCI, which offer configuration and setup for automating the build, test, and deployment process.

[0011] In some embodiments, the quality assurance environment infrastructure is the infrastructure associated with the quality assurance environment that is utilized by PDPR framework unit 240 to perform the PDPR operations described herein. In some embodiments, the quality assurance environment infrastructure may include infrastructure, such as, for example, quality assurance environment servers 152 (e.g., designated quality assurance environment servers 152 for provisioning of container pods and containers for deployment of an application under test), container pods and containers that have been provisioned as part of the provisioning operations performed by PDPR framework unit 240, and / or other hardware / software applications utilized as part of the quality assurance environment during the performance testing of an application under test.

[0012] In some embodiments, servers 150 are a plurality of servers that may be utilized for PDPR operations. For example, in some embodiments, servers 150 may be utilized forprovisioning and deployment operations in the quality assurance environment. Tn some embodiments, server 150 may be utilized for the provisioning of container pods and containers by, for example, container management system 160, for the deployment of an application under test (described further in detail herein). In some embodiments, the servers 150 may be utilized for performance testing in the network infrastructure 100. In some embodiments, quality assurance environment servers 152 are servers provisioned from servers 150 that have been selected as part of a quality assurance environment for deployment of an application under test, performance testing, and / or server retirement.

[0013] In some embodiments, container management system 160 is a server that includes executable code configured to function as a platform for network infrastructure 100 to enable the management and orchestration of servers (e.g., servers 150) and containers and container pods within the quality assurance environment, as well as the deployment of an application under test. In some embodiments, container management system 160 is configured to control (at the direction of CI / CD management system 170) servers 150 and associated containers and container pods, including tasks like creating, starting, stopping, scaling, and monitoring the servers, containers and container pods. In some embodiments, container management system 160 may utilize, for example, Docker or Kubernetes, in the implementation of embodiments described herein and provide an ecosystem for building, deploying, and managing containers and container pods utilized by network infrastructure 100.

[0014] Tn some embodiments, a pod or container pod is a higher-level abstraction used by container management system 160. In some embodiments, a pod is the smallest deployable unit in container management system 160 and represents a group of one or more tightly coupled containers that share the same namespace and may communicate with each other using a localhost. Tn some embodiments, containers within a pod may be scheduled and executed collectively on a same host, allowing the containers to communicate via inter-process communication mechanisms like shared volumes or network ports.

[0015] In some embodiments, a namespace is a virtual partition or logical boundary that provides a manner to segregate resources and objects within a cluster. In some embodiments, namespaces allow different teams or projects to use container management system 160 while maintaining isolation and separation of resources. In some embodiments, namespaces are used to organize and categorize container management system 160 resources, such as pods, services, deployments, and replica sets, into distinct groups. In some embodiments, each namespace may create a unique scope for resource names, such that resource names are unique within a namespace but may be repeated across different namespaces.

[0016] In some embodiments, a time series database, such as, for example, InfluxDB, may be utilized to monitor and analyze performance metrics throughout the software development and deployment process. In some embodiments, by using a time series database like InfluxDB, the CI / CD management system 170 may store and analyze historical and real-time metrics.

[0017] In some embodiments, the network 192, container management system 160, CI / CD management system 170, source code management system 180, and client device 190 are collectively configured to communicate operationally to automate the process of integrating code modifications and updates (e.g., source code modifications / updates) and deploy software applications (e g., applications under test) based upon the modified / updated source code according to the PDPR operations described herein. In some embodiments, in addition to performing traditional CI / CD operations, CI / CD management system 170 is configured to utilize a PDPR framework unit (described in further detail with reference to FIG. 2 - FIG. 4) to performPDPR operations that allow for the performance testing and server retirement operations described herein.

[0018] FIG. 2 illustrates a block diagram of an exemplary CI / CD management system 170 for implementing embodiments consistent with the present disclosure. In some embodiments, CI / CD management system 170 includes an input / output (IO) interface 201, processor / s 202, a storage interface 204, a network interface 203, and memory 205. In some embodiments, CI / CD management system 170 may be, for example, a server utilized to implement the embodiments described herein. In some embodiments, memory 205 may include an operating system (OS) 207, processes 220, and PDPR framework unit 240. In some nonlimiting embodiments or aspects, the PDPR framework unit 240 is configured to implement a PDPR method (e g., PDPR method 400 of FIG. 4) to implement embodiments consistent with the present disclosure.

[0019] In some embodiments, processors 202 may comprise at least one data processor for executing program components for dynamic resource allocation at run time. The processors 202 may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc. In some embodiments, the processors 202 may be disposed in communication with one or more input / output (I / O) devices (not shown) via an VO interface 201. The VO interface 201 may employ communication protocol s / methods such as, without limitation, audio, analog, digital, monoaural, RCA, stereo, IEEE-1394, serial bus, universal serial bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, digital visual interface (DVT), high-definition multimedia interface (HDMi), RF antennas, S-Video, VGA, IEEE 802.1 n / b / g / n / x, Bluetooth®, cellular (e.g., code-division multiple access (CDMA), high-speed packetaccess (HSPA+), global system for mobile communications (GSM), long-term evolution (LTE), WiMax®, or the like), etc.

[0020] In some embodiments, using the I / O interface 201, the CI / CD management system 170 may communicate with one or more I / O devices. For example, an input device (not shown) may be an antenna, keyboard, mousejoystick, (infrared) remote control, camera, card reader, fax machine, dongle, biometric reader, microphone, touch screen, touchpad, trackball, stylus, scanner, storage device, transceiver, video device / source, etc. An output device (not shown) may be a printer, fax machine, video display (e.g., cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), plasma, Plasma display panel (PDP), Organic light-emitting diode display (OLED) or the like), audio speaker, etc.

[0021] In some embodiments, the processors 202 may be disposed in communication with a communication network (e.g., network 192) or other type of network via a network interface 203. The network interface 203 may communicate with the communication network. The network interface 203 may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / Intemet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc. In some embodiments, the communication network may include, without limitation, a direct interconnection, e-commerce network, a peer to peer (P2P) network, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the internet, Wi-Fi®, etc. In some embodiments, using the network interface 203 and the communication network (e.g., network 192), the CI / CD management system 170 may communicate with other systems (e.g., servers, etc.) in network infrastructure 100. For example, in some embodiments, using the network interface 203 and network 192, the CI / CD management system 170 may communicate with acontainer management system 160, source code management system 180, servers 150, and client device 190 in network infrastructure 100.

[0022] In some non-limiting embodiments or aspects, the processors 202 may be disposed in communication with a memory 205 (e.g., RAM, ROM, etc.) via a storage interface 204. In some embodiments, the storage interface 204 may connect to memory 205 including, without limitation, memory drives, removable disc drives, etc., employing connection protocols such as serial advanced technology attachment (SATA), Integrated Drive Electronics (IDE), IEEE- 1394, Universal Serial Bus (USB), fiber channel, Small Computer Systems interface (SCSI), etc. In some embodiments, the memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, Redundant Array of Independent Discs (RAID), solid-state memory devices, solid- state drives, etc.

[0023] In some embodiments, the memory 205 may store a collection of program or database components and / or executable code, including, without limitation, a user interface, an operating system 207, processes 220, a PDPR framework unit 240, etc. In some non-limiting embodiments or aspects, the CVCD management system 170 may store user / application data, such as the data, variables, records, etc. In some embodiments, the databases may be implemented as fault-tolerant, relational, scalable, secure databases such as Oracle or Sybase.

[0024] In some embodiments, operating system 207 may facilitate resource management and operation of the CVCD management system 170. Examples of operating systems include, without limitation, APPLE® MACINTOSH® OS X®, UNIX®, UNIX-like system distributions (E.G, BERKELEY SOFTWARE DISTRIBUTION® (BSD), FREEBSD®, NETBSD®, OPENBSD, etc ), LINUX® DISTRIBUTIONS (E.G, RED HAT®, UBUNTU®, KUBUNTU®,etc ), IBM®OS / 2®, MICROSOFT® WINDOWS® (XP®, VTSTA® / 7 / 8, 10 etc ), APPLE®OS®, GOOGLE™ ANDROID™, BLACKBERRY® OS, or the like.

[0025] In some non-limiting embodiments or aspects, the CI / CD management system 170 may implement a web browser (not shown in the figures) stored program component. The web browser (not shown in the figures) may be a hypertext viewing application, such as MICROSOFT® INTERNET EXPLORER®, GOOGLE™ CHROME™, MOZILLA® FIREFOX®, APPLE® SAFARI®, etc. Secure web browsing may be provided using Secure Hypertext Transport Protocol (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), etc. Web browsers may utilize facilities such as AJAX, DHTML, ADOBE® FLASH®, JAVASCRIPT®, JAVA®, Application Programming Interfaces (APIs), etc.

[0026] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. In some embodiments, a computer- readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, e.g., non-transitory. Examples include Random Access Memory (RAM), Read-Only Memory (ROM), volatile memory, non-volatile memory, hard drives, Compact Disc (CD) ROMs, Digital Video Disc (DVDs), flash drives, disks, and any other known physical storage media.

[0027] FIG. 3 illustrates a block diagram of a PDPR framework unit 240 of FIG. 2 in accordance with some embodiments. In some embodiments, the PDPR framework unit 240 is executablecode configured to operate as part of a PDPR framework to perform provisioning operations, deployment operations, performance testing operations, and retirement operations for CI / CD management system 170. In some embodiments, the PDPR framework unit 240 includes a quality assurance environment provisioning unit 310, a continuous integration unit 320, a build automation unit 330, an artifact management unit 345, a continuous delivery / deployment unit 350, a deployment automation unit 360, a testing automation unit 340, a collaboration and notifications unit 370, and a quality assurance environment server retirement unit 380. In some embodiments, the quality assurance environment provisioning unit 310, continuous integration unit 320, build automation unit 330, artifact management unit 345, continuous delivery / deployment unit 350, deployment automation unit 360, testing automation unit 340, collaboration and notifications unit 370, and quality assurance environment server retirement unit 380 are collectively configured to implement embodiments to perform provisioning operations, deployment operations, performance testing operations, and retirement operations as described herein.

[0028] In some embodiments, the PDPR framework is a structured and organized framework that, by way of a PDPR method (e.g., PDPR method 400 of FIG. 4) executed by PDPR framework unit 240, utilizes a quality assurance environment server retirement pipeline job to perform provisioning operations, deployment operations, performance testing operations, and retirement operations for CI / CD management system 170. For example, in some embodiments, PDPR framework unit 240 may utilize the quality assurance environment server retirement pipeline job to provision a quality assurance environment and retire a quality assurance environment infrastructure associated with the quality assurance environment after execution of a performance test on an application under test. In some embodiments, as stated previously, aquality assurance environment is a target environment generated by, for example, CT / CD management system 170, that is specifically configured to retire a quality assurance environment infrastructure or portions thereof after execution of a performance test on an application under test. In some embodiments, the quality assurance environment infrastructure is the infrastructure associated with the quality assurance environment that is utilized by the PDPR framework unit 240 to perform PDPR operations as part of the PDPR framework. In some embodiments, as stated previously, the quality assurance environment infrastructure may include designated quality assurance environment servers 152 for provisioning of container pods and containers. In some embodiments, utilization of the PDPR framework unit 240 by CI / CD management system 170 to retire the quality assurance environment infrastructure (e.g., quality assurance environment servers 152, etc.) from use by the quality assurance environment after execution of the performance test, serves as an improvement over other computer systems by, for example, reducing the number of servers and associated resources continuously being utilized for performance testing purposes after an application under test has been tested and deployed.

[0029] In some embodiments, the quality assurance environment server retirement pipeline job is executable code configured for a quality assurance environment that includes a provisioning, deployment, and retirement job and a performance test execution job. In some embodiments, the provisioning, deployment, and retirement job is executable code configured to provision a quality assurance environment, provision a quality assurance environment infrastructure, deploy an application under test as part of the quality assurance environment, and retire the quality assurance environment infrastructure associated with the quality assurance environment after execution of a performance test. In some embodiments, provisioning the quality assurance environment and quality assurance environment infrastructure may include, for example,provisioning quality assurance environment servers 152 for the quality assurance environment, provisioning a required number of container pods and containers in quality assurance environment servers 152, and / or provisioning other applications required for the PDPR operations described herein.

[0030] In some embodiments, the provisioning, deployment, and retirement job may be executed using a master in a master-slave architecture of the CI / CD management system 170. In some embodiments, the master may be, for example, a central controller of the CI / CD management system 170 that may be configured to, in addition to managing the configuration, scheduling, and execution of jobs, manage the provisioning, deployment, and retirement job. In some embodiments, the master may be utilized to define and configure build pipelines, define security settings, and manage plugins and extensions. In some embodiments, a master may be, for example, a Jenkins master configured to perform some of the embodiments described herein.

[0031] In some embodiments, the performance test execution job is executable code configured to execute a performance test of the application under test. In some embodiments, the performance text execution job may be executed by testing automation unit 340 (described further in detail herein). In some embodiments, the performance test execution job is triggered using, for example, a slave (e g., a Jenkins slave) or agent. In some embodiments, a "slave" or “agent” may refer to a worker node configured to perform tasks instructed by a master in the CI / CD management system 170. In some embodiments, the slave may be a server, machine, or other system that is coupled to a master server (e g., Jenkins master) that can execute CI / CD management system jobs (e.g., Jenkins jobs). In some embodiments, use of the master-slave architecture in the CI / CD management system 170 allows the CI / CD management system 170 to distribute workloads across multiple machines, enabling parallel execution of jobs. In someembodiments, utilization of the slave and the associated slave commands allows for the distribution of build and test workloads and jobs across multiple machines.

[0032] In some embodiments, the performance test execution job may utilize, for example, a gatling script, a namespace of the application under test, and other performance test execution job parameters. In some embodiments, the gatling script, namespace, and other performance test execution parameters may be passed as attributes to a command (e.g., Maven command). Maven is an open-source build automation and project management tool primarily used for Java projects that simplifies the build process, manages dependencies, and facilitates project organization. In some embodiments, a maven command may refer to a specific action or task executed through, for example, an Apache Maven build tool. Maven commands may be executed in the commandline interface (CLI) or within build scripts, typically using an “mvn” command followed by a desired goal or task. In some embodiments, attributes to a command are additional options or parameters that may be specified along with the command (e.g., Maven command) to modify the behavior of the command or provide additional information. In some embodiments, the attributes may be used to customize the build process, control plugin execution, configure profiles, etc. In some embodiments, a gatling script refers to a performance test script written in Gatling, which is an open-source load testing tool. Gatling scripts are typically written in Scala (a programming language utilized to generate performance tests), and may describe, for example, user behavior, requests, and scenarios to simulate during a load test.

[0033] In some embodiments, quality assurance environment provision unit 310 is executable code configured to provision a quality assurance environment for the PDPR framework. In some embodiments, quality assurance environment provision unit 310 is configured to provision the quality assurance environment according to quality assurance environment parameters providedby a user of the CT / CD management system 170 for use in execution of the performance test. Tn some embodiments, the quality assurance environment parameters are provided via a quality assurance environment request provided from the user of CI / CD management system 170 (described further herein).

[0034] In some embodiments, the continuous integration unit 320 is executable code configured to monitor the source code management system 180 for modified code (e g., modified source code or software application). In some embodiments, the source code and / or software application modifications may be made by, for example, a software developer tasked with modifying the source code or software application utilizing, for example, client device 190.

[0035] In some embodiments, build automation unit 330 is executable code configured to compile code, resolve dependencies, and package source code and / or software applications into executable artifacts. In some embodiments, the artifacts (or build artifacts) are configured to represent the compiled, packaged, or processed files and resources that may be ready for deployment. In some embodiments, build automation unit 330 manages tasks such as, for example, dependency management, compilation, and packaging to produce a build of the updated or modified software application by using the source code management system 180.

[0036] In some embodiments, artifact management unit 345 is executable code configured to store and manage build artifacts generated by, for example, build automation unit 330. In some embodiments, artifact management unit 345 provides a centralized location in CI / CD management system 170 for artifact versioning, storage, and distribution to ensure availability for subsequent stages of, for example, CI / CD pipelines.

[0037] In some embodiments, continuous delivery / deployment unit 350 is executable code configured to manage the overall flow and coordination of the software delivery process and / orPDPR operations. In some embodiments, continuous delivery / deployment unit 350 may be configured to manage an entire end-to-end pipeline. In some embodiments, continuous delivery / deployment unit 350 may serve as the central controller of the PDPR framework unit240.

[0038] In some embodiments, deployment automation unit 360 is executable code configured to automate the deployment process of software applications (e.g., applications under test). In some embodiments, deployment automation unit 360 is configured to automate the deployment process of applications under test into quality assurance environment servers 152 associated with the quality assurance environment. In some embodiments, deployment automation unit 360 is configured to automate the deployment process of applications under test into container pods and containers provisioned onto quality assurance environment servers 152. In some embodiments, deployment automation unit 360 is configured to ascertain the artifacts generated by the build automation unit 330 (e.g., compiled code, container images) associated with the software code or application under test and deploy the artifacts to the quality assurance environment servers 152 associate with the quality assurance environment. In some embodiments, deployment automation unit 360 interacts with the container management system 160 or deployment platforms to provision, configure, and manage the deployment targets.

[0039] In some embodiments, testing automation unit 340 is executable code configured to execute performance tests on, for example, an application under test. In some embodiments, testing automation unit 340 may be configured to execute a performance test as part of a performance test execution job (described further herein). In some embodiments, testing automation unit 340 may also execute unit tests, integration tests, and functional tests, to validate the functionality, quality, and correctness of the modified software code.

[0040] In some embodiments, collaboration and notifications unit 370 is executable code configured to facilitate communication, collaboration, and notifications among users utilizing the network infrastructure 100 for PDPR purposes. In some embodiments, collaboration and notifications unit 370 integrates with collaboration tools, notification systems, or communication channels to provide updates, notifications, and feedback on the progress and status of the CI / CD pipelines (e.g., quality assurance environment server retirement pipeline job).

[0041] In some embodiments, quality assurance environment server retirement unit 380 is executable code configured to retire a quality assurance environment infrastructure associated with a quality assurance environment after execution of a performance test. As defined herein, “retire” or “retirement” refers to relinquishing use of a quality assurance infrastructure or portion thereof for purposes related to a specific performance test such that other users of a quality assurance environment may access the resources previously utilized during the specific performance test. In some embodiments, quality assurance environment server retirement unit 380 is configured to retire quality assurance environment servers 152 that have been provisioned as part of a quality assurance environment after a performance test associated with a quality assurance environment has been conducted (described further herein). In some embodiments, quality assurance environment server retirement unit 380 is configured to retire container pods and containers that have been provisioned on the quality assurance environment servers 152 as part of the quality assurance environment after the performance test has been conducted, allowing other systems in the network infrastructure 100 to utilize the quality assurance environment servers 152 for performance testing after the performance test has been completed.

[0042] With further reference to FIG. 3, in some embodiments, in operation, developers of a software application request commencement of a quality assurance environment by issuing aquality assurance environment request to quality assurance environment provisioning unit 310. In some embodiments, as stated previously, the quality assurance environment is a target environment generated by, for example, CI / CD management system 170, that is specifically configured to retire a quality assurance environment infrastructure or portions thereof after execution of a performance test on an application under test. In some embodiments, as part of the quality assurance environment request, the user provides quality assurance environment parameters to the quality assurance environment provision unit 310. In some embodiments, the quality assurance environment parameters are target parameters required to generate the quality assurance environment such that the associated quality assurance environment infrastructure may be retired upon completion of an associated performance test. In some embodiments, the quality assurance environment parameters may include the quality assurance environment infrastructure parameters, deployment parameters, retirement parameters, etc.

[0043] In some embodiments, upon receipt of the quality assurance environment request, quality assurance environment provision unit 310 of PDPR framework unit 240 provisions the quality assurance environment requested via client device 190. In some embodiments, quality assurance environment provision unit 310 provisions the quality assurance environment according to the quality assurance environment parameters for use in execution of the performance test. In some embodiments, quality assurance environment provision unit 310 provisions the quality assurance environment by implementing the quality assurance environment parameters (e.g., the quality assurance environment infrastructure parameters, deployment parameters, retirement parameters, etc.).

[0044] In some embodiments, after the quality assurance environment has been provisioned by quality assurance environment provisioning unit 310, developers of the software applicationcommit code modifications to the software application via client device 190 utilizing, for example, the source code management system 180 (illustrated in FIG. 1). Recall that, in some embodiments, the source code management system 180 stores code modifications in repositories 181. In some embodiment, the source code management system 180 may notify the continuous integration unit 320 of the software application modifications. In some embodiments, the continuous integration unit 320 monitors the repositories 181 of source code management system 180 for code modifications or updates made to the software application. In some embodiments, continuous integration unit 320 continuously monitors the source code management system 180 for triggering events indicative of modifications to the software application, such as code commits or pull requests. In some embodiments, when continuous integration unit 320 detects a triggering event, the continuous integration unit 320 triggers the build automation process via build automation unit 330 for the modified software application.

[0045] In some embodiments, after the build automation process has been triggered, build automation unit 330 receives the modified software application, along with any necessary build configuration and dependency definitions, as input from source code management system 180. In some embodiments, build automation unit 330 compiles the modified software application, resolves dependencies, and packages the software modified software application into executable artifacts. In some embodiments, as stated previously, the artifacts are configured to represent the compiled, packaged, or processed files and resources that are ready for deployment. In some embodiments, build automation unit 330 generates the build artifacts as output and provides the output to artifact management unit 345 as part of the quality assurance environment server retirement pipeline job.

[0046] Tn some embodiments, artifact management unit 345 receives the build artifacts from the build automation unit 330 as input and manages the storing of the build artifacts to ensure proper versioning, storage, and distribution of the build artifacts. In some embodiments, the artifacts are configured by the artifact management unit 345 such that the artifacts are organized, labeled, and retrievable during, for example, execution of the quality assurance environment server retirement pipeline job.

[0047] In some embodiments, after the build artifacts have been managed by the artifact management unit 345, the continuous delivery / deployment unit 350 determines the build artifacts (associated with the modified software application) to be retrieved for execution of a performance test. In some embodiments, the continuous delivery / deployment unit 350 may utilize various criteria, such as the source code version, branch, or specific file modifications, to identify the associated build artifacts for the modified code. In some embodiments, once the relevant build artifacts have been identified, the continuous delivery / deployment unit 350 interacts with the artifact management unit 345 to retrieve the required artifacts associated with the modified software application.

[0048] In some embodiments, continuous delivery / deployment unit 350 receives the build artifacts associated with the modified code as the application under test from the artifact management unit 345. In some embodiments, the scope of artifact retrieval may depend on the CI / CD management system 170 setup and configuration (e.g., quality assurance environment server retirement pipeline job). Tn some embodiments, when, for example, the quality assurance environment server retirement pipeline job is configured to perform incremental builds, the continuous delivery / deployment unit 350 may retrieve only the build artifacts related to the modified components or code changes. In some embodiments, for the incremental buildapproach, the incremental build approach optimizes the deployment process by avoiding unnecessary rebuilding and redeployment of unaffected parts of the application. In some embodiments, when, for example, the quality assurance environment server retirement pipeline job is set up to perform full builds for each modification, the continuous delivery / deployment unit 350 retrieves and deploys the complete set of build artifacts for the modified software application. In some embodiments, the decision of whether to perform an incremental build or a full build depends on factors such as, for example, the size of the software application, the extent of modifications, performance testing requirements, and deployment considerations. In some embodiments, the configuration and policies are set in the quality assurance environment server retirement pipeline job to determine the appropriate build. In some embodiments, in either case, continuous delivery / deployment unit 350 fetches the build artifacts necessary for the modified software application to facilitate the deployment of the application under test. In some embodiments, by retrieving the artifacts associated with the modifications to the software application or source code, continuous delivery / deployment unit 350 ensures that the performance testing of the application under test is conducted on the appropriate version or state of the software application.

[0049] In some embodiments, after receiving the build artifacts as the application under test from artifact management unit 345, continuous delivery / deployment unit 350 initiates, as part of the provisioning, deployment, and retirement job, the provisioning of the quality assurance environment infrastructure for use in execution of the performance test. Tn some embodiments, continuous delivery / deployment unit 350 initiates the quality assurance environment infrastructure provisioning process by sending a quality assurance environment infrastructure provisioning request to the container management system 160 specifying the quality assuranceenvironment infrastructure parameters necessary to provision the quality assurance environment infrastructure. In some embodiments, the quality assurance environment infrastructure parameters are parameters necessary to provision the quality assurance environment infrastructure to deploy the application under test, execute the performance test, and retire the infrastructure after the performance test of the application under test is completed in the quality assurance environment. In some embodiments, the quality assurance environment infrastructure parameters include, for example, the parameters necessary for provisioning the quality assurance environment servers and provisioning the container and container pods. In some embodiments, the quality assurance environment infrastructure parameters include, for example, a hardware configuration, a server configuration, a deployment configuration, a required number of container pods, resource requirements, a container image to be used for container pod creation and / or container creation, retirement parameters (e.g., retirement criteria), etc. In some embodiments, as stated previously, container management system 160 may utilize a container orchestration platform, such as, for example, Kubernetes or the like, to provision the quality assurance environment infrastructure according to the quality assurance environment infrastructure parameters.

[0050] In some embodiments, continuous delivery / deployment unit 350 may utilize a Representational State Transfer Application Programming Interface (REST API) associated with the container management system 160 to transmit the quality assurance environment infrastructure provisioning request to the container management system 160. Tn some embodiments, the quality assurance environment server retirement pipeline job may be configured to expose auto provisioning, auto deployment, and sizing requests for containers via the REST API (e.g., utilizing the REST API in the provisioning, deployment, and retirementjob). Tn some embodiments, the process of auto deployment and resizing for containers may be utilized for deploying the latest build of the application and providing the quality assurance environment infrastructure parameters for the required hardware, server configurations, deployment configurations, resource requirements, retirement parameters, and required number of container pods, etc., to execute the performance test for the quality assurance environment.

[0051] In some embodiments, container management system 160 receives the quality assurance environment infrastructure provisioning request from continuous delivery / deployment unit 350 and provisions the quality assurance environment infrastructure according to the quality assurance environment infrastructure parameters for use by quality assurance environment during the performance test. In some embodiments, container management system 160 provisions quality assurance environment servers and corresponding container pods and containers for use by the quality assurance environment during the associated performance test. In some embodiments, the container management system 160 provisions the container pods and containers according to the specifications specified in the quality assurance environment infrastructure parameters associated with the quality assurance environment. In some embodiments, container management system 160 provisions the container pods and containers associated with the quality assurance environment in quality assurance environment servers 152. In some embodiments, container management system 160 is configured to distribute containers and container pods across quality assurance environment servers 152 in a cluster based upon the quality assurance environment infrastructure parameters provided by the continuous delivery / deployment unit 350 as part of the provisioning, deployment, and retirement job.

[0052] In some embodiments, after the container management system 160 has provisioned the quality assurance environment infrastructure (e.g., quality assurance environment servers 152and the container pods and containers in quality assurance environment servers 152), deployment automation unit 360 deploys the application under test to the requisite quality assurance environment infrastructure (e.g., the container pods and containers associated with the quality assurance environment in quality assurance environment servers 152). In some embodiments, as stated previously, deployment automation unit 360 is configured to automate the deployment process, including the deployment of the application under test associated with the quality assurance environment.

[0053] In some embodiments, after the application under test has been deployed to quality assurance environment infrastructure (e.g., the container pods and containers in quality assurance environment servers 152 associated with the quality assurance environment), performance test execution job utilizes testing automation unit 340 to perform the performance test of the application under test. In some embodiments, the performance test execution job utilizes the testing automation unit 340 to execute the performance test of the application under test in the quality assurance environment. In some embodiments, testing automation unit 340 pulls the performance test script (e.g., Gatling script) from the source code management system 180 (e.g., Bitbucket Server) and executes the performance test on the application under test. In some embodiments, as stated previously, the performance test execution job may be triggered using, for example, a slave (e.g., a Jenkins slave utilizing Maven) or agent.

[0054] In some embodiments, after the performance test has been executed by testing automation unit 340, utilizing the provisioning, deployment, and retirement job, quality assurance environment server retirement unit 380 retires the quality assurance environment infrastructure associated with the performance test and the quality assurance environment (e.g., quality assurance environment servers 152, container pods, containers, etc. associated with theperformance test and the quality assurance environment) Tn some embodiments, quality assurance environment server retirement unit 380 retires the quality assurance environment infrastructure by releasing or directing container management system 160 to release all or a portion of the allocated resources associated with the quality assurance environment and / or quality assurance environment infrastructure (e.g., container pods, containers, and associated namespaces, etc.) that have been utilized during the performance test.

[0055] In some embodiments, quality assurance environment server retirement unit 380 retires the quality assurance environment infrastructure or portion thereof by sending a quality assurance environment retirement request to the container management system 160 to release the requisite quality assurance environment structure (e.g., the quality assurance environment structure provisioned or controlled by the container management system 160). In some embodiments, container management system 160 receives the quality assurance retirement request from the quality assurance environment server retirement unit 380 and releases the requisite quality assurance environment infrastructure associated with the performance test and the quality assurance environment provisioned or controlled by the container management system 160.

[0056] In some embodiments, quality assurance environment server retirement unit 380 retires the quality assurance environment infrastructure (e.g., quality assurance environment servers 152, container pods, containers, etc., associated with the performance test and the quality assurance environment) in order to make the quality assurance infrastructure immediately available for additional performance testing by other users of the network infrastructure 100.

[0057] In some embodiments, in addition to retiring the allocated quality assurance environment resources, after the performance test has been executed by testing automation unit 340,collaboration and notifications unit 370 may receive test reports associated with the performance from testing automation unit 340 as input for sharing with, for example, other developers of the software application. In some embodiments, collaboration and notifications unit 370 facilitates communication, collaboration, and notifications among team members involved in the CI / CD management process. In some embodiments, the collaboration and notifications unit 370 may provide notifications or updates of the testing process and test results to relevant stakeholders.

[0058] FIG. 4 is a flow diagram illustrating a PDPR method 400 in accordance with some embodiments. The method, process steps, or stages illustrated in the figures may be implemented as an independent routine or process, or as part of a larger routine or process. Note that each process step or stage depicted may be implemented as an apparatus that includes a processor executing a set of instructions, a method, or a system, among other embodiments. In some embodiments, the PDPR method 400 is described with reference to the figures described herein.

[0059] In some embodiments, at operation 420, quality assurance environment provisioning unit 310 of PDPR framework unit 240 provisions a quality assurance environment for use in execution of a performance test on an application under test. In some embodiments, as stated previously, quality assurance environment provisioning unit 310 provisions the quality assurance environment by utilizing the quality assurance environment parameters provided by a user (e.g., software developer) via client device 190 utilizing a quality assurance environment provisioning request. In some embodiments, after the quality assurance environment has been provisioned by quality assurance environment provisioning unit 310, operation 420 proceeds to operation 430.

[0060] In some embodiments, at operation 430, container management system 160 provisions the quality assurance environment infrastructure for use by the quality assurance environment to execute the performance test. In some embodiments, container management system 160provisions the quality assurance environment infrastructure according to the quality assurance environment infrastructure parameters provided by the continuous delivery / deployment unit 350 via a quality assurance environment infrastructure provisioning request. In some embodiments, continuous delivery / deployment unit 250 provisioning the quality assurance environment infrastructure includes provisioning quality assurance environment servers 152, provisioning the container pods and containers associated with the provisioned quality assurance environment servers 152, etc. In some embodiments, after the quality assurance environment infrastructure has been provisioned by the container management system 160, operation 430 proceeds to operation 440.

[0061] In some embodiments, at operation 440, deployment automation unit 260 deploys the application under test to quality assurance environment infrastructure (e.g., quality assurance environment servers 152 and associated container pods and containers utilized by the quality assurance environment) utilized by the quality assurance environment. In some embodiments, after the application under test has been deployed to the quality assurance environment infrastructure, operation 440 proceeds to operation 450.

[0062] In some embodiments, at operation 450, testing automation unit 340 executes a performance test of the application under test. In some embodiments, as stated previously, the performance test execution job utilizes testing automation unit 340 to perform the performance test of the application under test. In some embodiments, after the performance test of the application under test is executed by the testing automation unit 340, operation 450 proceeds to operation 460.

[0063] In some embodiments, at operation 460, upon completion of the performance test, server retirement unit 280 retires (or directs container management system 160 to retire) the qualityassurance environment infrastructure utilized by the quality assurance environment (e g., quality assurance environment servers 152 and associated container pods and containers utilized by the quality assurance environment to execute the performance test). In some embodiments, server retirement unit 280 retires the quality assurance environment infrastructure by releasing or directing container management system 160 to release all or a portion of the allocated resources associated with the quality assurance environment and / or quality assurance environment infrastructure (e.g., container pods, containers, and associated namespaces, etc.) that have been utilized during the performance test.

[0064] In some embodiments, a computer-implemented method includes, provisioning a quality assurance environment for use in execution of a performance test on an application under test, the quality assurance environment being configured to be provisioned on demand; provisioning a quality assurance environment infrastructure for use by the quality assurance environment to execute the performance test; deploying the application under test to the quality assurance environment infrastructure of the quality assurance environment; executing the performance test on the application under test; and retiring the quality assurance environment infrastructure utilized by the quality assurance environment upon completion of the performance test.

[0065] In some embodiments of the computer-implemented method, a quality assurance environment server retirement pipeline job is utilized to implement a provisioning, deployment, and retirement job and a performance test execution job in the quality assurance environment.

[0066] In some embodiments of the computer-implemented method, the provisioning, deployment, and retirement job is utilized to provision quality assurance environment servers as part of the quality assurance environment infrastructure for use by the quality assurance environment to execute the performance test.

[0067] Tn some embodiments of the computer-implemented method, container pods and containers are provisioned as part of the provisioning of the quality assurance environment servers.

[0068] In some embodiments of the computer-implemented method, the performance test execution job is utilized to execute the performance test.

[0069] In some embodiments of the computer-implemented method, the performance test is implemented using a performance test script.

[0070] In some embodiments of the computer-implemented method, the performance test script is a Gatling Script.

[0071] In some embodiments of the computer-implemented method, the quality assurance environment server retirement pipeline job is executed using a continuous integration / continuous deployment (CI / CD) management system.

[0072] In some embodiments of the computer-implemented method, the CI / CD management system utilizes Jenkins for execution of the quality assurance environment server retirement pipeline job.

[0073] In some embodiments of the computer-implemented method, the quality assurance environment is implemented using a shift left environment.

[0074] In some embodiments, a system includes a processor; and a non-transitory computer readable medium coupled to the processor, the non-transitory computer readable medium including code that: provisions a quality assurance environment for use in execution of a performance test on an application under test, the quality assurance environment being configured to be provisioned on demand; provisions quality assurance environment servers for use by the quality assurance environment to execute the performance test; deploys theapplication under test to the quality assurance environment servers of the quality assurance environment; executes the performance test on the application under test; and retires the quality assurance environment servers utilized by the quality assurance environment upon completion of the performance test.

[0075] In some embodiments of the system, a quality assurance environment server retirement pipeline job is utilized to implement a provisioning, deployment, and retirement job and a performance test execution job in the quality assurance environment.

[0076] In some embodiments of the system, the provisioning, deployment, and retirement job is utilized to provision the quality assurance environment servers for use by the quality assurance environment to execute the performance test.

[0077] In some embodiments of the system, the provisioning, deployment, and retirement job provisions container pods and containers.

[0078] In some embodiments of the system, the performance test execution job is utilized to execute the performance test.

[0079] In some embodiments of the system, the performance test is implemented using a performance test script.

[0080] In some embodiments of the system, the quality assurance environment server retirement pipeline job is executed using a continuous integration / continuous deployment (CI / CD) management system.

[0081] In some embodiments of the system, the quality assurance environment is implemented utilizing a shift left environment.

[0082] In some embodiments, a computer-implemented method includes, executing a quality assurance environment server retirement pipeline job, the quality assurance environment serverretirement pipeline job being configured to utilize a provisioning, deployment, and retirement job to provision container pods for execution of a performance test on an application under test; deploying the application under test to the container pods; utilizing a performance test execution job to execute the performance test of the application under test; and based on a completion of the performance test, retiring the container pods utilized to execute the performance test.

[0083] In some embodiments of the computer-implemented method, the performance test is implemented using a performance test script.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising: provisioning a quality assurance environment for use in execution of a performance test on an application under test, the quality assurance environment being configured to be provisioned on demand; provisioning a quality assurance environment infrastructure for use by the quality assurance environment to execute the performance test; deploying the application under test to the quality assurance environment infrastructure of the quality assurance environment; executing the performance test on the application under test; and retiring the quality assurance environment infrastructure utilized by the quality assurance environment upon completion of the performance test.

2. The computer-implemented method of claim 1, wherein: a quality assurance environment server retirement pipeline job is utilized to implement a provisioning, deployment, and retirement j ob and a performance test execution job in the quality assurance environment.

3. The computer-implemented method of claim 2, wherein: the provisioning, deployment, and retirement job is utilized to provision quality assurance environment servers as part of the quality assurance environment infrastructure for use by the quality assurance environment to execute the performance test.

4. The computer-implemented method of claim 3, wherein: container pods and containers are provisioned as part of the provisioning of the quality assurance environment servers.

5. The computer-implemented method of claim 4, wherein: the performance test execution job is utilized to execute the performance test.

6. The computer-implemented method of claim 5, wherein: the performance test is implemented using a performance test script.

7. The computer-implemented method of claim 6, wherein: the performance test script is a Gatling Script.

8. The computer-implemented method of claim 7, wherein: the quality assurance environment server retirement pipeline job is executed using a continuous integration / continuous deployment (CI / CD) management system.

9. The computer-implemented method of claim 8, wherein: the CI / CD management system utilizes Jenkins for execution of the quality assurance environment server retirement pipeline job.

10. The computer-implemented method of claim 9, wherein: the quality assurance environment is implemented using a shift left environment.

11. A system, comprising: a processor; and a non-transitory computer readable medium coupled to the processor, the non-transitory computer readable medium including code that: provisions a quality assurance environment for use in execution of a performance test on an application under test, the quality assurance environment being configured to be provisioned on demand; provisions quality assurance environment servers for use by the quality assurance environment to execute the performance test;deploys the application under test to the quality assurance environment servers of the quality assurance environment; executes the performance test on the application under test; and retires the quality assurance environment servers utilized by the quality assurance environment upon completion of the performance test.

12. The system of claim 11, wherein: a quality assurance environment server retirement pipeline job is utilized to implement a provisioning, deployment, and retirement j ob and a performance test execution job in the quality assurance environment.

13. The system of claim 12, wherein: the provisioning, deployment, and retirement j ob is utilized to provision the quality assurance environment servers for use by the quality assurance environment to execute the performance test.

14. The system of claim 13, wherein: the provisioning, deployment, and retirement j ob provisions container pods and containers.

15. The system of claim 14, wherein: the performance test execution job is utilized to execute the performance test.

16. The system of claim 15, wherein: the performance test is implemented using a performance test script.

17. The system of claim 16, wherein: the quality assurance environment server retirement pipeline job is executed using a continuous integration / continuous deployment (CI / CD) management system.

18. The system of claim 17, wherein: the quality assurance environment is implemented utilizing a shift left environment.

19. A computer-implemented method, comprising: executing a quality assurance environment server retirement pipeline job, the quality assurance environment server retirement pipeline job being configured to utilize a provisioning, deployment, and retirement job to provision container pods for execution of a performance test on an application under test; deploying the application under test to the container pods; utilizing a performance test execution job to execute the performance test of the application under test; and based on a completion of the performance test, retiring the container pods utilized to execute the performance test.

20. The computer-implemented method of claim 19, wherein: the performance test is implemented using a performance test script.

Images