Performance variability evaluator

Abstract

One or more computer processors identify one or more similar historical regression tests and historical builds using a similarity measure between the computed regression build and one or more historical builds made on the same release cycle, wherein the identified one or more similar historical regression tests and historical builds are K nearest neighbors of the regression build; predict a time to expiration of the one or more summary analysis regression tests using a KNN algorithm that includes the K nearest neighbors each weighted by a corresponding average distance to the test point and the time to expiration as a target variable; determine that the regression build is an actual regression in response to the predicted time to expiration exceeding an actual time to expiration associated with the regression build; and apply one or more mitigation actions to the regression build in response to determining that the regression build is not due to volatility.

Description

Background Technology

[0001] This invention relates generally to the field of machine learning, and more specifically to predicting and mitigating performance variability in regression testing.

[0002] Regression testing involves performing multiple functional and non-functional tests to ensure that previously developed and tested software still performs properly after modifications or changes. If the software fails to perform properly or performs with reduced efficiency, this is a regression. Regression testing can be triggered by bug fixes, software enhancements, configuration changes, and hardware replacements.

[0003] The K-Nearest Neighbors (KNN) algorithm is a nonparametric method for classification and regression, where the input consists of the k nearest training examples in the feature space. KNN is a type of instance-based learning where the function is approximated only locally, and all computation is deferred until the function is evaluated. For regression purposes (e.g., KNN regression), the output is a predicted value of the input, where the value can be the mean or median. Summary of the Invention

[0004] Embodiments of the present invention disclose a computer-implemented method, computer program product, and system for predicting and mitigating variability in regression testing. The computer-implemented method includes one or more computer processors identifying one or more similar historical regression tests and historical builds using a similarity metric between a computed regression build and one or more historical builds performed at the same release period, wherein the identified one or more similar historical regression tests and historical builds are the K nearest neighbors of the regression build. The one or more computer processors use a KNN algorithm comprising a weighted average distance of the K nearest neighbors for each free distance test point and the elapsed time as the target variable to predict the elapsed time of the one or more profiling regression tests. In response to the predicted elapsed time exceeding the actual elapsed time associated with the regression build, the one or more computer processors determine that the regression build is an actual regression. In response to determining that the regression build is not due to variability, the one or more computer processors apply one or more mitigation actions to the regression build based on the one or more similar historical builds. Attached Figure Description

[0005] Figure 1 This is a functional block diagram illustrating a computing environment according to an embodiment of the present invention;

[0006] Figure 2 It is a depiction of an embodiment according to the present invention. Figure 1 A flowchart of the operational steps of a program on a server computer within a computing environment used to predict and mitigate variability in regression testing;

[0007] Figure 3 An exemplary table according to an embodiment of the present invention is shown;

[0008] Figure 4 An exemplary diagram is shown according to an embodiment of the present invention;

[0009] Figure 5 An exemplary diagram is shown according to an embodiment of the present invention; and

[0010] Figure 6 This is a block diagram of the components of a server computer according to an embodiment of the present invention. Detailed Implementation

[0011] Traditionally, organizations perform multiple assessments and tests for software versions (e.g., alpha, beta, version candidates, etc.), which involve running multiple performance and edge case workloads (i.e., continuous integration and continuous deployment) on one or more builds. In response to the completion of these assessments and tests, traditional systems record diagnostics and use statistical functions to compare the diagnostic results with historical results (e.g., baselines) and identify potential regressions. Traditional systems are limited to comparing only a subset of diagnostic results; for example, the system may only consider centrally processed statistics over a finite time period due to computational constraints imposed by system limitations. Furthermore, traditional systems utilize significant computational resources to identify and compute baselines and / or associated thresholds.

[0012] Embodiments of the present invention predict the likelihood of potential regressions (i.e., regression tests, builds, software, etc.) undergoing variability or actual regressions (i.e., not undergoing variability). Embodiments of the present invention perform one or more probabilistic tests on the elapsed time of execution of one or more queries on one or more baseline builds and regression builds. Embodiments of the present invention are associated with one or more dynamic thresholds for each project, build, module, or submodule. Embodiments of the present invention recognize that threshold determination is critical for efficient integration and deployment of software. Embodiments of the present invention reduce false alarms and allow for effective testing, modification, and subsequent deployment of approved software (e.g., no regressions or confirmed false alarms). Embodiments of the present invention initiate mitigation actions in response to regression alarms. Implementations of embodiments of the present invention can take many forms, and exemplary implementation details are discussed subsequently with reference to the accompanying drawings.

[0013] The invention will now be described in detail with reference to the accompanying drawings.

[0014] Figure 1 This is a functional block diagram illustrating a computing environment with an overall designation of 100 according to an embodiment of the present invention. As used herein, the term "computing" describes a computer system comprising multiple physically distinct devices that operate together as a single computer system. Figure 1This illustration provides only one implementation and does not imply any limitation on the environments in which different embodiments may be implemented. Many modifications can be made to the described environments by those skilled in the art without departing from the scope of the invention as set forth in the claims.

[0015] Computing environment 100 includes a server computer 120 connected via a network 102. Network 102 can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) (such as the Internet), or a combination of all three, and can include wired, wireless, or fiber optic connections. Network 102 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, and / or video signals, including multimedia signals containing voice, data, and video information. Generally, network 102 can be any combination of connections and protocols that enable communication between server computer 120 and other computing devices (not shown) within computing environment 100. In various embodiments, network 102 operates locally via wired, wireless, or optical connections and can be any combination of connections and protocols (e.g., personal area network (PAN), near field communication (NFC), laser, infrared, ultrasound, etc.).

[0016] Server computer 120 may be a standalone computing device, management server, network server, mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, server computer 120 may represent a server computing system, such as one that utilizes multiple computers as server systems in a cloud computing environment. In another embodiment, server computer 120 may be a laptop computer, tablet computer, netbook computer, personal computer (PC), desktop computer, personal digital assistant (PDA), smartphone, or any programmable electronic device capable of communicating with other computing devices (not shown) within computing environment 100 via network 102. In another embodiment, server computer 120 represents a computing system utilizing cluster computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within computing environment 100. In the depicted embodiments, server computer 120 includes repository 122 and programs 150. In other embodiments, server computer 120 may include other applications, databases, programs, etc., not described in computing environment 100. In embodiments, server computer 120 is a source code management system and / or CI / CD system. Server computer 120 may include, for example, regarding Figure 6 A more detailed description and depiction of the internal and external hardware components.

[0017] Repository 122 is a storehouse of data used by program 150. In the depicted embodiment, repository 122 resides on server computer 120. In another embodiment, repository 122 may reside elsewhere within computing environment 100, as long as program 150 has access to repository 122. A database is an organized collection of data. Repository 122 can be implemented using any type of storage device capable of storing data and configuration files accessible and utilized by program 150, such as a database server, hard disk drive, or flash memory. In an embodiment, repository 122 stores data used by program 150, such as historical tests, test cases, modules, associated profilers, thresholds, etc. In an embodiment, repository 122 is a source code management system that allows multiple users to push code into the system and allows other users to view and download the included code repositories. In another embodiment, automated testing is performed on pushed code and / or software errors and potential regressions. In an embodiment, the source code management system stores data used by program 150, such as historical code repositories and associated modifications.

[0018] Program 150 is a program for predicting and mitigating performance variability in regression testing. In various embodiments, program 150 may perform the following steps: identifying one or more similar historical regression tests and historical builds using a similarity metric between a calculated regression build and one or more historical builds performed at the same release cycle, wherein the identified one or more similar historical regression tests and historical builds are the K nearest neighbors of the regression build; predicting the elapsed time of the one or more profiling regression tests using a KNN algorithm, the KNN algorithm including the K nearest neighbors (each nearest neighbor weighted by a corresponding average distance from the test point) and the elapsed time as a target variable; determining that the regression build is an actual regression in response to the predicted elapsed time exceeding the actual elapsed time associated with the regression build; and applying one or more mitigation actions to the regression build based on the one or more similar historical builds in response to determining that the regression build is not due to variability. In the depicted embodiments, program 150 is a standalone software program. In another embodiment, the functionality of program 150 or any combination thereof may be integrated into a single software program. In some embodiments, program 150 may reside on a separate computing device (not depicted) but may still communicate via network 102. In different embodiments, a client version of program 150 resides on any other computing device (not depicted) within computing environment 100. Reference Figure 2 The procedure 150 is described and illustrated in more detail.

[0019] This invention may include various accessible data sources (such as repository 122), which may include personal storage devices, data, content, or information that the user wishes not to be processed. Processing refers to any automated or non-automated operation or set of operations, such as collecting, recording, organizing, structuring, storing, adapting, altering, retrieving, consulting, using, disseminating, or otherwise making personal data available, combined, restricted, erased, or destroyed. Procedure 150 provides informed consent with a notification of personal data collection, allowing the user to opt in or opt out of processing personal data. Consent may take several forms. Opt-in consent may impose an affirmative action on the user before personal data is processed. Alternatively, opt-out consent may impose an affirmative action on the user to prevent personal data from being processed before it is processed. Procedure 150 enables authorized and secure processing of user information (such as tracking information) and personal data (such as personally identifiable information or sensitive personal information). Procedure 150 provides information about the nature of the personal data and processing (e.g., type, scope, purpose, duration, etc.). Procedure 150 provides the user with a copy of the stored personal data. Procedure 150 allows for the correction or completion of incorrect or incomplete personal data. Procedure 150 also allows for the immediate deletion of personal data.

[0020] Figure 2 A flowchart 200 is depicted, illustrating the operational steps of a procedure 150 for predicting and mitigating variability in regression testing according to an embodiment of the present invention.

[0021] Procedure 150 initiates regression testing on the software (step 202). Procedure 150 initiates one or more evaluation, regression, unit, functional, and performance tests (hereinafter referred to as tests) on one or more software, modules, applications, functions, queries, containers, code repositories, deployments, and / or code portions (hereinafter referred to as software). In embodiments, procedure 150 monitors a source code management system (e.g., repository 122) or is notified by said system in response to pushes, modifications, and / or stored software. In various embodiments, procedure 150 acts as an inline and / or transparent proxy between a 'sitting' computing device (not depicted) and a destination server (i.e., server computer 120). In another embodiment, procedure 150 monitors git clients or servers to determine pushes or code modifications to / from the source code management system. In one embodiment, procedure 150 identifies a code repository being pushed or pulled and pauses, delays, suspends, or stops git actions (i.e., pushes or pulls) or subsequent software deployments. In different embodiments, program 150 receives notifications about push, store, or pull builds and associated information (e.g., code metadata, dependencies, associated tests, etc.). In one embodiment, program 150 automatically retrieves all associated historical builds contained, referenced, or stored within the source code management system.

[0022] Procedure 150 performs profile analysis on regression tests (step 204). In response to procedure 150 initiating one or more regression tests, procedure 150 performs profile analysis (e.g., monitoring, scanning, etc.) on the tests. For example, procedure 150 records memory and CPU statistics associated with one or more tests of the software (specifically modified database queries). In another example, procedure 150 profiles the time complexity, frequency / duration of one or more function calls. In an embodiment, procedure 150 utilizes binary tools (such as code profilers (e.g., event-based, statistical, instrumentation, and simulation methods)) to profile the tests. In this embodiment, procedure 150 utilizes monitoring tools to gain deep insights (e.g., statistical insights), said tools including operating system monitoring tools (e.g., vmstat for monitoring CPU utilization) and profile analysis tools, such as CPU profile analysis monitoring multiple associated CPU statistics (e.g., execution time, temperature, minimum CPU utilization, maximum CPU utilization, average CPU utilization, memory utilization, system temperature, etc.), such as database function profile analysis (e.g., profiling). In various embodiments, program 150 utilizes system profiling, including CPU profiling, GPU profiling, input / output profiling, and network profiling. In various embodiments, diagnostics such as CPU profiler data or vmstat are transformed or modified upon generation or receipt. For example, program 150 filters one or more results generated from the CPU profiler, retaining only database functions invoked during a specific query execution. In a further embodiment, program 150 reduces the dimensionality of the results and their associations by aggregating data points (e.g., based on CPU ticks spent in a specified component aggregation function). In another instance, program 150 creates a results table with a column for each component name. In this example, program 150 updates the results table such that program 150 modifies only the rows for each test. In a further example, program 150 reduces dimensionality by discarding all columns (such as "empty") that have constant values ​​for all rows.

[0023] Procedure 150 detects regression (step 206). In one embodiment, as described in step 204, procedure 150 detects potential regressions based on monitored profiler data during one or more ongoing tests. In one embodiment, procedure 150 utilizes a common regression threshold to detect regression tests and associated builds. In this embodiment, procedure 150 flags a regression if one or more test points (e.g., data points generated from the profiler) exceed the common regression threshold. In another embodiment, procedure 150 receives alerts or notifications from the profiler regarding one or more test cases with potential regressions (i.e., regression builds). In various embodiments, procedure 150 compares ongoing test data to identify potential regressions, for example, procedure 150 flags all tests that deviate 5% from a baseline (e.g., historical average, etc.) or a threshold. In this embodiment, procedure 150 utilizes an inclusive (e.g., allowing multiple potential regressions) threshold, thereby allowing procedure 150 to dynamically adjust subsequent thresholds and analyze potential regressions.

[0024] Procedure 150 identifies similar historical tests and associated mitigation actions (step 208). In one embodiment, procedure 150 identifies and determines multiple historical tests based on data generated from one or more profiling tools and related diagnostics in response to the detection of a regression. For example, procedure 150 retrieves one or more previous builds associated with the current build. In one embodiment, procedure 150 calculates a similarity metric or score (e.g., numerical or probabilities) between the regression build and a subset of all historical builds or build / test histories (such as tests specific to release cycles, sets, or versions). In another embodiment, procedure 150 generates similarity scores from diagnostic results such as CPU, graphics processing unit (GPU), system, networking, database, and memory profiling tool statistics. In a further embodiment, procedure 150 weights different profiling tool statistics based on the correlation between diagnostic results and regressions. In one embodiment, procedure 150 utilizes a nonparametric algorithm to calculate a similarity metric (e.g., Euclidean distance) between the test sample (i.e., the regression build / test) and all training samples (i.e., historically similar builds / tests). In different embodiments, procedure 150 calculates a similarity metric based on CPU profiling statistics, utilizing regressions of builds / runs and all past builds / runs on the same release. In different embodiments, procedure 150 determines that historical tests and builds are similar based on associated data exceeding a similarity threshold. In another embodiment, procedure 150 determines that all historical tests and builds in the same release cycle are similar.

[0025] Procedure 150 predicts the elapsed time of a regression test based on identified similar historical tests (step 210). In one embodiment, procedure 150 utilizes profiling of identified similar historical tests and associations (as described in step 208) to predict and / or generate the elapsed time of the current test and association construct. In another embodiment, procedure 150 utilizes any identified historical tests with a similarity score greater than (i.e., exceeding) a specified similarity score threshold (e.g., 75% similarity) or K nearest neighbors (e.g., K=5) to predict the target variable and / or dependent variable of the current test. In this embodiment, procedure 150 utilizes profiling data as input to a KNN, where the elapsed time is the target variable. In another embodiment, procedure 150 utilizes K nearest neighbor voting to predict the elapsed time of the test, with each test weighted by its corresponding average distance from the test point (e.g., regression data point).

[0026] In various embodiments, in response to identifying and retrieving data associated with the determined K nearest neighbor regression tests, procedure 150 utilizes the K nearest neighbor aggregation or average to predict the elapsed time of the regression test, construction, function, and / or query. In one embodiment, procedure 150 utilizes a modified KNN algorithm that implements the option to eliminate test data points (i.e., test data points are not considered neighbors), removes bias, and provides the option to specify whether to use the average or median of the elapsed time to predict the elapsed time of the test. This embodiment improves prediction quality by reducing the influence of outlier neighbors, where outliers are influenced and compounded by variability. In this embodiment, procedure 150 prioritizes median utilization over average utilization (e.g., applying greater weight).

[0027] In various embodiments, program 150 dynamically adjusts one or more thresholds that are crucial for detecting regressions and degradations. In one example, program 150 modifies (i.e., adjusts) one or more thresholds based on factors associated with the relevant software (e.g., code complexity, execution time, and / or system specifications). For example, for software including long (e.g., greater than (i.e., exceeding) timeout periods (e.g., 30 seconds)) running queries, program 150 fine-tunes (i.e., modifies) the degradation threshold to 5% of the associated baseline. In this example, for short-running queries, the threshold could lead to variability issues (i.e., false alarms). For example, if the query runtime is 5 seconds, an increase of 0.25 seconds would trigger a regression alarm. In different embodiments, program 150 utilizes multiple thresholds to further segment and analyze potential regressions.

[0028] If the predicted elapsed time is less than (or does not exceed) the actual elapsed time (“Yes” branch, decision box 212), then procedure 150 terminates. In one embodiment, procedure 150 determines that the regression (i.e., the test of the regression) is subject to variability (i.e., false positives), and procedure 150 suspends, terminates, releases the isolated code, containerizes, packages, and / or deploys the software to one or more environments (e.g., servers, cloud, etc.). In response, procedure 150 determines with high confidence (e.g., >95%) that the regression is attributable to variability and therefore no further analysis is required.

[0029] If the predicted elapsed time is greater than the actual elapsed time, procedure 150 applies a mitigation action to the software (step 214). In one embodiment, procedure 150 determines whether the predicted elapsed time is greater than (i.e., exceeds) the actual elapsed time or greater than a regression (i.e., degradation) threshold relative to an associated baseline. If so, procedure 150 determines with high confidence (e.g., >95%) that the regression is not due to variability and requires further action. In response, procedure 150 performs one or more mitigation actions based on associated diagnostics and historical actions. For example, procedure 150 identifies multiple similar historical builds, regressions, and associated mitigation actions used to repair the historical regression. In different embodiments, procedure 150 determines multiple possible mitigation actions and weights each action based on build similarity (e.g., generated similarity score), effectiveness, computational cost, and susceptibility to further regression (e.g., susceptibility percentage). In one embodiment, procedure 150 prompts the user to select a mitigation action. In another embodiment, procedure 150 automatically initiates one or more mitigation actions based on the associated similarity score and associated historical tests. In a further embodiment, program 150 recommends one or more adjustments to code, such as patches or bug fixes. In another embodiment, program 150 pushes code to an isolated branch of the source code tree. In yet another embodiment, program 150 concurrently deploys known working builds instead of regression builds.

[0030] In one embodiment, program 150 may utilize various communication and transmission methods to notify, push, and / or transmit (e.g., send) one or more notifications to one or more computing devices (not depicted) associated with a user or one or more administrators. These communication and transmission methods include, but are not limited to, Short Message Service (SMS), email, push notifications, automated phone calls, text-to-voice communication, Git client alerts, etc. In one embodiment, program 150 transmits data containing at least one of the following: identified confidential information, identified security vulnerabilities, associated code snippets (e.g., a code snippet containing an exposed database connection string), and associated remedial actions, as described below. In a different embodiment, program 150 suspends activity (e.g., retains code in an isolated area) until a response is received from a licensed user. In another embodiment, program 150 deletes and removes isolated codebases for which program 150 has not received a user response. In this embodiment, program 150 specifies a response threshold for setting a time period for user responses. In a different embodiment, as detailed below, program 150 transmits generated reports.

[0031] In another embodiment, program 150 generates a report containing test results and diagnostics (i.e., identified regressions, historical similarity tests and software, historical mitigation actions, generated scores, etc.). In various embodiments, program 150 constructs a document (e.g., a downloadable document, spreadsheet, image, graph, etc.) containing the generated report. In this embodiment, the document is a digital or physical document (e.g., printable). In another embodiment, program 150 creates a visual representation of the report, allowing the user to interact, add, modify, and / or remove one or more scans and / or tests. In yet another embodiment, program 150 presents one or more scan and test results on a graphical user interface (not depicted) or a web graphical user interface (e.g., generating a Hypertext Markup Language containing the generated results). Program 150 may output the scan and test results in various suitable formats, such as text files, HTML files, CSS files, JavaScript files, documents, spreadsheets, etc.

[0032] Figure 3Table 300 is depicted according to an illustrative embodiment of the invention. Table 300 represents a table created from one or more data mining and text analysis software applications, based on the modified KNN utilized in step 210 and the identified nearest neighbors identified in step 208. Table 300 contains multiple rows, each representing a regression test, wherein each test contains multiple columns containing information about the predicted elapsed time, the actual elapsed time, and the K nearest neighbors of multiple aggregates with associated computed distances (i.e., similarity). In the example, program 150 predicts an elapsed time of 16.272 and determines that the actual elapsed time is 67.948. Here, program 150 determines that the regression is a true regression and is not subject to variability.

[0033] Figure 4 A graph 400 is depicted according to an illustrative embodiment of the invention. Graph 400 compares the regression percentages specific to the database query workload constructed for multiple tests. Graph 400 shows the actual regression value and the predicted regression value for each test point. Here, procedure 150 identifies that the predicted elapsed time has significantly less fluctuation (i.e., variability) compared to the actual elapsed time regression. In the example, procedure 150 adjusts the regression threshold at a 5% baseline deviation to reduce the occurrence of false positives.

[0034] Figure 5 A graph 500 is depicted according to an illustrative embodiment of the invention. Graph 500 depicts the results from a comparison of mean / median predictions. Graph 500 further depicts the percentage degradation relative to a baseline of predicted elapsed time calculated using actual elapsed time and aggregated from the mean / median of the 10 nearest neighbors (e.g., K=10). Graph 500 shows that the predicted elapsed time is less volatile compared to the actual elapsed time. Graph 500 further shows that the elapsed time based on the mean / median prediction follows similarly. Graph 500 shows that, at a degradation threshold >= 5%, the mean-based prediction, labeled 21 / 68, is degraded to the actual regression, while the median-based prediction is only labeled 12 / 68.

[0035] Figure 6 A block diagram 600 depicts components of a server computer 120 according to an illustrative embodiment of the present invention. It should be understood that... Figure 6 This illustration provides only one possible implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment.

[0036] Each of the server computers 120 includes a communication structure 604 that provides communication between a cache 603, a memory 602, a persistent memory 605, a communication unit 607, and an input / output (I / O) interface 606. The communication structure 604 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communication structure 604 can be implemented using one or more buses or crossover switches.

[0037] Memory 602 and persistent memory 605 are computer-readable storage media. In this embodiment, memory 602 includes random access memory (RAM). Typically, memory 602 may include any suitable volatile or non-volatile computer-readable storage medium. Cache memory 603 is a fast memory that enhances the performance of computer processor 601 by retaining recently accessed data from memory 602 and data in proximity to the accessed data.

[0038] Program 150 can be stored in persistent memory 605 and memory 602 for execution by one or more of the corresponding computer processors 601 via cache 603. In an embodiment, persistent memory 605 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent memory 605 may include a solid-state drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0039] The media used in persistent memory 605 can also be removable. For example, a removable hard disk drive can be used for persistent memory 605. Other examples include optical discs and disks, thumb drives, and smart cards, which are inserted into the drive for transfer to another computer-readable storage medium that is also part of persistent memory 605. Software and data 612 can be stored in persistent memory 605 for access and / or execution by one or more of the respective processors 601 via cache 603.

[0040] In these examples, communication unit 607 provides communication with other data processing systems or devices. In these examples, communication unit 607 includes one or more network interface cards. Communication unit 607 can provide communication via physical and / or wireless communication links. Program 150 can be downloaded to permanent memory 605 via communication unit 607.

[0041] I / O interface 606 allows for data input and output to other devices that can be connected to server computer 120. For example, I / O interface 606 may provide connectivity to multiple external devices 608, such as keyboards, keypads, touchscreens, and / or other suitable input devices. External devices 608 may also include portable computer-readable storage media, such as thumb drives, portable optical discs or disks, and memory cards. Software and data (e.g., program 150) used to implement embodiments of the invention may be stored on such portable computer-readable storage media and may be loaded onto permanent memory 605 via I / O interface 606. I / O interface 606 is also connected to display 609.

[0042] The display 609 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0043] The procedures described herein are identified based on their implementation in specific embodiments of the invention. However, it should be understood that any particular procedural terminology used herein is for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such terminology.

[0044] The present invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0045] Computer-readable storage media can be tangible devices that can hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0046] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0047] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++; conventional procedural programming languages ​​such as the "C" programming language or similar programming languages; quantum programming languages ​​such as the "Q" programming language, Q#, the Quantum Computing Language (QCL) or similar programming languages; and low-level programming languages ​​such as assembly language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized to execute computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform aspects of the present invention.

[0048] This document describes various aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0049] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0050] These computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, thereby causing the instructions to be executed on the computer, other programmable apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0051] The flowcharts and block diagrams in the figures (i.e., the accompanying drawings) illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0052] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to techniques found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: In response to the detection that a client is pushing a first software build to the code management system, the first software build is regressed and tested by one or more computer processors. A summary analysis of the central processing unit (CPU) associated with a regression test built with first software, using one or more computer processors, monitors CPU execution time, CPU temperature, minimum CPU utilization, maximum CPU utilization, average CPU utilization, and memory utilization. In response to the detection of regression in a first software build, one or more computer processors identify one or more historical regression tests and the one or more historical software builds using a computed similarity metric between the first software build and one or more historical software builds performed at the same release cycle, wherein the identified one or more historical regression tests and historical software builds are the K nearest neighbors (KNN) of the first software build, and wherein the similarity metric is computed using CPU profiling analysis. The elapsed time of the regression test is predicted by one or more computer processors using the KNN algorithm, which includes the K nearest neighbors weighted by the corresponding average distance of each free distance test point and the elapsed time as the target variables. In response to the predicted elapsed time exceeding the actual elapsed time associated with the first software build, one or more computer processors determine that the detected regression is the actual regression; as well as In response to determining that the detected regression is not due to variability, one or more computer processors mitigate the first software build based on mitigation actions associated with one or more identified historical software builds.

2. The method according to claim 1, wherein, Predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, further includes: The KNN algorithm is modified by one or more computer processors to remove test point bias by removing test points that are not associated with the K nearest neighbors.

3. The method according to claim 1, wherein, Predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, includes: The elapsed time is predicted by one or more computer processors using the average of the K nearest neighbors.

4. The method according to claim 1, wherein, Predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, includes: The elapsed time is predicted by one or more computer processors using the median of the K nearest neighbors.

5. The method of claim 1, further comprising: In response to the predicted elapsed time not exceeding the actual elapsed time associated with the first software build, one or more computer processors determine that the regression build is due to variability; as well as In response to the determination that the regression build is due to variability, the first software build is deployed in one or more environments by one or more computer processors.

6. The method according to claim 1, wherein, The similarity calculations are based on statistics from the system profiling analyzer.

7. The method according to claim 6, wherein, The system profile analyzer statistics include CPU profile analysis, GPU profile analysis, input / output profile analysis, and network profile analysis.

8. The method according to claim 7, wherein, The CPU profiling analysis generates statistics, including execution time, temperature, minimum CPU utilization, maximum CPU utilization, media CPU utilization, average CPU utilization, and associated memory utilization.

9. The method according to claim 1, wherein, The similarity calculation utilizes Euclidean distance.

10. A computer program product, comprising: One or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, wherein the stored program instructions include: In response to the detection that the client is pushing the first software build to the code management system, regression testing is performed on the program instructions of the first software build; The summary analysis is performed on the program instructions of the central processing unit (CPU) associated with the regression test of the first software build, wherein the summary analysis monitors CPU execution time, CPU temperature, minimum CPU utilization, maximum CPU utilization, average CPU utilization and memory utilization. In response to the detection of regression in the first software build, one or more historical regression tests and program instructions of the one or more historical software builds are identified using a calculated similarity metric between the first software build and one or more historical software builds performed in the same release cycle, wherein the identified one or more historical regression tests and historical software builds are the K nearest neighbors (KNN) of the first software build, and wherein the similarity metric is calculated using CPU profiling analysis. The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors weighted by the corresponding average distance of each free distance test point and the elapsed time as the target variable; In response to the predicted elapsed time exceeding the actual elapsed time associated with the first software build, the program instruction that the detected regression is the actual regression is determined; and In response to determining that the detected regression is not due to variability, the program instructions of the first software build are mitigated based on mitigation actions associated with one or more identified historical software builds.

11. The computer program product according to claim 10, wherein, The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, further include: The KNN algorithm is modified to include program instructions that remove test point bias by removing test points that are not associated with the K nearest neighbors.

12. The computer program product according to claim 10, wherein, The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, further include: The program instruction for the elapsed time is predicted using the average of the K nearest neighbors.

13. The computer program product according to claim 10, wherein, The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, further include: The program instruction for predicting the elapsed time is used by the median of the K nearest neighbors.

14. The computer program product according to claim 10, wherein, The similarity calculations are based on statistics from the system profiling analyzer.

15. A computer system comprising: One or more computer processors; One or more computer-readable storage media; as well as Program instructions stored on the computer-readable storage medium for execution by at least one of the one or more processors, the stored program instructions including: In response to the detection that the client is pushing the first software build to the code management system, regression testing is performed on the program instructions of the first software build; The summary analysis is performed on the program instructions of the central processing unit (CPU) associated with the regression test of the first software build, wherein the summary analysis monitors CPU execution time, CPU temperature, minimum CPU utilization, maximum CPU utilization, average CPU utilization and memory utilization. In response to the detection of regression in the first software build, one or more historical regression tests and program instructions of the one or more historical software builds are identified using a calculated similarity metric between the first software build and one or more historical software builds performed in the same release cycle, wherein the identified one or more historical regression tests and historical software builds are the K nearest neighbors (KNN) of the first software build, and wherein the similarity metric is calculated using CPU profiling analysis. The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors weighted by the corresponding average distance of each free distance test point and the elapsed time as the target variable; In response to the predicted elapsed time exceeding the actual elapsed time associated with the first software build, the program instruction that the detected regression is the actual regression is determined; and In response to determining that the detected regression is not due to variability, the program instructions of the first software build are mitigated based on mitigation actions associated with one or more identified historical software builds.

16. The computer system according to claim 15, wherein, The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, further include: The program instructions modify the KNN algorithm to remove test point bias by removing test points that are not associated with the K nearest neighbors.

17. The computer system according to claim 15, wherein, The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, further include: The program instruction for the elapsed time is predicted using the average of the K nearest neighbors.

18. The computer system according to claim 15, wherein, The program instructions for predicting the elapsed time of the regression test using the KNN algorithm, which includes the K nearest neighbors and the elapsed time as the target variable, further include: The program instruction for predicting the elapsed time is used by the median of the K nearest neighbors.

19. The computer system according to claim 15, wherein, The similarity calculations are based on statistics from the system profiling analyzer.

20. The computer system according to claim 15, wherein, The similarity calculation utilizes Euclidean distance.

Images