Scalable Code Testing and Benchmarking

Abstract

A method or system for testing and benchmarking commits made in source code. The system extracts commits from a history of a first code history that records a sequence of commits made thereon. The extracted commits are combined into a sequence of patches, each patch including changes made during successive commits. The system also establishes a connection with a system under test (SUT) having a second code repository that corresponds to the historical version of the first code repository before the extracted commits are made, and sequentially applies the sequence of patches to the second code repository. Performance of the SUT is monitored during application of the sequence of patches to determine whether the SUT performs as expected after application of each patch.

Description

[Technical Field]

[0001] The present disclosure relates generally to source code management (SCM), and more particularly to testing and benchmarking commits made to a source code repository in a continuous delivery (CI) and / or continuous deployment (CD) process. [Background technology]

[0002] This application claims the benefit of priority to U.S. Application No. 17 / 964,832, entitled "Scalable Code Testing and Benchmarking," filed October 12, 2022, which is incorporated herein by reference.

[0003] Agile methods in software development involve self-organizing, cross-functional teams working collaboratively with customers and / or end users to identify requirements and improve solutions, adaptive planning, evolutionary development, early delivery, continuous improvement, and flexibility in responding to changes in requirements, capabilities, and understanding of the problem to be solved. Recently, more and more developers have adopted agile software development. A recent survey by KPMG of 120 companies in 17 countries revealed that 70% of them have already moved to agile or are in the process of moving to agile. The primary motivation for this move is to deliver products at a faster rate while increasing customer satisfaction.

[0004] The growing popularity and adoption of source code management (SCM) systems combined with continuous integration and continuous delivery (CI / CD) processes has contributed to the adoption of agile software development. While SCM systems allow developers to track code changes, maintain a history of these changes, and roll back or revert changes, CI / CD provides automation and tooling around the integration and deployment of code changes (e.g., compilation, syntax checks, compliance checks, running unit tests, security testing, etc.). Summary of the Invention

[0005] The principles described herein relate to a method for testing and benchmarking commits to a code repository. The method may be performed by a source code management (SCM) system. The SCM system extracts commits made from the history of a first (source) code repository. The history records a sequence of commits to the first code repository. Each commit includes at least one change made to at least one file in the first code repository. The SCM system combines at least a subset of the commits into a sequence of patches. Each patch includes code changes that occurred during successive commits. The SCM system establishes a connection with a system under test (SUT) that has a second (target) code repository. The second code repository corresponds to a historical version of the first code repository before the extracted commits were made. The sequence of patches is then sequentially applied to the second code repository. Applying the sequence of patches includes applying one patch in the sequence to the second code repository and monitoring performance of the SUT after applying the patch to the SUT. In response to determining that the SUT operates as expected after application of the patch, the next patch in the sequence is applied to the second code repository.

[0006] In some embodiments, monitoring the performance of the SUT after applying the patches includes executing a script configured to collect data associated with the performance of the SUT after the patches are applied, and determining whether the SUT is operating as expected based on the collected data. In some embodiments, the data is collected via an API that triggers the execution of the script in response to the application of each patch.

[0007] In some embodiments, extracting the commits from the history includes extracting a subset of the commits made based on specified criteria and combining only the subset of the commits into the sequence of patches. In some embodiments, the specified criteria includes a time frame having a start time and an end time, and extracting the subset of commits includes extracting commits made within the time frame.

[0008] In some embodiments, the specified criteria include a permission list specifying one or more files that are allowed to be modified, and extracting the subset of commits includes extracting commits that modify at least one file specified in the permission list. In some embodiments, extracting commits that modify at least one file specified in the permission list includes, for each commit in the history, generating a first hash based on one or more files in the permission list before the commit was made, generating a second hash based on the one or more files after the commit was made, and determining whether the first hash and the second hash are different. In response to determining that the first hash and the second hash are different, it is determined that the commit modifies at least one file in the permission list. In response to determining that the commit modifies at least one file in the permission list, the commit is extracted.

[0009] In some embodiments, combining the extracted commits into a sequence of batches includes combining N consecutive commits of the extracted commits into a single patch.

[0010] In some embodiments, the method further includes initializing a new repository on the SUT as the second code repository based on a historical version of the first code repository from before at least a subset of the commits were made. In some embodiments, the method further includes initializing the SUT in the container based on the container image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of a networked computing environment in which a source code management (SCM) system may be implemented, according to one embodiment. [Figure 2] FIG. 2 is a block diagram of the test and benchmark module shown in FIG. 1 according to one embodiment. [Figure 3] FIG. 3 is a flowchart of a method for testing and benchmarking commits to a code repository, according to one embodiment. [Figure 4] FIG. 4 is a flowchart of a method for sequentially applying a sequence of patches to a code repository of a system under test (SUT), according to one embodiment. [Figure 5] FIG. 5 is a block diagram illustrating an exemplary computer suitable for use in the network computing environment of FIG. 1, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The drawings and the following description describe specific embodiments for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of structure and methods may be employed without departing from the principles described. Wherever possible, like or similar reference numbers have been used in the figures to indicate similar or similar functionality. When elements share a common digit followed by another letter, this indicates that the elements are similar or identical. Unless the context dictates otherwise, reference to a digit alone generally refers to any one or any combination of such elements.

[0013] Software developers often collaborate with operations and security teams to integrate security early in the application lifecycle, taking responsibility for application security and operations as part of their workflow. This software development philosophy is also known as DevSecOps. Continuous integration (CI) and / or continuous deployment (CD) practices build on the foundation of DevSecOps and aim to streamline the software development process by automatically ensuring a certain level of quality (fewer bugs and fewer security issues) for newly released software features while maintaining high release velocity and providing short feedback cycles to customers. These short feedback cycles can be used to monitor the impact of features from the moment they are released and to inform developers and product managers about the success or failure of specific deployments. This allows data-driven decisions to be made regarding feature integration, retirement, and / or improvement. This data-driven decision-making (DDDM) process is a valuable part of agile software development, providing systematic evolution of software products or services.

[0014] However, this leads to a chicken-or-egg problem: even small features must be deployed or released before their impact can be measured. DDDM generally requires data to be collected first to assess the value of a feature. Releasing a feature without data on its impact risks having to remove it if it doesn't work as expected. Furthermore, integrating or partially integrating a proof-of-concept (which is not necessarily a robust product feature) can increase risks to the stability and security of the product. To mitigate these issues, developers and product managers may limit the rate at which new features are developed and deployed, which goes against the agile philosophy.

[0015] Commonly applied software testing best practices, such as the use of staging environments, automated system / integration testing strategies, and feature flags that allow functionality to be gated, may help address some of the stability and potential security issues, but do not provide insight into DDDM, where more targeted testing strategies, such as variant testing, exist.

[0016] A / B testing, an example of variant testing, can aid DDDM by running controlled experiments that have not yet been fully deployed. New features are evaluated by assessing their interaction with a selected group of users who are presented with different variants of the test target. A / B testing is most commonly applied to evaluate user interface changes. However, variant testing strategies such as A / B testing are typically applied to monitor customer experience, primarily through user interface integrations. These forms of testing strategies are not always applicable in highly automated environments.

[0017] The SCM system and / or testing and benchmarking methods described herein address the above problems by allowing developers and product managers to run and evaluate the results of DDDM experiments without deploying features to a production environment, and therefore not limit the speed at which developers can prototype new ideas.

[0018] System Example 1 is a block diagram of one embodiment of a network computing environment 100 in which an SCM system may be used. In the embodiment shown, environment 100 includes a developer device 110, a source repository 120, an SCM system 130, and a system under test (SUT) 140. In other embodiments, network computing environment 100 may include different or additional elements. Furthermore, functionality may be distributed among elements in a manner different from that described.

[0019] Developer device 110 is a computing device configured to make changes or commits to source repository 120 (also referred to as the first repository). A commit is a change made to a file or a set of changes made to a set of files. SCM system 130 is a computer system that tracks and provides control over changes to code stored in source repository 120. For example, in some embodiments, when a commit is made, SCM system 130 may create a unique ID and keep a record of the specific changes committed along with who made them, when the changes were made, etc. Such a record is also referred to as a commit record (denoted C). For simplicity, hereafter, the terms "commit" and "commit record" are used interchangeably. The collection 122 of all commits to source repository 120 is also referred to as a "history" (denoted H), where H={c1, c2, ..., c n}, where c1 is the first commit, c n indicates the last commit, and n indicates the index number of the last commit. In some embodiments, each commit in history 122 isx The timestamp t(c x ), where t(c x )≦t(c x+1 ) The history 122 may be stored on the developer device 110 in association with the source repository 120, cloud storage, and / or the SCM system 130.

[0020] The SCM system 130 includes a test and benchmark module 132 configured to run tests and benchmarks on commits made to the source repository 120 via a system under test (SUT) 140. The SUT 140 can be a local or remote service accessible by the test and benchmark module 132 via an API 146, a local tool, or provided in the form of a container image (e.g., a Docker image). The SUT 140 has a target repository 142 (also referred to as a second repository). The target repository 142 contains code corresponding to historical versions of the source repository 120. The SUT 140 also runs one or more services 144. In a CI / CD situation, a pushed commit can trigger one or more services 144 on the SUT. The one or more services 144 can include, but are not limited to, program analysis tools, build automation tools, etc.

[0021] Test and benchmark module 132 applies at least a subset of the commits to source repository 120 onto target repository 142 and runs benchmark tests on SUT 140 to determine whether SUT 140 with the commits functions as intended. In some embodiments, the test and benchmark process includes three phases: (A) a record phase extracts a subset of the commits applied to the source code repository; (B) a patch sequence generation phase prepares a sequence of patches to be applied to SUT 140; and (C) a monitor phase in which the actual patches are applied and the behavior of the SUT is observed and evaluated.

[0022] During the record phase, the test and benchmark module 132 extracts commits from the source history 122 that are relevant for a given test criterion. During the patch sequence generation phase, the test and benchmark module 132 generates a sequence of patches. During the monitor phase, the test and benchmark module 132 replays the generated sequence on the SUT 140 and captures results and / or metrics by running evaluation scripts to capture timing information, API 146 responses, files, etc. from the SUT 140.

[0023] In some embodiments, in the context of applying DDDM for newly implemented functionality, test and benchmark module 132 can be used to apply an automated form of variant testing by feeding the data recorded in the record phase to the original, unmodified SUT and to the modified SUT that includes the functionality to be evaluated. In some embodiments, test and benchmark module 132 evaluates both systems by replaying the same source history to both of them and capturing results and metrics calculated by configurable evaluation scripts.

[0024] In some embodiments, the captured evaluation results and metrics can be used for DDDM by (1) helping developers and product managers better understand and quantify the impact of newly developed features before integrating them into the product, and (2) discovering and identifying bugs and / or regressions while replaying events from the project history.

[0025] 2 is a block diagram of one embodiment of test and benchmark module 132. In the embodiment shown, test and benchmark module 132 includes a recording module 210, a patch sequence generation module 220, and a monitor module 230. In other embodiments, test and benchmark module 132 includes different or additional elements. Furthermore, functionality may be distributed among the elements in a manner different from that described.

[0026] The recording module 210 extracts at least a subset of relevant commits for a given test criterion from the history H of the source repository 120. The subset of commits is a slice S={c1, c2, ..., c m The slice S is then persisted to the record store 212. Extracting a subset of commits based on test criteria not only reduces response time but also allows the analysis to be focused on specific files, which allows the test and benchmark module 132 to scale to large repositories, both in terms of history size and the number of files stored in the repository.

[0027] The test criteria is the start time T min Starts at and ends at time T max A specified time window T=[T min , T max In such cases, T is used to extract a subset of commits that occurred within the time window T, which is called a time slice. A time slice is defined as S`={c s |c s ∈H∧Tmin ≦t(c s )≦T max Additionally or alternatively, the test criteria may include a permission list containing the names of one or more files on which tests and benchmarks are focused. Thus, a slice S can be expressed as S={c s |c s ∈S`∧allowlist(c s )}, where commit c s If the is related to a file specified in the allow list, then allowlist(c s ) evaluates to true. If no allowlist is specified, allowlist(c s ) always evaluates to true. The order of commits is preserved, so t(c s )≦t(c s+1 ) generally remains true.

[0028] In one embodiment, extracting the subset of commits includes extracting commits that modify at least one file specified in the permission list. The subset of commits may be extracted by, for each commit in history H122, generating a first hash of one or more files in the permission list before the commit was made, generating a second hash of the one or more files after the commit was made, and determining whether the first hash and the second hash are different. If the first hash and the second hash are different, the commit modified at least one file in the permission list and the commit is extracted.

[0029] The patch sequence generation module 220 generates a sequence of patches 222 based on the slice S recorded in the record store 212. In one embodiment, a patch includes code changes made in at least one commit that will be automatically applied to the target repository 142 on the SUT 140. A patch may include a sampling number N of consecutive commits, where N is referred to as the patch sampling number. Thus, N consecutive commits can be combined into a single patch. In such a case, the patch sequence may be expressed as a fraction from the slice S in the following equation:

[0030]

number

[0031] For example, for a slice S={c1, c2, c3, c4, c5, c6} with a patch sampling count of N=2, the patch sequence generation module 220 obtains a patch sequence PS={{c1, c2}, {c3, c4}, {c5, c6}}, where each of the inner sets represents a patch (e.g., patch1={c1, c2} includes two commits that are automatically applied to the target code repository 142 on the SUT 140). In addition to the time window T and the permission list, the patch sampling count is another tool that increases the scalability of the testing and benchmarking module by enabling projects with large histories to be tested.

[0032] 2, monitor module 230 includes replay module 232 and evaluation module 234. During the monitor phase, replay module 232 replays sequence of patches 222 on target code repository 142 on SUT 140. In some embodiments, replay module 232 makes no assumptions about SUT 140 and only requires that SUT 140 be accessible by API 146 that enables monitor module 230 or any external system to (1) wait for the execution of services / actions / jobs that may be triggered when applying patches, and (2) extract relevant test / evaluation data. In some embodiments, SUT 140's API 146 communication and data extraction may be provided in the form of scripts as part of the configuration of monitor module 230.

[0033] In one embodiment, the monitor module 230 natively supports the presence of a target code repository 142 within the SUT 140, which may spawn additional jobs, such as CI / CD jobs. The additional jobs may be considered to belong to the SUT 140 itself. In the presence of a target code repository, before initiating replay of a sequence of patches, the replay module 232 may initialize a new target code repository with a history based on the history or partial history of the source repository on the SUT 240.

[0034] In some embodiments, the replay module 230 can also overwrite specific files on the SUT 240. This can be advantageous when handling situations where specific code files stored on the SUT 240 may affect the replay. In a CI / CD environment, it is common for the CI / CD configuration itself to be stored in the SCM system 130. However, it is not always necessary to run the entire set of CI / CD jobs. When benchmark tests are run to evaluate specific jobs that are part of a larger CI / CD deployment, it is sufficient to run the jobs individually, which can be controlled using a custom CI / CD that can override the standard configuration.

[0035] In one embodiment, the replay module 232 sequentially and iteratively applies each patch in the sequence of patches on the target code repository 142 on the SUT 140 via the API 146. In the presence of a target repository, the patch may be submitted directly to the target repository. After applying the patch, the monitor module 234 monitors the effects of the patch application. This step (1) ensures that the SUT behaves as expected (also called testing) and (2) extracts and / or collects results and / or metrics from the SUT 140 while applying the patch by invoking an evaluation script (also called benchmark). The evaluation script may be user-defined or provided by the test and benchmark module 132. For example, the evaluation script may collect data from the API 146 exposed by the SUT 140 and / or parse a structured data file (e.g., a JSON file or an XML file) generated by the SUT 140.

[0036] Exemplary Methods 3 is a flowchart of a method 300 for testing and benchmarking commits to a code repository, according to one embodiment. The steps of FIG. 3 are described from the perspective of SCM 130 performing method 300. However, some or all of the steps may be performed by other entities or components. Additionally, some embodiments may perform steps in parallel, perform steps in a different order, or perform different steps.

[0037] The SCM system relies on a version-controlled source code repository 120, with test data maintained in the repository's history 122. Because the SCM system leverages the source code repository 120 as input data to be tested and benchmarked, it is able to replay the history of the source code project, which is particularly useful when applying DDDM to test and evaluate newly developed functionality in a CI / CD context, including but not limited to:

[0038] In particular, the SCM system extracts 310 commits from the history 122 of the first code repository 120. The history 122 records a sequence of commits to the first code repository 120. Each commit includes at least one change made to at least one code file in the first code repository 120. Extracting 310 commits may include extracting a subset of commits from the history 122 based on specified criteria. In one embodiment, for each commit, the history 122 records the time the commit was made. The specified criteria include a time frame having a start time and an end time. Extracting the commits includes extracting commits made within the time frame. Additionally or alternatively, the specified criteria include a permission list specifying one or more files that are allowed to be modified. In such a case, extracting 310 commits includes extracting commits that modify at least one file specified in the permission list.

[0039] Regardless of the exact method used to extract 310 the commits, the SCM system combines 320 the extracted commits into a sequence of patches, where each patch includes changes made in one or more code files that occurred during successive commits. In one embodiment, combining at least a subset of the commits into a sequence of batches includes combining N consecutive commits of the extracted commits into a single patch, where N is a positive integer.

[0040] The SCM system establishes 330 a connection with the SUT 140 having the second code repository 142. In one embodiment, the connection with the SUT 140 is established via an API 146. The SCM system 140 may create the second code repository 142 based on a historical version of the first code repository 120 from before the extracted commit was made. Alternatively, the SCM system 140 may initialize an existing repository on the SUT 140 as the second code repository 142 based on a historical version of the first code repository 120 from before the extracted commit was made. In some embodiments, the SCM system initializes the SUT 140 in a container based on a container image.

[0041] The SCM system applies 340 the sequence of patches to the second code repository 142. Figure 4 is a flowchart of a method 400 for applying 240 the sequence of patches to the second code repository 142, according to one embodiment. In the embodiment shown, the SCM system 130 applies 410 the patches in the sequence to the second code repository 142. Each patch is generated by combining one or more successive commits. Thus, the patch includes one or more changes made to one or more code files that occurred between one or more successive commits.

[0042] The SCM system 130 monitors 420 the effects of the application of the patch to the SUT 140. The SCM system 130 determines 430 whether the SUT 140 is operating as expected. In one embodiment, the SUT 140 monitors 420 the effects of the patch by executing a script that collects data associated with the performance of the SUT after the patch is applied, and determines 430 whether the SUT is operating as expected based on the collected data. The data may be collected via an API 146 that triggers the execution of the script in response to the application of the patch.

[0043] If SUT 140 does not behave as expected, SCM 130 stops further application of patches. In one embodiment, SCM 130 notifies the developer that the corresponding commit did not cause SUT 140 to behave as expected. Alternatively or additionally, SCM 130 may present the collected data to the developer for review. Conversely, if SUT 140 behaves as expected, SCM 130 applies 440 the next patch in the sequence to second code repository 142. This process is repeated until all patches in the sequence have been applied or until a particular commit causes the SUT to behave differently from the behavior expected at the particular commit.

[0044] Computing System Architecture 5 is a block diagram of an exemplary computer 500 suitable for use as a developer device 110, an SCM system 130, or an SUT 140, or for hosting a source code repository 120. The exemplary computer 500 includes at least one processor 502 coupled to a chipset 504. The chipset 504 includes a memory controller hub 520 and an input / output (I / O) controller hub 522. A memory 506 and a graphics adapter 512 are coupled to the memory controller hub 520, and a display 518 is coupled to the graphics adapter 512. A storage device 508, a keyboard 510, a pointing device 514, and a network adapter 516 are coupled to the I / O controller hub 522. Other embodiments of the computer 500 have different architectures.

[0045] 5, storage device 508 is a non-transitory computer-readable storage medium such as a hard drive, compact disc read-only memory (CD-ROM), DVD, or solid-state memory device. Memory 506 holds instructions and data used by processor 502. Pointing device 514 is a mouse, trackball, touchscreen, or other type of pointing device and is used in combination with keyboard 510 (which may be an on-screen keyboard) to input data into computer system 500. Graphics adapter 512 displays images and other information on display 518. Network adapter 516 couples computer system 500 to one or more computer networks, such as network 170.

[0046] 1 and 2 may vary depending on the embodiment and the processing power required by the entities. For example, a system hosting source code repository 120 may include multiple blade servers operating together to provide the described functionality, while developer device 110 may be a desktop workstation or tablet. Additionally, computer 500 may lack some of the components described above, such as keyboard 510, graphics adapter 512, and display 518.

[0047] Additional Considerations In some portions of the above description, embodiments are described in terms of algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the computing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, they will be understood to be implemented by computer programs including instructions for execution by a processor or equivalent electrical circuits, microcode, or the like. Further, it has proven convenient at times to refer to these arrangements of functional operations as modules, without loss of generality.

[0048] As used herein, a reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment. Similarly, the use of "a" or "an" before an element or component is done merely for convenience. This description should be understood to mean that there are one or more of the element or component, unless it is clear that something else is meant.

[0049] When values ​​are described as "about" or "substantially" (or derivatives thereof), such values ​​should be construed as being exact + / - 10%, unless otherwise clear from the context. From the example, "about 10" should be understood to mean "within the range of 9 to 11."

[0050] As used herein, the terms "comprise," "comprising," "include," "including," "having," "having," or other variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. Furthermore, unless expressly stated otherwise, "or" refers to an inclusive "or," not an exclusive "or." For example, condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0051] Upon reading this disclosure, those skilled in the art will recognize still additional alternative structural and functional designs for systems and processes for tracking vulnerabilities using scope and offset values ​​to reduce instances of overlap. Thus, while specific embodiments and applications have been illustrated and described, it should be understood that the described subject matter is not limited to the precise structures and components disclosed. The scope of protection should be limited only by the scope of any claims at issue.

Claims

1. 1. A method for testing and benchmarking commits to source code, comprising: extracting commits from a history of a first code repository, the history recording a sequence of commits made on the first code repository, each commit in the sequence of commits including at least one change made on at least one code file in the first code repository; combining the extracted commits into a sequence of patches, each patch containing changes to one or more code files that occurred between successive commits; establishing a connection with a system under test (SUT) having a second code repository, the second code repository corresponding to a historical version of the first code repository before the extracted commit was made; sequentially applying the sequence of patches to the second code repository, the step of sequentially applying the sequence of patches comprising: applying the patches in the sequence to the second code repository; monitoring the performance of the SUT after applying the patch to the SUT; applying the next patch in the sequence to the second code repository in response to determining that the SUT operates as expected after applying the patch; A method comprising:

2. monitoring the performance of the SUT after applying the patches in the sequence, running a script configured to collect data associated with the performance of the SUT after the patch has been applied; and determining whether the SUT is operating as expected based on the collected data.

3. The method of claim 2 , wherein the data is collected via an API that triggers execution of the script in response to applying the patch.

4. The method of claim 1 , wherein extracting commits comprises extracting a subset of commits from the history based on specified criteria.

5. For each commit, The history records the time when the commit was made; the specified criteria include a time frame having a start time and an end time; The method of claim 4 , wherein extracting the subset of commits comprises extracting commits that occurred within the time window.

6. 5. The method of claim 4, wherein the specified criteria includes a permission list specifying one or more files that are allowed to be modified, and extracting the subset of commits includes extracting commits that modify at least one file specified in the permission list.

7. The step of extracting commits that modify at least one file specified in the permission list includes: For each commit in the history, generating a first hash based on the one or more files in the permission list before the commit occurs; After the commit occurs, generating a second hash based on the one or more files; determining whether the first hash and the second hash are different; determining that the commit modifies at least one file in the permission list in response to determining that the first hash and the second hash are different; extracting the commit in response to determining that the commit modifies at least one file in the permission list.

8. The method of claim 1 , wherein combining the extracted commits into a sequence of patches comprises combining N consecutive ones of the extracted commits into a single patch.

9. 2. The method of claim 1, further comprising: initializing a repository on the SUT as the second code repository based on at least a historical version of the first code repository before the extracted commit occurred.

10. The method of claim 1 , further comprising: initializing the SUT in a container based on a container image.

11. 1. A computer program product having a non-transitory computer-readable storage medium encoded with instructions that, when executed by a processor, cause the processor to: extracting commits from a history of a first code repository, the history recording a sequence of commits made on the first code repository, each commit in the sequence of commits including at least one change made on at least one code file in the first code repository; Combining the extracted commits into a sequence of patches, each patch including changes to one or more code files that occurred between successive commits; establishing a connection with a system under test (SUT) having a second code repository, the second code repository corresponding to a historical version of the first code repository before the extracted commit was made; sequentially applying the sequence of patches to the second code repository, wherein sequentially applying the sequence of patches includes: applying the patches in the sequence to the second code repository; monitoring the performance of the SUT after applying the patch to the SUT; and in response to determining that the SUT operates as expected after applying the patch, applying a next patch in the sequence to the second code repository.

12. Monitoring performance of the SUT after applying the patches in the sequence includes: running a script configured to collect data associated with the performance of the SUT after the patch is applied; and determining whether the SUT is operating as expected based on the collected data.

13. 13. The computer program product of claim 12, wherein the data is collected via an API that triggers execution of the script in response to applying the patch.

14. 12. The computer program product of claim 11, wherein extracting commits comprises extracting a subset of commits from the history based on specified criteria.

15. For each commit, The history records the time when the commit was made; the specified criteria include a time frame having a start time and an end time; 15. The computer program product of claim 14, wherein extracting the subset of commits comprises extracting commits that occurred within the time window.

16. 15. The computer program product of claim 14, wherein the specified criteria includes a permission list specifying one or more files that are allowed to be modified, and extracting the subset of commits includes extracting commits that modify at least one file specified in the permission list.

17. Extracting commits that modify at least one file specified in the permission list includes: For each commit in the history, generating a first hash based on the one or more files in the permission list before the commit occurs; generating a second hash based on the one or more files after the commit is made; determining whether the first hash and the second hash differ; determining that the commit modifies at least one file in the permission list in response to determining that the first hash and the second hash are different; and extracting the commit in response to determining that the commit modifies at least one file in the permission list.

18. 12. The computer program product of claim 11, wherein combining the extracted commits into a sequence of patches comprises combining N consecutive ones of the extracted commits into a single patch.

19. When executed by a processor, the processor:

12. The computer program product of claim 11, further comprising the non-transitory computer-readable storage medium encoded with additional instructions that cause the computer to initialize a repository on the SUT as the second code repository based on at least a historical version of the first code repository before the extracted commit was made.

20. When executed by a processor, the processor:

12. The computer program product of claim 11, further comprising the non-transitory computer-readable storage medium encoded with additional instructions that cause the non-transitory computer-readable storage medium to: initialize the SUT within a container based on a container image.

Images