Scalable code testing and benchmarking

Abstract

A method or system for testing and benchmarking commits made in source code. The system extracts commits from a history of a first code history that records a sequence of commits made thereon. The extracted commits are combined into a sequence of patches, each patch including changes made during successive commits. The system also establishes a connection with a system under test (SUT) having a second code repository that corresponds to the historical version of the first code repository before the extracted commits are made, and sequentially applies the sequence of patches to the second code repository. Performance of the SUT is monitored during application of the sequence of patches to determine whether the SUT performs as expected after application of each patch.

Description

[Technical Field]

[0001] This disclosure generally relates to source code management (SCM), and in particular to continuous Integration This relates to testing and benchmarking commits made to source code repositories in CI (Continuous Integration) and / or Continuous Deployment (CD) processes. [Background technology]

[0002] This application claims priority to U.S. Application No. 17 / 964,832, entitled “Scalable Code Testing and Benchmarking,” filed on 12 October 2022, which is incorporated herein by reference.

[0003] Agile methodologies in software development include identifying requirements and improving solutions through self-organizing and cross-functional teams collaborating with customers and / or end-users, adaptive planning, evolutionary development, early delivery, continuous improvement, and flexibility in responding to changes in the understanding of requirements, capabilities, and problems to be solved. Recently, more and more developers are adopting agile software development. A recent KPMG survey of 120 companies in 17 countries revealed that 70% have already moved to agile or are in the process of moving to agile. The main motivation for this shift is to deliver products faster while increasing customer satisfaction.

[0004] The growing popularity and adoption of source code management (SCM) systems combined with continuous integration and continuous delivery (CI / CD) processes is contributing to the adoption of agile software development. SCM systems allow developers to track code changes, maintain a history of those changes, and roll back or revert changes, while CI / CD provides automation and tools for integrating and deploying code changes (e.g., compilation, syntax checking, compliance checking, unit test execution, security testing, etc.). [Overview of the project]

[0005] The principles described herein relate to methods for testing and benchmarking commits to a code repository. These methods may be performed by a source code management (SCM) system. The SCM system extracts commits made from the history of a first (source) code repository. The history records a sequence of commits to the first code repository. Each commit contains at least one change made to at least one file in the first code repository. The SCM system combines at least a subset of the commits into a sequence of patches. Each patch contains code changes that occurred during a sequence of commits. The SCM system establishes a connection to a system under test (SUT) having a second (target) code repository. The second code repository corresponds to the historical version of the first code repository prior to the extracted commits. The sequence of patches is then applied sequentially to the second code repository. Applying the sequence of patches includes applying one patch in the sequence to the second code repository and monitoring the performance of the SUT after the patching is complete. In response to the decision that the SUT works as expected after the patch is applied, the next patch in the sequence is applied to the second code repository.

[0006] In some embodiments, monitoring the performance of the SUT after patching includes running a script configured to collect data associated with the performance of the SUT after patching, and determining whether the SUT is operating as expected based on the collected data. In some embodiments, the data is collected via an API that triggers the execution of the script in response to each patch application.

[0007] In some embodiments, extracting commits from history involves extracting a subset of commits made based on specified criteria and combining only the subset of commits into a patch sequence. In some embodiments, the specified criteria include a time frame having a start time and an end time, and extracting a subset of commits involves extracting commits made within that time frame.

[0008] In some embodiments, the specified criteria include a allowlist that specifies one or more files that are permitted to be modified, and extracting a subset of commits includes extracting commits that modify at least one file specified in the allowlist. In some embodiments, extracting commits that modify at least one file specified in the allowlist includes, for each commit in the history, generating a first hash based on one or more files in the allowlist before the commit is made, generating a second hash based on one or more files after the commit is made, and determining whether the first and second hashes are different. In response to determining that the first and second hashes are different, it is determined that the commit modifies at least one file in the allowlist. In response to determining that the commit modifies at least one file in the allowlist, the commit is extracted.

[0009] In some embodiments, combining extracted commits into a batch sequence involves combining N consecutive commits of the extracted commits into a single patch.

[0010] In some embodiments, the method further includes initializing a new repository on the SUT as a second code repository based on a historical version of the first code repository prior to at least a subset of commits being made. In some embodiments, the method further includes initializing the SUT in a container based on a container image. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a block diagram of a network computing environment in which a source code management (SCM) system may be implemented according to one embodiment. [Figure 2] Figure 2 is a block diagram of the test and benchmark module shown in Figure 1, according to one embodiment. [Figure 3] Figure 3 is a flowchart of a method for testing and benchmarking commits to a code repository according to one embodiment. [Figure 4] Figure 4 is a flowchart of a method for sequentially applying a patch sequence to the code repository of a system under test (SUT) according to one embodiment. [Figure 5] Figure 5 is a block diagram showing an exemplary computer according to one embodiment, suitable for use in the network computing environment of Figure 1. [Modes for carrying out the invention]

[0012] The drawings and the following description illustrate specific embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structure and method can be adopted without departing from the principles described. Wherever possible, similar or similar reference numerals are used in the drawings to indicate similar or similar functions. If elements share a common numeral followed by another letter, this indicates that the elements are similar or identical. Unless the context indicates otherwise, a reference to a numeral alone generally refers to any one or any combination of such elements.

[0013] Software developers often collaborate with operations and security teams to integrate security early in the application lifecycle, taking responsibility for application security and operations as part of the workflow. This software development philosophy is also known as DevSecOps. Continuous Integration (CI) and / or Continuous Deployment (CD) practices build the foundation for DevSecOps and aim to streamline the software development process by maintaining rapid releases and providing customers with short feedback cycles while automatically ensuring a certain level of quality for newly released software features (reducing bugs and mitigating security issues). These short feedback cycles allow for monitoring the impact of a feature from the moment it is released and notifying developers and product managers of the success or failure of a particular deployment. This enables data-driven decision-making regarding feature integration, deprecation, and / or improvement. This data-driven decision-making (DDDM) process is a valuable part of agile software development, providing a systematic evolution of software products or services.

[0014] However, this leads to the chicken-or-egg problem. Even for a small feature, it is necessary to deploy or release the feature before measuring its impact. DDDM generally requires collecting data first to evaluate the value of the feature. Releasing a feature without data on its impact risks having to remove the feature if it does not function as expected. Furthermore, integrating or partially integrating a proof of concept (which is not necessarily a robust product feature) can increase risks related to product stability and security. To mitigate these issues, developers and product managers may limit the speed at which new features are developed and deployed, but this goes against the Agile philosophy.

[0015] Generally applied software testing best practices, such as the use of staging environments, automated system / integration test strategies, and feature flags that can gate features, may help address some of the stability and potential security issues, but do not provide insights into DDDM for more targeted test strategies such as variant testing.

[0016] An example of variant testing, A / B testing, can assist DDDM by conducting controlled experiments that are not yet fully deployed. New features are evaluated by assessing their interaction with a selected user group that presents different variants for testing. A / B testing is most commonly applied to evaluate changes to the user interface. However, variant testing strategies such as A / B testing are typically applied mainly to monitor the customer experience through the integration of the user interface. These forms of test strategies are not always applicable in highly automated environments.

[0017] The SCM system and / or test and benchmarking methods described herein address the above problems by enabling developers and product managers to execute and evaluate the results of DDDM experiments without deploying functionality to the production environment, and thus do not restrict the speed at which developers can prototype new ideas.

[0018] Example of a system FIG. 1 is a block diagram of one embodiment of a network computing environment 100 in which an SCM system may be used. In the illustrated embodiment, environment 100 includes a developer device 110, a source repository 120, an SCM system 130, and a system under test (SUT) 140. In other embodiments, network computing environment 100 may include different or additional elements. Further, functionality may be distributed among the elements in a manner different from that described.

[0019] Developer device 110 is a computing device configured to modify or commit to source repository 120 (also referred to as the first repository). A commit is a change made to a file or a series of changes made to a series of files. SCM system 130 is a computer system that tracks changes to the code stored in source repository 120 and provides control. For example, in some embodiments, when a commit is made, SCM system 130 may create a unique ID and maintain a record of the specific changes committed along with who made them, when the changes were made, etc. Such a record is also referred to as a commit record (denoted as C). For simplicity, hereinafter, the terms "commit" and "commit record" are used interchangeably. The set 122 of all commits to source repository 120 is also referred to as the "history" (denoted as H), where H = {c1, c2,..., c n}, where c1 is the first commit, c n is the last commit, and n indicates the index number of the last commit. In some embodiments, each commit within history 122 is cx The timestamp t(c) indicates the point in time when it was included. x ) has, where t(c x )≦t(c x+1 The history 122 may be stored on the developer device 110 in relation to the source repository 120, cloud storage, and / or SCM system 130.

[0020] The SCM system 130 includes a test and benchmark module 132 configured to perform tests and benchmarks on commits made to the source repository 120 via the system under test (SUT) 140. The SUT 140 may be a local or remote service accessible by the test and benchmark module 132 via an API 146, a local tool, or provided in the form of a container image (e.g., a Docker image). The SUT 140 has a target repository 142 (also called a second repository). The target repository 142 contains code corresponding to the historical versions of the source repository 120. The SUT 140 also runs one or more services 144. In a CI / CD scenario, a pushed commit can trigger one or more services 144 on the SUT. One or more services 144 may include, but are not limited to, program analysis tools, build automation tools, etc.

[0021] The test and benchmark module 132 applies at least a subset of commits to the source repository 120 onto the target repository 142 and performs benchmark tests on the SUT 140 to determine whether the SUT 140 with the commits functions as intended. In some embodiments, the test and benchmark process includes three phases: (A) a record phase in which a subset of commits applied to the source code repository is extracted; (B) a patch sequence generation phase in which a sequence of patches to be applied to the SUT 140 is prepared; and (C) a monitor phase in which the actual patches are applied and the behavior of the SUT is observed and evaluated.

[0022] During the recording phase, the test and benchmark module 132 extracts commits from the relevant source history 122 with respect to a given test criterion. During the patch sequence generation phase, the test and benchmark module 132 generates a patch sequence. During the monitoring phase, the test and benchmark module 132 plays back the generated sequence on the SUT 140 and captures results and / or metrics by running an evaluation script to capture timing information, API 146 responses, files, etc. from the SUT 140.

[0023] In some embodiments, when applying DDDM with respect to newly implemented features, an automated form of variant testing can be applied using the test and benchmark module 132 by feeding data recorded in the recording phase to the original, unmodified SUT and the modified SUT containing the features to be evaluated. In some embodiments, the test and benchmark module 132 evaluates both systems by replaying the same source history to both and by capturing results and metrics calculated by a configurable evaluation script.

[0024] In some embodiments, the captured evaluation results and metrics can be used in DDDM by (1) helping developers and product managers better understand and quantify the impact of newly developed features before integrating them into the product, and (2) discovering and identifying bugs and / or regressions by playing back events from the project history.

[0025] FIG. 2 is a block diagram of one embodiment of test and benchmark module 132. In the embodiment shown, test and benchmark module 132 includes a recording module 210, a patch sequence generation module 220, and a monitor module 230. In other embodiments, test and benchmark module 132 includes different or additional elements. Further, the functionality can be distributed among the elements in a manner different from that described.

[0026] Recording module 210 extracts at least a subset of the commits relevant to a given test criterion from the history H of source repository 120. The subset of commits is also referred to as a slice S = {c1, c2,..., c m}. The slice S is then persisted in record store 212. Extracting a subset of commits based on a test criterion not only reduces the response time but also allows the analysis to be focused on specific files, which enables the test and benchmark module 132 to scale to large repositories in terms of both the history size and the number of files stored in the repository.

[0027] The test criterion can include a specified time frame T = [T min starting at start time T max and ending at end time T min , T max . In such a case, T is used to extract a subset of the commits that occurred within the time frame T, which is referred to as a time slice. The time slice is S` = {c s |c s ∈H ∧ Tmin ≤ t(c s )≦T max It can be expressed as}. Additionally or alternatively, the test criteria may include an allowlist containing the names of one or more files in which the tests and benchmarks are concentrated. Thus, the slice S can be expressed as S={c s |c s ∈S`∧allowlist(c s Defined as )}. Here, commit c s If it is related to a file specified in the allowlist, allowlist(c s ) evaluates to true. If no allowlist is specified, allowlist(c s ) always evaluates to true. The order of commits is preserved, so t(c s )≦t(c s+1 ) generally remains true.

[0028] In one embodiment, extracting a subset of commits includes extracting commits that modify at least one file specified in the allowlist. A subset of commits can be extracted by, for each commit in history H122, generating a first hash of one or more files in the allowlist before the commit is made, generating a second hash of one or more files after the commit is made, and determining whether the first and second hashes are different. If the first and second hashes are different, the commit modifies at least one file in the allowlist and the commit is extracted.

[0029] The patch sequence generation module 220 generates a patch sequence 222 based on a slice S recorded in the record store 212. In one embodiment, a patch includes code changes made in at least one commit which will be automatically applied to the target repository 142 on the SUT 140. A patch may include a number of samples N of consecutive commits, where N is called the patch sampling number. Thus, N consecutive commits can be combined into a single patch. In such a case, the patch sequence may be represented as a fraction from the slice S in the following equation:

[0030]

number

[0031] For example, if the number of patch samples is N=2 and the slice S={c1, c2, c3, c4, c5, c6}, the patch sequence generation module 220 will obtain the patch sequence PS={{c1, c2}, {c3, c4}, {c5, c6}}, where each of the internal sets represents a patch (for example, patch1={c1, c2} contains two commits that are automatically applied to the target code repository 142 on SUT140). In addition to the time frame T and allowlist, the number of patch samples is another tool that enhances the scalability of the test and benchmark modules by enabling the testing of projects with large histories.

[0032] In the embodiment shown in Figure 2, the monitor module 230 includes a replay module 232 and an evaluation module 234. During the monitoring phase, the replay module 232 replays the patch sequence 222 on the target code repository 142 on the SUT 140. In some embodiments, the replay module 232 makes no assumptions about the SUT 140 and only requires that the SUT 140 is accessible by an API 146 that allows the monitor module 230 or any external system to (1) wait for the execution of a service / action / job that may be triggered when the patch is applied, and (2) extract the relevant test / evaluation data. In some embodiments, the API 146 communication and data extraction of the SUT 140 may be provided in the form of a script as part of the configuration of the monitor module 230.

[0033] In one embodiment, the monitor module 230 natively supports the presence of a target code repository 142 within the SUT 140, which may generate additional jobs such as CI / CD jobs. These additional jobs may be considered to belong to the SUT 140 itself. In the presence of the target code repository, before initiating a patch sequence replay, the replay module 232 may initialize a new target code repository with a history based on the history or partial history of the source repository on the SUT 240.

[0034] In some embodiments, the replay module 232It is also possible to overwrite specific files on the SUT240. This can be advantageous when dealing with situations where a specific code file stored on the SUT240 might affect replays. In a CI / CD environment, the CI / CD configuration itself is typically stored in the SCM system 130. However, it is not always necessary to run the entire set of CI / CD jobs. If benchmark tests are performed to evaluate specific jobs that are part of a larger CI / CD deployment, it may suffice to simply run the jobs individually, which can be controlled using a custom CI / CD that can override the standard configuration.

[0035] In one embodiment, the replay module 232 sequentially and iteratively applies each patch in the patch sequence on the target code repository 142 on the SUT 140 via API 146. Where a target repository exists, patches may be submitted directly to the target repository. After the patch is applied, the monitor module... 230 This monitors the effect of applying the patch. This step (1) ensures that the SUT (also called a test) operates as expected, and (2) extracts and / or collects results and / or metrics from SUT140 while applying the patch by calling an evaluation script (also called a benchmark). The evaluation script may be user-defined or may be provided by the test and benchmark module 132. For example, the evaluation script may collect data from API 146 exposed by SUT140 and / or parse structured data files (such as JSON or XML files) generated by SUT140.

[0036] Exemplary Method Figure 3 is a flowchart of Method 300 for testing and benchmarking commits to a code repository according to one embodiment. The steps in Figure 3 are described from the perspective of SCM130 performing Method 300. However, some or all of the steps may be performed by other entities or components. In addition, in some embodiments, the steps may be performed in parallel, in a different order, or different steps may be performed.

[0037] The SCM system relies on a version-controlled source code repository 120, where test data is stored in the repository's history 122. Because the SCM system leverages the source code repository 120 as input data to be tested and benchmarked, it can replay the history of a source code project (but not limited to this), which is particularly useful when applying DDDM to test and evaluate newly developed features in a CI / CD environment.

[0038] In particular, the SCM system extracts commits from the history 122 of the first code repository 120. The history 122 records a sequence of commits to the first code repository 120. Each commit contains at least one change made to at least one code file in the first code repository 120. Extracting commits 310 may include extracting a subset of commits from the history 122 based on specified criteria. In one embodiment, for each commit, the history 122 records the time the commit was made. The specified criteria include a time frame having a start time and an end time. Extracting commits includes extracting commits made within the time frame. Additionally or alternatively, the specified criteria include a allowlist specifying one or more files that are permitted to be modified. In such a case, extracting commits 310 includes extracting commits that modify at least one file specified in the allowlist.

[0039] Regardless of the exact method used to extract the commits, the SCM system combines the extracted commits into a sequence of patches. Each patch contains changes made to one or more code files that occurred during a sequence of commits. In one embodiment, combining at least a subset of commits into a sequence of batches involves combining N consecutive commits from the extracted commits into a single patch, where N is a positive integer.

[0040] The SCM system establishes a connection with the SUT140, which has a second code repository 142. In one embodiment, the connection with the SUT140 is established via API 146. 130 A second code repository 142 may be created based on the historical version of the first code repository 120 prior to the extracted commit. Alternatively, the SCM system 130 The system may initialize an existing repository on SUT140 as a second code repository 142 based on the historical version of the first code repository 120 prior to the extracted commit. In some embodiments, the SCM system initializes SUT140 in a container based on a container image.

[0041] The SCM system applies the patch sequence to the second code repository 142. Figure 4 is a flowchart of a method 400 for applying a patch sequence to the second code repository 142 according to one embodiment. In the embodiment shown, the SCM system 130 applies the patches in the sequence to the second code repository 142. Each patch is generated by combining one or more consecutive commits. Thus, a patch includes one or more changes made to one or more code files that occurred between one or more consecutive commits.

[0042] The SCM system 130 monitors the effect of applying a patch to the SUT 140 420. The SCM system 130 determines whether the SUT 140 is operating as expected 430. In one embodiment, the SUT 140 monitors the effect of the patch by running a script that collects data associated with the performance of the SUT after the patch has been applied 420, and determines whether the SUT is operating as expected based on the collected data 430. The data may be collected via an API 146 that triggers the execution of a script in response to the application of a patch.

[0043] If SUT140 does not behave as expected, SCM130 stops applying further patches. In one embodiment, SCM130 notifies the developer that the corresponding commit caused SUT140 to behave as expected. Alternatively or additionally, SCM130 may present the collected data to the developer for review. Conversely, if SUT140 behaves as expected, SCM130 applies the next patch in the sequence to the second code repository 440. This process is repeated until all patches in the sequence are applied, or until a particular commit causes the SUT to behave differently than expected at that particular commit.

[0044] Computing system architecture Figure 5 is a block diagram of an exemplary computer 500 suitable for use as a developer device 110, an SCM system 130, or a SUT 140, or for hosting a source code repository 120. The exemplary computer 500 includes at least one processor 502 coupled to a chipset 504. The chipset 504 includes a memory controller hub 520 and an input / output (I / O) controller hub 522. Memory 506 and a graphics adapter 512 are coupled to the memory controller hub 520, and a display 518 is coupled to the graphics adapter 512. A storage device 508, a keyboard 510, a pointing device 514, and a network adapter 516 are coupled to the I / O controller hub 522. Other embodiments of computer 500 have different architectures.

[0045] In the embodiment shown in Figure 5, the storage device 508 is a non-temporary computer-readable storage medium such as a hard drive, compact disc read-only memory (CD-ROM), DVD, or solid-state memory device. Memory 506 holds instructions and data used by the processor 502. The pointing device 514 is a mouse, trackball, touchscreen, or other type of pointing device used in conjunction with the keyboard 510 (which may be an on-screen keyboard) to input data into the computer system 500. The graphics adapter 512 displays images and other information on the display 518. The network adapter 516 connects the computer system 500 to one or more computer networks, such as network 170.

[0046] The type of computer used by the entities in Figures 1 and 2 may vary depending on the embodiment and the processing power required by the entities. For example, the system hosting the source code repository 120 may include multiple blade servers working together to provide the described functionality, while the developer device 110 could be a desktop workstation or a tablet. Furthermore, the computer 500 may lack some of the aforementioned components, such as the keyboard 510, the graphics adapter 512, and the display 518.

[0047] Additional considerations In some parts of the above description, embodiments are described in terms of algorithmic processes or operations. These descriptions and representations of algorithms are commonly used by those skilled in computing technology to effectively communicate the nature of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, they are understood to be implemented by computer programs, including instructions for execution by a processor or equivalent electrical circuit, microcode, etc. Furthermore, it has proven that it is sometimes convenient to refer to these arrangements of functional operations as modules without losing universality.

[0048] Where used herein, a reference to “one embodiment” or “one embodiment” means that a particular element, feature, structure, or characteristic described in relation to that embodiment is included in at least one embodiment. The phrase “in one embodiment” appearing in various parts of this specification does not necessarily all refer to the same embodiment. Similarly, the use of “a” or “an” before an element or component is done solely for convenience. This description should be understood to mean that one or more of the elements or components exist, unless it becomes clear that there is another meaning.

[0049] When a value is described as "approximately" or "effectively" (or a derivative thereof), such a value should be interpreted as an exact + / - 10% unless the context makes a different meaning clear. From the example, "approximately 10" should be understood to mean "within the range of 9 to 11."

[0050] As used herein, the terms “to include,” “to have,” “to have,” “to include,” “to have,” “to have,” or other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements alone, and may include other elements not expressly described in or inherent to such process, method, article, or apparatus. Furthermore, unless expressly stated otherwise, “or” means an inclusive “or” and not an exclusive “or.” For example, condition A or B is satisfied by any one of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0051] A person skilled in the art will understand, upon reading this disclosure, further additional alternative structural and functional designs for systems and processes for tracking vulnerabilities using scope and offset values ​​to reduce duplicate instances. Therefore, while specific embodiments and applications are illustrated and described, it should be understood that the subject matter described is not limited to the exact structures and components disclosed. The scope of protection should be limited only by the claims of any of the issues concerned.

Claims

1. A computer-based method for testing and benchmarking commits to source code, A step of extracting commits from the history of a first code repository, wherein the history records a sequence of commits made on the first code repository, and each commit in the sequence of commits includes at least one change made on at least one code file in the first code repository. The step of combining the extracted commits into a patch sequence, wherein each patch includes changes to one or more code files that occurred between consecutive commits, A step of establishing a connection with a system under test (SUT) having a second code repository, wherein the second code repository corresponds to the historical version of the first code repository before the extracted commit was made, A step of sequentially applying the sequence of patches to the second code repository, wherein the step of sequentially applying the sequence of patches is: The steps include applying the patch in the sequence to the second code repository, The steps include: monitoring the performance of the SUT after applying the patch to the SUT; The applying step includes, in response to determining that the SUT operates as expected after applying the aforementioned patch, applying the next patch in the sequence to the second code repository; A computer implementation method, including

2. The step of monitoring the performance of the SUT after the step of applying the patch in the sequence is: The steps include: running a script configured to collect data associated with the performance of the SUT after the patch has been applied; The computer implementation method according to claim 1, comprising the step of determining whether the SUT operates as expected based on the collected data.

3. The computer implementation method according to claim 2, wherein the data is collected via an API that triggers the execution of the script in response to the application of the patch.

4. The computer implementation method according to claim 1, wherein the step of extracting commits includes the step of extracting a subset of commits from the history based on specified criteria.

5. For each commit, The history records the time when the commit was made. The aforementioned specified criteria include a time frame having a start time and an end time, The computer implementation method according to claim 4, wherein the step of extracting a subset of the commits includes the step of extracting commits made within the time frame.

6. The computer implementation method according to claim 4, wherein the specified criteria include a permission list that specifies one or more files that are permitted to be modified, and the step of extracting a subset of commits includes the step of extracting commits that modify at least one file specified in the permission list.

7. The step of extracting commits that modify at least one file specified in the aforementioned allowlist is: For each commit in the aforementioned history, Before the aforementioned commit is performed, the steps include generating a first hash based on one or more files in the allowlist, After the aforementioned commit is performed, the step of generating a second hash based on the one or more files, A step of determining whether the first hash and the second hash are different, In response to determining that the first hash and the second hash are different, the commit modifies at least one file in the allowlist; A computer implementation method according to claim 6, comprising the step of extracting a commit in response to determining that the commit modifies at least one file in the allow list.

8. The computer implementation method according to claim 1, wherein the step of combining the extracted commits into a patch sequence includes combining N consecutive commits of the extracted commits into a single patch.

9. The computer implementation method according to claim 1, further comprising the step of initializing the repository on the SUT as the second code repository based on at least the historical version of the first code repository prior to the extraction of the commits.

10. The computer implementation method according to claim 1, further comprising the step of initializing the SUT based on a container image.

11. When executed by the processor, the processor: Extracting commits from the history of a first code repository, wherein the history records a sequence of commits made on the first code repository, and each commit in the sequence of commits includes at least one change made on at least one code file in the first code repository. The process of combining the extracted commits into a patch sequence, wherein each patch includes changes to one or more code files that occurred between consecutive commits. Establishing a connection with a system under test (SUT) having a second code repository, wherein the second code repository corresponds to the historical version of the first code repository before the extracted commit was made, Applying the sequence of patches sequentially to the second code repository, wherein applying the sequence of patches sequentially means Applying the patch in the sequence to the second code repository, The performance of the SUT after applying the aforementioned patch to the SUT is monitored, A non-temporary, computer-readable medium for storing encoded instructions that causes an operation to be performed, which includes applying the next patch in the sequence to the second code repository in response to the SUT determining that it operates as expected after applying the aforementioned patch.

12. Monitoring the performance of the SUT after applying the patch in the sequence is After the patch is applied, run a script configured to collect data associated with the performance of the SUT, A non-temporary computer-readable medium according to claim 11, comprising determining whether the SUT operates as expected based on the collected data.

13. The data is collected via an API that triggers the execution of the script in response to the application of the patch, in a non-transient computer-readable medium according to claim 12.

14. The non-temporary computer-readable medium according to claim 11, wherein extracting commits includes extracting a subset of commits from the history based on specified criteria.

15. For each commit, The history records the time when the commit was made. The aforementioned specified criteria include a time frame having a start time and an end time, Extracting a subset of the commits includes extracting commits made within the time frame, according to claim 14, for a non-temporary computer-readable medium.

16. The non-temporary computer-readable medium according to claim 14, wherein the specified criteria include a permission list that specifies one or more files that are permitted to be modified, and extracting a subset of commits includes extracting commits that modify at least one file specified in the permission list.

17. Extracting commits that modify at least one file specified in the aforementioned allowlist is: For each commit in the aforementioned history, Before the aforementioned commit is performed, a first hash is generated based on one or more files in the allowlist, After the aforementioned commit is performed, a second hash is generated based on the one or more files, To determine whether the first hash and the second hash are different, In response to determining that the first hash and the second hash are different, it is determined that the commit modifies at least one file in the allowlist, A non-temporary computer-readable medium according to claim 16, comprising extracting the commit in response to determining that the commit modifies at least one file in the allow list.

18. The non-temporary computer-readable medium according to claim 11, wherein combining the extracted commits into a patch sequence includes combining N consecutive commits of the extracted commits into a single patch.

19. When executed by the processor, the processor will A non-temporary computer-readable medium according to claim 11, which encodes additional instructions causing the repository on the SUT to be initialized as the second code repository based on at least the historical version of the first code repository prior to the extracted commit being made.

20. When executed by the processor, the processor will A non-temporary computer-readable medium according to claim 11, wherein additional instructions are encoded to cause the SUT to be initialized based on a container image.

Images