Regression detection test method

Abstract

Methods, systems, and non-transitory processor-readable storage media for a regression detection test system are provided herein. An example method includes receiving to merge software code changes and parsing a pull request code difference file to identify a plurality of software modules and software functions. A test case attribute associated with the test case is queried to identify a software module associated with the test case. If the plurality of software modules and the software function include a software module, a test case is added to a first plurality of test cases. If the plurality of software modules and software functions include at least one software function and the test case includes the software function, the test case is added to a second plurality of test cases. At least one of the first and second plurality of test cases is executed on the test system.

Description

Technical Field

[0001] The field generally relates to optimized selection of regression test cases, and more particularly to optimized selection of regression test cases in information processing systems. Background Art

[0002] During the life cycle of complex software system development, many tests need to be performed to ensure high quality and effective functionality. New features are gradually added to previous software versions. Regression testing is performed to ensure that new code changes have not introduced new problems. For complex information processing systems, such as storage products, there may be thousands of system regression test cases. Summary of the invention

[0003] The illustrative embodiments provide techniques for implementing a regression detection test system in a storage system. For example, the illustrative embodiments receive a pull request to merge a software code change with a software project library on an information system through a code management tool associated with a software testing lifecycle system. The regression detection test system parses the pull request code difference file to identify the software code change, wherein the software code change includes multiple software modules and software functions. For each test case in a plurality of test cases, the regression detection test system queries the test case attribute associated with the test case to identify at least one software module associated with the test case. If the plurality of software modules and software functions include at least one software module, the regression detection test system adds the test case to the first plurality of test cases. If the plurality of software modules and software functions include at least one software function, and the test case includes the software function, the regression detection test system adds the test case to the second plurality of test cases. The software testing lifecycle system executes at least one of the first plurality of test cases and the second plurality of test cases on the test system to perform a target regression test on the software code change. Other types of processing devices may be used in other embodiments. These and other illustrative embodiments include, but are not limited to, apparatus, systems, methods, and processor-readable storage media. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An information processing system that includes a regression detection testing system in an illustrative embodiment is shown.

[0005] Figure 2 A flow chart of a process for a regression detection testing system in an illustrative embodiment is shown.

[0006] Figure 3 The workflow of the regression detection testing system in an illustrative embodiment is shown.

[0007] Figure 4 and Figure 5An example of a processing platform that may be used to implement at least a portion of an embodiment of a regression detection testing system is shown. DETAILED DESCRIPTION

[0008] Illustrative embodiments will be described herein with reference to an exemplary computer network and associated computers, servers, network devices, or other types of processing devices. However, it should be understood that these and other embodiments are not limited to use with the specific illustrative network and device configurations shown. Therefore, the term "computer network" as used herein is intended to be interpreted broadly to include, for example, any system including multiple networked processing devices.

[0009] A technique for implementing a regression detection test system is described below, which can be used to provide regression test case selection optimization, among other things, by receiving a pull request to merge a software code change with a software project library on an information system by a code management tool associated with a software testing lifecycle system. The regression detection test system parses the pull request code difference file to identify the software code change, wherein the software code change includes a plurality of software modules and software functions. For each test case in a plurality of test cases, the regression detection test system queries a test case attribute associated with the test case to identify at least one software module associated with the test case. If the plurality of software modules and software functions include at least one software module, the regression detection test system adds the test case to a first plurality of test cases. If the plurality of software modules and software functions include at least one software function, and the test case includes the software function, the regression detection test system adds the test case to a second plurality of test cases. The software testing lifecycle system executes at least one of the first plurality of test cases and the second plurality of test cases on the test system to perform a target regression test on the software code change.

[0010] It is often not feasible to execute all available test cases in continuous integration (CI) testing when merging pull requests. Typically, a group of "must pass" test cases is selected as the "must pass" test suite for each pull request to be executed, and selecting test cases for the "must pass" group is a challenge. If the total execution time of the "must pass" group is too long, precious and limited resources will be occupied, and pull requests must wait for these tests to be completed before they can be merged. In addition, although the "must pass" test case group may have hundreds of test cases, these test cases may not target testing the software code changes in the pull request. Regression test cases must be executed to ensure that the newly implemented features do not affect existing functionality. As mentioned above, there may be thousands of regression test cases. However, the resources to execute these regression test cases may be limited. Therefore, it is critical to select the most valuable set of regression test examples from the regression test case set to execute.

[0011] Conventional technology does not select a test case set to perform regression testing that covers modules and functions with software code changes in a pull request to obtain the most targeted regression verification results. Conventional technology does not provide selection of optimal regression test cases to appropriately provide regression test coverage of software code changes in pull requests, especially when resources are limited. Conventional technology does not map test cases and / or test case suites to covered product code modules and functions by analyzing the coverage differences between the coverage results of the test execution cycle and the basic code coverage results. Conventional technology does not provide a mapping relationship to automatically update the mapping. Conventional technology does not provide a method for selecting test cases based on product code changes in pull requests. Conventional technology does not provide a "lightweight" test case set that targets software code changes in pull requests. Conventional technology does not provide a selectable test case set for regression testing software code changes in pull requests.

[0012] In contrast, in at least some implementations of the current technology described herein, regression test case selection is optimized by receiving a pull request that merges software code changes with a software project library on an information system by a code management tool associated with a software testing lifecycle system. The regression detection test system parses the pull request code difference file to identify software code changes, wherein the software code changes include multiple software modules and software functions. For each test case in a plurality of test cases, the regression detection test system queries the test case attributes associated with each test case to identify at least one software module associated with each test case. If the plurality of software modules and software functions include at least one software module, the regression detection test system adds the test case to the first plurality of test cases. If the plurality of software modules and software functions include at least one software function, and the test case includes the software function, the regression detection test system adds the test case to the second plurality of test cases. The software testing lifecycle system executes at least one of the first plurality of test cases and the second plurality of test cases on the test system to perform a target regression test on the software code change.

[0013] Therefore, the goal of the present technology is to provide a method and system for selecting a test case set to perform regression testing, which covers modules and functions with software code changes in pull requests to obtain the most targeted regression verification results. Another goal is to provide the best regression test case selection to fully provide regression test coverage of software code changes in pull requests, especially in the case of limited resources. Another goal is to map test cases and / or test case suites to covered product code modules and functions by analyzing the coverage differences between the coverage results of the test execution cycle and the basic code coverage results. Another goal is to provide a mapping relationship and automatically update the mapping. Another goal is to provide a method for selecting test cases based on product code changes in pull requests. Another goal is to provide a "lightweight" test case set that targets software code changes in pull requests. Another goal is to provide a selectable test case set for regression testing software code changes in pull requests.

[0014] In at least some implementations of the current technology described herein, use of a regression detection testing system can provide one or more of the following advantages: selecting a set of test cases to perform regression testing that covers modules and functions with software code changes in a pull request to obtain the most targeted regression verification results, providing selection of optimal regression test cases to appropriately provide regression test coverage of software code changes in a pull request, especially when resources are limited, by analyzing the coverage differences between coverage results of a test execution cycle and basic code coverage results, mapping test cases and / or test case suites to covered product code modules and functions, providing mapping relationships to automatically update mappings, providing a way to select test cases based on product code changes in a pull request, providing a "lightweight" test case set that targets software code changes in a pull request, and providing a selectable test case set for regression testing software code changes in a pull request.

[0015] In contrast to conventional techniques, in at least some implementations of the current techniques described herein, regression test case selection is optimized by receiving a pull request to merge a software code change with a software project library on an information system by a code management tool associated with a software testing lifecycle system. The regression detection test system parses the pull request code difference file to identify software code changes, wherein the software code changes include multiple software modules and software functions. For each test case in a plurality of test cases, the regression detection test system queries the test case attributes associated with each test case to identify at least one software module associated with each test case. If the plurality of software modules and software functions include at least one software module, the regression detection test system adds the test case to the first plurality of test cases. If the plurality of software modules and software functions include at least one software function, and the test case includes the software function, the regression detection test system adds the test case to the second plurality of test cases. The software testing lifecycle system executes at least one of the first plurality of test cases and the second plurality of test cases on the test system to perform a target regression test on the software code change.

[0016] In an example embodiment of the present technology, a software testing lifecycle system completes a pull request to merge software code changes with a software project repository.

[0017] In an example embodiment of the present technology, a data set identifies software code changes, wherein each element in the data set includes a subset of a plurality of software functions, and wherein each subset is associated with a respective software module of a plurality of software modules.

[0018] In an example embodiment of the present technology, a code management tool receives a request to perform targeted regression testing on a software code change.

[0019] In an example embodiment of the present technology, a regression detection testing system creates a pull request code difference file by analyzing software code changes in a pull request.

[0020] In an example embodiment of the present technology, a regression detection testing system identifies each of the software modules associated with each test case.

[0021] In an example embodiment of the current technology, a regression detection testing system executes a code coverage analysis tool on multiple test case groups to identify whether each test case provides coverage for at least one of at least one software module and at least one software function, wherein the multiple test case groups include each test case, and updates test case properties when the test case provides coverage.

[0022] In an example embodiment of the present technology, a regression detection test system executes a code coverage analysis tool to determine test coverage differences between baseline code coverage results and test execution cycle coverage results.

[0023] In an example embodiment of the present technology, a regression detection testing system executes a code coverage analysis tool on a plurality of test case groups associated with a product code integration branch associated with a code management tool.

[0024] In an exemplary embodiment of the present technology, when each test case includes at least one software module, the regression detection testing system updates the test case attributes.

[0025] In an exemplary embodiment of the present technology, when at least one software module includes at least one software function, the regression detection testing system updates a test case attribute.

[0026] In an example embodiment of the present technology, the regression detection testing system updates test case properties when each test case is created.

[0027] In an example embodiment of the present technology, the regression detection testing system updates test case properties when each test case is modified.

[0028] In an example embodiment of the present technology, the regression detection testing system determines that at least one software module associated with each test case includes at least a first software function before adding each test case to the first plurality of test cases.

[0029] In an example embodiment of the present technology, a regression detection testing system selects at least one of a first plurality of test cases and a second plurality of test cases based on a pull request.

[0030] In an example embodiment of the present technology, a regression detection testing system determines that a software code change affects more a product code change in a software module among a plurality of software modules and software functions, and selects a first plurality of test cases to execute on the testing system.

[0031] In an example embodiment of the present technology, the regression detection testing system determines that the software code change affects more the product code change in the software functions in the plurality of software modules and software functions, and selects a second plurality of test cases to execute on the testing system.

[0032] In an example embodiment of the present technology, a regression detection testing system determines not to execute a first plurality of test cases and, in response, selects a second plurality of test cases to execute on the testing system.

[0033] Figure 1A computer network (also referred to herein as an information processing system) 100 is shown configured in accordance with an illustrative embodiment. Computer network 100 includes software testing lifecycle system 101, code management tool 103, regression detection test system 105, and test system 102-N. Software testing lifecycle system 101, code management tool 103, regression detection test system 105, and test system 102-N are coupled to network 104, where network 104 in this embodiment is assumed to represent a sub-network or other related portion of a larger computer network 100. Therefore, both elements 100 and 104 are referred to herein as "networks," but the latter is not necessarily the same in this example. Figure 1 Components of the former are assumed in the context of the embodiments. Also coupled to the network 104 is a regression detection test system 105 which may reside on a storage system. Such a storage system may include any of a variety of different types of storage, including network attached storage (NAS), storage area network (SAN), direct attached storage (DAS), and distributed DAS, as well as combinations of these and other storage types, including software defined storage.

[0034] Each of the test systems 102-N may include, for example, a server and / or portions of one or more server systems, as well as devices such as mobile phones, laptop computers, tablet computers, desktop computers, or other types of computing devices. Such devices are examples of what are more generally referred to herein as "processing devices." Some of these processing devices are also generally referred to herein as "computers."

[0035] In some embodiments, test system 102-N includes respective computers associated with a particular company, organization, or other enterprise. In addition, at least portions of computer network 100 may also be referred to herein as collectively forming an "enterprise network." As will be appreciated by those skilled in the art, many other operational scenarios involving various types and arrangements of processing devices and networks are possible.

[0036] Furthermore, it should be understood that in this context and elsewhere herein, the term "user" is intended to be interpreted broadly to include, for example, a person, a hardware, software, or firmware entity, and various combinations of these entities.

[0037] Network 104 is assumed to comprise a portion of a global computer network such as the Internet, although other types of networks may be part of computer network 100, including a wide area network (WAN), a local area network (LAN), a satellite network, a telephone or wired network, a cellular network, a wireless network such as a Wi-Fi or WiMAX network, or various portions or combinations of these and other types of networks. Thus, in some embodiments, computer network 100 comprises a combination of multiple different types of networks, each of which includes processing devices configured to communicate using the Internet Protocol (IP) or other related communication protocols.

[0038] Also associated with the regression detection test system 105 are one or more input-output devices, which illustratively include a keyboard, a display, or any combination of other types of input-output devices. For example, such input-output devices can be used to support one or more user interfaces of the regression detection test system 105, and to support communication between the regression detection test system 105 and other related systems and devices not explicitly shown. For example, a dashboard can be provided for users to view the execution progress of the regression detection test system 105. One or more input-output devices can also be associated with any test system 102-N.

[0039] In addition, assuming Figure 1 The regression detection test system 105 in the embodiment is implemented using at least one processing device. Each such processing device generally includes at least one processor and associated memory, and implements one or more functional modules for controlling certain features of the regression detection test system 105.

[0040] More specifically, the regression detection test system 105 in this embodiment may include a processor coupled to a memory and a network interface.

[0041] A processor illustratively includes a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other type of processing circuit, as well as portions or combinations of such circuit elements.

[0042] The memory illustratively includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory and other memories disclosed herein may be viewed as examples of what is more generally referred to as a "processor-readable storage medium" that stores executable computer program code or other types of software programs.

[0043] One or more embodiments include an article of manufacture such as a computer-readable storage medium. Examples of articles of manufacture include, but are not limited to, storage devices such as storage disks, storage arrays, or integrated circuits containing memory, as well as various other types of computer program products. As used herein, the term "article of manufacture" should be understood to exclude transient propagating signals. These and other references to "discs" mentioned herein are intended to refer generally to storage devices including solid-state drives (SSDs), and therefore should not be considered to be limited in any way to spin-on magnetic media.

[0044] The network interface allows the regression detection test system 105 to communicate with the software testing lifecycle system 101 , the code management tool 103 , and the test systems 102 -N over the network 104 , and illustratively includes one or more conventional transceivers.

[0045] The regression detection test system 105 may be implemented at least in part in the form of software stored in a memory and executed by a processor, and may reside in any processing device. The regression detection test system 105 may be a stand-alone plug-in that may be included within a processing device.

[0046] It should be understood that Figure 1 The specific set of elements of the software testing lifecycle system 101, code management tool 103, and test system 102-N for the regression detection test system 105 shown in the figure are presented only as illustrative examples, and additional or alternative elements may be used in other embodiments. Therefore, another embodiment includes additional or alternative systems, devices, and other network entities, as well as different arrangements of modules and other components. For example, in at least one embodiment, one or more of the regression detection test systems 105 can be located on the same processing platform and / or on parts of the same processing platform.

[0047] References such as Figure 2 The flowchart of FIG. 1 describes an exemplary process of the regression detection testing system 105 in the computer network 100 in more detail.

[0048] Figure 2 is a flow chart of a process for executing the regression detection testing system 105 in an illustrative embodiment. It should be understood that this particular process is merely an example, and additional or alternative processes may be performed in other embodiments.

[0049] At 200, a code management tool 103 associated with a software testing lifecycle system 101 receives a pull request to merge software code changes with a software project library on an information system. For example, a bug fix pull request may have smaller software code changes than issuing a bubble pull request or a feature merge pull request. Therefore, targeted regression testing is required, in which test cases and / or test case suites are mapped to the product code modules and functions they cover respectively, and then test cases are selected to perform targeted regression testing to test the code changes in the pull request. More importantly, a selectable level of test cases is preferred. In an example embodiment, a regression detection test system 105 receives a request to perform a targeted regression test on a software code change.

[0050] At 202, the regression detection testing system 105 parses the pull request code difference file to identify software code changes, wherein the software code changes include a plurality of software modules and software functions represented as "A". In an example embodiment, the regression detection testing system 105 obtains the software code changes from a target product code merge pull request (PR). In an example embodiment, the regression detection testing system 105 creates a pull request code difference file by analyzing the software code changes in the pull request. In an example embodiment, the pull request difference file can be created, for example, using any kind of "diff" command in which two files are compared to each other and the output of the command is the difference between the two files.

[0051] In an example embodiment, the result of the parsing is a data set identifying software code changes, wherein each element in the data set includes a subset of multiple software functions, and wherein each subset is associated with a corresponding software module in multiple software modules. In an example embodiment, when the regression detection test system 105 obtains software code changes from the target product code merge pull request (PR), the software code changes are stored in the data set. For example, "A" can represent a data set identifying software code changes in the pull request "PR", where A = [M1 pr ,M2 pr ,…MK pr ], and among them Mk pr is a list of functions in the pull request that have software code changes in module k, and where k = [1, 2, ..., K]. In an example embodiment, if there are no software code changes for module k in the pull request, then Mk pr =[]. Listed below are sample usages of the "diff" command:

[0052]

[0053]

[0054] In the example embodiment, based on the output of the "diff" command, an element Mx is constructed for module x. pr :

[0055]

[0056] At 204, for each test case d in the plurality of test cases, n , the regression detection test system queries the test case d n Associated test case attributes Cd n , to identify the test case d n At least one software module associated with it. n Represents a mapping relationship, which records the test case d n The software modules and software functions of the target product covered (i.e. tested). Mapping relationship Cd n =[M1d n ,M2d n ,…,MKd n ], where K is the number of modules of the target product code that the test case focuses on, and where Mkd n Recording test cases n Whether to cover the product code module k. When test case d n When overriding function i in module k, add function i to Mkd n In an example embodiment, the regression detection test system 105 updates the test case attributes in the test case management tool as each test case is created. In another example embodiment, the regression detection test system 105 updates the test case attributes in the test case management tool as each test case is modified.

[0057] In the exemplary embodiment, the regression detection testing system 105 identifies the n In an exemplary embodiment, the regression detection testing system 105 executes a code coverage analysis tool on the plurality of test case groups to identify each test case in the plurality of test case groups. n Whether to provide coverage of software modules and / or software functions. Then, when the test case d n When providing coverage, the regression detection test system 105 updates the test case attribute Cd nIn an example embodiment, the regression detection test system 105 executes a code coverage analysis tool to determine the test coverage difference between the baseline code coverage result and the test execution cycle coverage result. In an example embodiment, the regression detection test system 105 executes the code coverage analysis tool on a plurality of test case groups associated with a product code integration branch associated with a code management tool. Therefore, the regression detection test system 105 utilizes the code coverage analysis tool to compare the test case d n The code coverage of the execution cycle is compared with the base code coverage results (i.e., comparing the baseline to get the difference) to analyze the incremental coverage. For example, the code coverage analysis tool will generate the current baseline code coverage results for the product code on a git branch. The test cases used in the baseline can be predefined tests, such as by project. The coverage results of the test execution cycle are the result of inputting the set of test cases defined in a group file into the code coverage analysis tool.

[0058] In an example embodiment, the regression detection test system 105 parses the incremental coverage results to obtain the test case d n Or for example, the target product code modules and functions covered by the test case set. Each element Mkd n It is obtained by comparing the code coverage results. The following lists each element Mkd n Example of construction:

[0059]

[0060] In an example embodiment, when a test case in a plurality of test cases d n When at least one software module is included, the regression detection test system 105 updates the test case attribute. In an exemplary embodiment, when at least one software module includes at least one software function, the regression detection test system 105 updates the test case attribute.

[0061] Figure 3 The workflow of the regression detection test system as an illustrative embodiment is shown. D represents multiple test cases, D = [d1, d2, ..., d N ], where d n The test case is test case n, or in some embodiments, test case suite n, and N is the number of test cases in the plurality of test cases. PR represents a pull request in the code management tool 103, and PR diff is a pull request code difference file. As described above, the test case attributes (also referred to as "mapping relationships") Cd n =[M1d n ,M2d n ,…,MKd n], where K is the number of modules of the target product code on which the test cases focus, and where Mkd n Recording test cases n Whether it covers the product code module k. The first plurality of test cases are denoted as "level 1 buckets" and the second plurality of test cases are denoted as "level 2 buckets". In the example embodiment, in the query Cd n Previously, the regression detection test system 105 initialized n = 1 and k = 1. The regression detection test system 105 may also empty the first plurality of test cases (ie, "level 1 buckets") and the second plurality of test cases (ie, "level 2 buckets").

[0062] At 206, if the plurality of software modules and software functions MK pr includes at least one software module k, then the regression detection test system 105 will test case d n In the example embodiment, before adding the test case d n Before adding to the first plurality of test cases, the regression detection testing system 105 determines the test case d n Whether the associated software module k includes at least the first software function. In other words, Figure 3 As shown, if module k exists in Mk pr and module k has a record in Mkd n If at least one function in , the regression detection test system 105 will d n Added to the first multiple test cases.

[0063] At 208, if multiple software modules and software functions Mk pr Includes at least one software function f and test case d n If the software function f is included, the regression detection test system 105 adds the test case to the second plurality of test cases. In other words, if Mk pr Also included in Mkd n If the function f is found in , the regression detection test system 105 will test case d n Added to the second more test case.

[0064] In the exemplary embodiment, the regression detection test system 105 performs the Mk test for K and N. pr and Mkd n This iterative comparison is performed between modules and functions in Figure 3 As shown), where Cd n =[M1d n ,M2d n ,…,MKd n], and K is the number of modules of the target product code that the test case focuses on, where D represents multiple test cases, D = [d1, d2, …, d N ], where d n The test case is test case n, or in some embodiments, test case suite n, and N is the number of test cases in the plurality of test cases. Once the iterative comparison is completed, the regression detection testing system 105 outputs the first plurality of test cases and the second plurality of test cases.

[0065] At 210, the software testing lifecycle system 101 executes at least one of the first plurality of test cases and the second plurality of test cases on the test system to perform a target regression test on the software code change. In an example embodiment, the regression detection test system 105 selects at least one of the first plurality of test cases and the second plurality of test cases based on the pull request. In an example embodiment, the second plurality of test cases provide more specific testing because the second plurality of test cases provide test coverage for the modules affected by the pull request change and the functions affected by the pull request change. On the other hand, the use of the first plurality of test cases to perform regression testing is more aggressive than the second plurality of test cases. For example, the regression detection test system 105 may determine that the software code change affects more product code changes in the software modules in the plurality of software modules and software functions, and in this example scenario, the regression detection test system 105 selects the first plurality of test cases to execute on the test system. In another example embodiment, the regression detection test system 105 determines that the software code change affects more product code changes in the software functions in the plurality of software modules and software functions, and in this example scenario, the regression detection test system 105 selects the second plurality of test cases to execute on the test system. In yet another example embodiment, the regression detection test system 105 may determine not to execute the first plurality of test cases, but to select the second plurality of test cases to execute on the test system. Thus, the first plurality of test cases and the second plurality of test cases are selectable, and the regression detection test system 105 may select any plurality of test cases to run regression testing.

[0066] In an example embodiment, the software testing lifecycle 101 completes a pull request to merge software code changes with a software project repository.

[0067] In an example embodiment, there are 1000 target product code modules in one of the main product areas; K = 1000. In another example embodiment, there are 1,000 test cases or test case suites for performing tests on the 1000 target product code modules; N = 1000, D = [1, 2, ..., 1000], d n =n. Each test case d nEach test case management tool has coverage pattern information for target product modules and functions, which is stored in the database of the test case management tool. This information can be updated by comparing the current execution of test cases and test case suites with the coverage results of the baseline test. This leads to the Cd n Construction and Cd n Each element of is denoted as Mkd n . In an example embodiment, there is a "must pass" group of 200 test cases for the main areas of the product; must_pass_group = [1,5,11,…,995]. The "must pass" group is a set of test cases used to ensure that the pull request software code changes do not introduce any serious problems to the main target product functionality. Even though the number of regression tests in the "must pass" group is limited, the test cases in the "must pass" group may still take hours to execute. In addition, the "must pass" group may not be related to the software code changes in the pull request, and if regression issues are not discovered before the pull request code is merged, these issues may be ignored. must_pass_group = [1,5,11,…,995] is finally compared with the test cases selected by the regression detection testing system 105.

[0068] The regression detection test system 105 identifies the target product code pull request differences, generates and parses the pull request code difference file, and obtains the modules and functions whose codes have changed in the pull request:

[0069]

[0070] The regression detection test system 105 is then based on Figure 3 The workflow shown selects the first plurality of test cases and the second plurality of test cases. The result is:

[0071] First multiple test cases: [5,102,123,234,322,567,679,680]

[0072] Second most test cases: [102,234,567,680]

[0073] The test cases identified by the regression detection testing system 105 are more relevant to verifying the code changes in the pull request. The number of test cases identified by the regression detection testing system 105 is much smaller than the "must pass" group of test cases, but is still effective in detecting regression issues. The regression detection testing system 105 can select which test level, or the user can have the option to select the test level. Executing the first plurality of test cases provides more aggressive testing than the second plurality of test cases.

[0074] Therefore, combined with Figure 2The specific processing operations and other functions described in the flowcharts are presented only as illustrative examples and should not be interpreted as limiting the scope of the present disclosure in any way. For example, in other embodiments, the order of the process steps can be changed, or some steps can be performed simultaneously with each other rather than sequentially.

[0075] The above illustrative embodiments provide significant advantages over traditional methods. For example, some embodiments are configured to significantly improve the identification of the best subset of regression test cases for software code changes in pull requests. Compared with traditional methods, these and other embodiments can effectively improve the regression testing of each pull request. For example, the embodiments disclosed herein select a test case set to perform regression testing, which covers modules and functions with software code changes in pull requests to obtain the most targeted regression verification results. The embodiments disclosed herein provide the selection of the best regression test cases to fully provide regression test coverage for software code changes in pull requests, especially when resources are limited. The embodiments disclosed herein map test cases and / or test case suites to covered product code modules and functions by analyzing the coverage differences between the coverage results of the test execution cycle and the basic code coverage results. The embodiments disclosed herein provide a mapping relationship to automatically update the mapping. The embodiments disclosed herein provide a way to select test cases based on product code changes in pull requests. The embodiments disclosed herein provide a "lightweight" test case set targeting software code changes in pull requests. Embodiments disclosed herein provide a selectable set of test cases for regression testing software code changes in pull requests.

[0076] It should be understood that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and need not exist in other embodiments. In addition, the specific types of information processing system features and functions shown in the drawings and described above are merely exemplary, and many other arrangements may be used in other embodiments.

[0077] As previously described, at least a portion of the information handling system 100 may be implemented using one or more processing platforms. A given such processing platform includes at least one processing device, which includes a processor coupled to a memory. In some embodiments, the processor and memory include corresponding processor and memory elements of a virtual machine or container provided using one or more underlying physical machines. The term "processing device" used herein is intended to be broadly interpreted as including a variety of different physical processors, memory and other device components and virtual instances of these components. For example, in some embodiments, a "processing device" may include one or more virtual processors or execute across one or more virtual processors. Therefore, a processing device may be physical or virtual and may execute across one or more physical or virtual processors. It should also be noted that a given virtual device may be mapped to part of a physical device.

[0078] Some illustrative embodiments of a processing platform for implementing at least part of an information handling system include a cloud infrastructure that includes virtual machines implemented using a hypervisor running on a physical infrastructure. The cloud infrastructure also includes a set of applications running on each virtual machine under the control of the hypervisor. Multiple hypervisors may also be used, each hypervisor providing a set of virtual machines using at least one underlying physical machine. Different sets of virtual machines provided by one or more hypervisors may be used to configure multiple instances of various components of the system.

[0079] These and other types of cloud infrastructures can be used to provide what is also referred to herein as a multi-tenant environment. One or more system components or portions thereof are illustratively implemented for use by tenants of such a multi-tenant environment.

[0080] As previously mentioned, the cloud infrastructure disclosed herein may include a cloud-based system. In an illustrative embodiment, a virtual machine provided in such a system may be used to implement at least part of a computer system.

[0081] In some embodiments, the cloud infrastructure additionally or alternatively includes multiple containers implemented using a container host device. For example, as described in detail herein, a given container of the cloud infrastructure illustratively includes a Docker container or other type of Linux container (LXC). The container runs on a virtual machine in a multi-tenant environment, although other arrangements are possible. Containers are used to implement various different types of functions within the information processing system 100. For example, containers can be used to implement corresponding processing devices that provide computing and / or storage services for a cloud-based system. Similarly, containers can be used in conjunction with other virtualization infrastructures, such as virtual machines implemented using a hypervisor.

[0082] Now refer to Figure 4 and Figure 5Illustrative embodiments of the processing platforms are described in further detail. Although described in the context of information handling system 100, in other embodiments, these platforms may also be used to implement at least portions of other information handling systems.

[0083] Figure 4 An example processing platform is shown that includes a cloud infrastructure 400. The cloud infrastructure 400 includes a combination of physical and virtual processing resources for implementing at least a portion of the information processing system 100. The cloud infrastructure 400 includes a plurality of virtual machines (VMs) and / or a set of containers 402-1, 402-2, ... 402-L implemented using a virtualization infrastructure 404. The virtualization infrastructure 404 runs on a physical infrastructure 405 and illustratively includes one or more hypervisors and / or an operating system-level virtualization infrastructure. The operating system-level virtualization infrastructure illustratively includes a kernel control group of a Linux operating system or other type of operating system.

[0084] Cloud infrastructure 700 also includes application sets 410-1, 410-2, ... 410-L that run on a corresponding one of VM / container sets 402-1, 402-2, ... 402-L under the control of virtualization infrastructure 404. VM / container set 402 includes a corresponding VM, a corresponding set of one or more containers, or a corresponding set of one or more containers running in a VM. Figure 4 In some implementations of the embodiments, the VM / container set 402 includes respective VMs implemented using a virtualization infrastructure 404 including at least one hypervisor.

[0085] A hypervisor platform may be used to implement a hypervisor within the virtualized infrastructure 404, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines include one or more distributed processing platforms including one or more storage systems.

[0086] exist Figure 4 In other implementations of the embodiment, the VM / container set 402 includes a corresponding container implemented using a virtualization infrastructure 404, which provides operating system-level virtualization functions, such as support for Docker containers running on a bare metal host or Docker containers running on a VM. The container is illustratively implemented using a corresponding kernel control group of the operating system.

[0087] As is apparent from the above, one or more processing modules or other components of information handling system 100 may each run on a computer, server, storage device, or other processing platform element. A given such element is considered an example of what is more generally referred to herein as a "processing device." Figure 4The cloud infrastructure 400 shown may represent at least a portion of a processing platform. Another example of such a processing platform is Figure 5 A processing platform 500 is shown.

[0088] The processing platform 500 in this embodiment comprises a portion of the information handling system 100 and includes a plurality of processing devices, denoted as 502 - 1 , 502 - 2 , 502 - 3 , . . . 502 -K, which communicate with each other via a network 504 .

[0089] Network 504 includes any type of network including, for example, a global computer network such as the Internet, a WAN, a LAN, a satellite network, a telephone or wired network, a cellular network, a wireless network such as a Wi-Fi or WiMAX network, or various portions or combinations of these and other types of networks.

[0090] The processing device 502 - 1 in the processing platform 500 includes a processor 810 coupled to a memory 512 .

[0091] Processor 510 includes a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other types of processing circuits, as well as portions or combinations of these circuit elements.

[0092] The memory 512 includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory 512 and other memories disclosed herein should be viewed as illustrative examples of what is more generally referred to as a "processor-readable storage medium" that stores executable program code for one or more software programs.

[0093] Articles of manufacture including such processor-readable storage media are considered to be illustrative embodiments. Such articles of manufacture given include, for example, a storage array, a storage disk, or an integrated circuit containing RAM, ROM or other electronic memory, or any of various other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagation signals. Many other types of computer program products including processor-readable storage media may be used.

[0094] Also included in the processing device 502-1 is a network interface circuit 514 for interfacing the processing device with the network 804 and other system components, and may include a conventional transceiver.

[0095] It is assumed that the other processing devices 502 of the processing platform 500 are configured in a manner similar to the processing device 502 - 1 shown in the figure.

[0096] Likewise, the specific processing platform 500 shown in the figure is presented only as an example, and information processing system 100 may include additional or alternative processing platforms, as well as any combination of many different processing platforms, each such platform including one or more computers, servers, storage devices or other processing devices.

[0097] For example, other processing platforms for implementing the illustrative embodiments may include different types of virtualization infrastructures, instead of or in addition to virtualization infrastructures including virtual machines. Such virtualization infrastructures illustratively include container-based virtualization infrastructures configured to provide Docker containers or other types of LXC.

[0098] As another example, in some embodiments, portions of a given processing platform may include a converged infrastructure.

[0099] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used.At least a subset of these elements may be implemented collectively on a common processing platform, or each such element may be implemented on a separate processing platform.

[0100] Additionally, many other arrangements of computers, servers, storage products or devices, or other components are possible in information handling system 100. These components may communicate with other elements of information handling system 100 via any type of network or other communications medium.

[0101] For example, in the illustrative embodiments, specific types of storage products that may be used to implement a given storage system of a distributed processing system include all-flash and hybrid flash storage arrays, scale-out all-flash storage arrays, scale-out NAS clusters, or other types of storage arrays. In the illustrative embodiments, a combination of multiple of these and other storage products may also be used to implement a given storage system.

[0102] It should be emphasized again that the above-described embodiments are presented for illustrative purposes only. Many variations and other alternative embodiments may be used. In addition, the specific configurations of the system and device elements illustratively shown in the accompanying drawings and the associated processing operations may be changed in other embodiments. Thus, for example, the specific types of processing devices, modules, systems, and resources deployed in a given embodiment and their respective configurations may be different. In addition, the various assumptions made above in the process of describing the illustrative embodiments should also be regarded as exemplary, rather than requirements or limitations of the present disclosure. Many other optional embodiments within the scope of the appended claims will be clear to those skilled in the art.

Claims

1. A method comprising: receiving, by a code management tool associated with a software testing lifecycle system, a pull request to merge a software code change with a software project repository on an information system; Parsing the pull request code difference file by the regression detection test system to identify the software code change, wherein the software code change includes a plurality of software modules and software functions; For each test case in the plurality of test cases, the regression detection test system queries a test case attribute associated with each test case to identify at least one software module associated with each test case; adding each of the test cases to a first plurality of test cases if the plurality of software modules and software functions includes the at least one software module; if the plurality of software modules and software functions include at least one software function, and the each test case includes the at least one software function, adding the each test case to a second plurality of test cases; as well as At least one of the first plurality of test cases and the second plurality of test cases is executed on a test system by the software testing lifecycle system to perform targeted regression testing on the software code change, wherein the method is implemented by at least one processing device including a processor coupled to a memory.

2. The method according to claim 1, further comprising: The pull request is completed to merge the software code changes with the software project repository.

3. The method of claim 1 , wherein a data set identifies the software code changes, wherein each element in the data set comprises a subset of the plurality of software functions, and wherein each of the subsets is associated with a corresponding software module in the plurality of software modules.

4. The method of claim 1 , wherein receiving, by the code management tool associated with a software testing lifecycle system, the pull request to merge software code changes comprises: A request is received to perform the targeted regression test on the software code change.

5. The method of claim 1 , wherein parsing, by the regression detection testing system, the pull request code difference file to identify the software code changes comprises: The pull request code diff file is created by analyzing the software code changes in the pull request.

6. The method according to claim 1, wherein querying the test case attribute by the regression detection test system comprises: Each of the software modules associated with each of the test cases is identified.

7. The method according to claim 1, wherein querying the test case attribute by the regression detection test system comprises: executing a code coverage analysis tool on a plurality of test case groups to identify whether each of the test cases provides coverage for at least one of the at least one software module and the at least one software function, wherein the plurality of test case groups include each of the test cases; as well as The test case properties are updated when the test case provides the coverage.

8. The method according to claim 7, wherein executing the code coverage analysis tool on the plurality of test case groups comprises: The code coverage analysis tool is executed to determine the test coverage difference between the baseline code coverage result and the test execution cycle coverage result.

9. The method according to claim 7, wherein executing the code coverage analysis tool on the plurality of test case groups comprises: The code coverage analysis tool is executed on the plurality of test case groups associated with a product code integration branch associated with the code management tool.

10. The method according to claim 1, wherein querying the test case attribute comprises: The test case attributes are updated when each of the test cases includes the at least one software module.

11. The method according to claim 10, further comprising: The test case attribute is updated when the at least one software module includes the at least one software function.

12. The method according to claim 1, wherein querying the test case attribute by the regression detection test system comprises: The test case properties are updated when each test case is created.

13. The method according to claim 1, wherein querying the test case attribute comprises: The test case properties are updated when each test case is modified.

14. The method of claim 1 , wherein if the plurality of software modules and software functions include the at least one software module, adding the each test case to the first plurality of test cases comprises: Prior to adding each test case to the first plurality of test cases, determining that the at least one software module associated with each test case includes at least a first software function.

15. The method of claim 1 , wherein executing the at least one of the first plurality of test cases and the second plurality of test cases comprises: At least one of the first plurality of test cases and the second plurality of test cases is selected based on the pull request.

16. The method of claim 15, wherein selecting the at least one of the first plurality of test cases and the second plurality of test cases based on the pull request comprises: determining that the software code change affects further product code changes in software modules in the plurality of software modules and the software functions; as well as The first plurality of test cases are selected for execution on the test system.

17. The method of claim 15, wherein selecting the at least one of the first plurality of test cases and the second plurality of test cases based on the pull request comprises: determining that the software code change affects further product code changes of software functions among the plurality of software modules and the software functions; as well as The second plurality of test cases are selected for execution on the test system.

18. The method of claim 15, wherein selecting the at least one of the first plurality of test cases and the second plurality of test cases based on the pull request comprises: Determining not to execute the first plurality of test cases; as well as In response, the second plurality of test cases is selected for execution on the test system.

19. A system comprising: at least one processing device including a processor coupled to a memory; The at least one processing device is configured to: receiving, by a code management tool associated with a software testing lifecycle system, a pull request to merge a software code change with a software project repository on an information system; Parsing the pull request code difference file by the regression detection test system to identify the software code change, wherein the software code change includes a plurality of software modules and software functions; For each test case in the plurality of test cases, the regression detection test system queries a test case attribute associated with each test case to identify at least one software module associated with each test case; adding each of the test cases to a first plurality of test cases if the plurality of software modules and software functions includes the at least one software module; if the plurality of software modules and software functions include at least one software function, and the each test case includes the at least one software function, adding the each test case to a second plurality of test cases; as well as At least one of the first plurality of test cases and the second plurality of test cases is executed on a test system by the software testing lifecycle system to perform targeted regression testing on the software code change.

20. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code, when executed by at least one processing device, causes the at least one processing device to: receiving, by a code management tool associated with a software testing lifecycle system, a pull request to merge a software code change with a software project repository on an information system; Parsing the pull request code difference file by the regression detection test system to identify the software code change, wherein the software code change includes a plurality of software modules and software functions; For each test case in the plurality of test cases, the regression detection test system queries a test case attribute associated with each test case to identify at least one software module associated with each test case; adding each of the test cases to a first plurality of test cases if the plurality of software modules and software functions includes the at least one software module; if the plurality of software modules and software functions include at least one software function, and the each test case includes the at least one software function, adding the each test case to a second plurality of test cases; as well as At least one of the first plurality of test cases and the second plurality of test cases is executed on a test system by the software testing lifecycle system to perform targeted regression testing on the software code change.

Images