Code path multi-level coverage test case selection algorithm

Abstract

This disclosure relates to a multi-level code path coverage test case selection algorithm. This document provides methods, systems, and non-transitory processor-readable storage media for a code path test selection system. An example method includes a code path test selection system defining a multi-level test coverage target based on code paths associated with multiple test cases, where the multiple levels correspond to different test hit rates. The code path test selection system generates a test target code path vector representing the code paths to be tested, including vector elements weighted based on the importance of the corresponding code paths, and generates a test case code path vector representing the code paths covered by the multiple test cases. The code path test selection system selects a set of test cases from the multiple test cases based on the test case code path vectors and the test target code path vectors. A CI / CD pipeline system executes this set of test cases on a test system.

Description

Technical Field

[0001] This field typically involves test cases that select multiple levels of coverage based on code path selection. Background Technology

[0002] During the software project lifecycle, source code is updated to add new features and fix discovered issues / bugs. Choosing appropriate test cases to test these new features and bug fixes is a crucial step in the software project lifecycle. Summary of the Invention

[0003] Illustrative embodiments provide techniques for implementing a code path test selection system in a storage system. For example, in an illustrative embodiment, the code path test selection system defines (“defines”) multi-level test coverage targets based on code paths associated with multiple test cases, where the multiple levels correspond to different test hit rates. The code path test selection system generates a test target code path vector representing the code paths to be tested, including vector elements weighted based on the importance of the corresponding code paths. The code path test selection system generates a test case code path vector representing the code paths covered by multiple test cases. The code path test selection system selects a set of test cases from the multiple test cases based on the test case code path vectors and the test target code path vectors. A continuous integration / continuous delivery (CI / CD) pipeline system executes the test case set on the test system to provide multi-level test coverage. Other types of processing devices may be used in other embodiments. These and other illustrative embodiments include, but are not limited to, apparatus, systems, methods, and processor-readable storage media. Attached Figure Description

[0004] Figure 1 An information processing system including a code path test selection system is shown in an illustrative embodiment.

[0005] Figure 2 A flowchart illustrating the process of the code path test selection system in an illustrative embodiment is shown.

[0006] Figure 3 The principle of multi-level code path coverage test selection in the illustrative embodiment is shown.

[0007] Figure 4 The relationship between each vector of multi-level test coverage based on code path is shown in the example embodiment.

[0008] Figure 5 A flowchart illustrating the selection of a test case set in an illustrative embodiment is shown.

[0009] Figure 6The use case index and test case code path vector are shown in the illustrative embodiment.

[0010] Figure 7 The code path vector for the sorting test cases in the illustrative embodiment is shown.

[0011] Figure 8 The first step in iteratively generating a test case set in the example embodiment is shown.

[0012] Figure 9 The second step in iteratively generating a test case set in the example embodiment is shown.

[0013] Figure 10 The twelfth step in iteratively generating a test case set is shown in the example embodiment.

[0014] Figure 11 The final step in iteratively generating a test case set is shown in the example embodiment.

[0015] Figure 12 and Figure 13 An example of a processing platform that can be used to implement at least a portion of an embodiment of a code path test selection system is shown. Detailed Implementation

[0016] This document will describe illustrative embodiments with reference to exemplary computer networks and related computers, servers, network devices, or other types of processing devices. However, it should be understood that these and other embodiments are not limited to use with the specific illustrative network and device configurations shown. Therefore, the term "computer network" as used herein is intended to be interpreted broadly to include, for example, any system comprising, multiple networked processing devices.

[0017] The following describes a technique for implementing a code path test selection system to ensure multi-level testing of critical code paths. The code path test selection system defines multi-level test coverage targets based on code paths associated with multiple test cases, where each level corresponds to a different test hit rate. The code path test selection system generates a test target code path vector representing the code path to be tested, including vector elements weighted based on the importance of the corresponding code path. The system also generates a test case code path vector representing the code paths covered by multiple test cases. Based on the test case code path vectors and the test target code path vectors, the system selects a set of test cases from the multiple test cases. This set of test cases is then executed on a test system by a continuous integration / continuous delivery (CI / CD) pipeline system to provide multi-level test coverage.

[0018] Conventional techniques fail to select test cases for testing source code change requirements, such as source code updates, new features, and code path-based bug fixes, ensuring that the selected test cases accurately target the desired test points. Conventional techniques use a binary approach to select test cases for code updates; they either execute the test cases or not, but they fail to consider the test intensity of various testing requirements. Conventional techniques fail to recognize that important test code paths require more focused and repeated testing, and lack quantitative methods for achieving multiple coverages of different code paths. Conventional techniques cannot support multi-level quantitative test coverage of critical code paths, ensuring that critical code paths of critical test modules are covered multiple times for better overall test quality. Conventional techniques cannot achieve hierarchical test coverage of test objectives through multi-level test code path coverage to ensure multiple tests of critical modules.

[0019] In contrast, in at least some implementations of the current technology described herein, multi-level test code path coverage is implemented by a code path test selection system to optimize test quality. The code path test selection system defines multi-level test coverage targets based on code paths associated with multiple test cases, where multiple levels correspond to different test hit rates. The code path test selection system generates a test target code path vector representing the code path to be tested, including weighted vector elements based on the importance of the corresponding code path. The code path test selection system generates a test case code path vector representing the code paths covered by multiple test cases. The code path test selection system selects a set of test cases from multiple test cases based on the test case code path vector and the test target code path vector. This set of test cases is executed on a test system by a continuous integration / continuous delivery (CI / CD) pipeline system to provide multi-level test coverage.

[0020] Therefore, the current technical objective is to provide a method and system for providing a code path test selection system that selects test cases based on code paths to test source code change requirements, such as source code updates, new features, and bug fixes, ensuring that the selected test cases accurately target the required test points. Another objective is to select test cases to test the source code by considering the test intensity of various test requirements. Yet another objective is to recognize that important test code paths require more focused and repeated testing and to apply quantitative methods to achieve multiple coverages of different code paths. Another objective is to support multi-level quantitative test coverage of critical code paths, ensuring that critical code paths of critical test code modules are covered multiple times for better overall test quality. Another objective is to achieve hierarchical test coverage of test objectives through multi-level test code path coverage to ensure multiple tests of critical modules.

[0021] In at least some implementations of the current technology described herein, the use of a code path test selection system can provide one or more of the following advantages: It provides methods and systems for testing source code change requirements, such as source code updates, new features, and bug fixes, based on code path selection test cases; it ensures that the selected test cases accurately target the required test points; it selects test cases to test the source code by considering the test intensity of various test requirements; it identifies important test code paths that require more focused and repeated testing; it applies quantitative methods to achieve multiple coverages of different code paths; it supports multi-level quantitative test coverage of critical code paths; it ensures that critical code paths of critical test code modules are covered multiple times to achieve better overall test quality; and it achieves hierarchical test coverage of test objectives through multi-level test code path coverage, ensuring multiple tests of critical modules.

[0022] Compared to conventional techniques, in at least some implementations of the current techniques described herein, multi-level test code path coverage is implemented to optimize test quality by a code path test selection system. This system defines multi-level test coverage targets based on code paths associated with multiple test cases, where each level corresponds to a different test hit rate. The code path test selection system generates a test target code path vector representing the code paths to be tested, including weighted vector elements based on the importance of the corresponding code paths. The system also generates a test case code path vector representing the code paths covered by multiple test cases. Based on the test case code path vector and the test target code path vector, the system selects a set of test cases from the multiple test cases. This set of test cases is then executed on a test system by a continuous integration / continuous delivery (CI / CD) pipeline system to provide multi-level test coverage.

[0023] In an example embodiment of the current technology, the CI / CD pipeline system triggers a code path test selection system to select a set of test cases to ensure multiple tests on critical code paths.

[0024] In an exemplary embodiment of the present technology, the multi-level test coverage target enables digital quantification of test intensity for multiple test cases.

[0025] In a current example embodiment of the technology, the code paths of multiple test cases are analyzed at the module level as the smallest unit of invocation.

[0026] In an example embodiment of the current technology, a test target code path vector is generated by analyzing code push requests to identify the modified code and related code paths associated with the modified code, wherein multiple test cases include the modified code.

[0027] In an example embodiment of the current technology, the code path test selection system merges the modified code path and related code paths into the test target code path vector.

[0028] In exemplary embodiments of the present technology, the weighting of vector elements in the test target code path vector is based on factors including the complexity of code modifications, the impact on other modules, and the criticality of the modules.

[0029] In example embodiments of the current technology, the weighting of vector elements is based on low importance level, medium importance level, and high importance level.

[0030] In example embodiments of the current technology, weights are assigned numerical values, with higher values ​​corresponding to higher importance levels.

[0031] In an example embodiment of the current technology, the code path test selection system represents the test target code path vector as a 1×n vector, where n indicates the number of code paths in multiple test cases.

[0032] In an example embodiment of the current technology, if the corresponding code path has changed, each element in the test target code path vector is set to 1, and if the corresponding code path has not changed, it is set to 0.

[0033] In an example embodiment of the current technology, the code path test selection system represents each test case code path vector as a 1×n vector, where n indicates the number of code paths in multiple test cases.

[0034] In an example embodiment of the current technology, if the corresponding code path is covered by a test case, the code path vector for each test case is set to 1, and if the corresponding code path is not covered, it is set to 0.

[0035] In an example embodiment of the current technology, the test case code path vector is represented as a matrix, with each row corresponding to a test case and each column corresponding to a code path.

[0036] In an example embodiment of the current technology, the code path test selection system sorts the test case code path vectors based on the total number of code paths covered by each test case.

[0037] In an example embodiment of the current technology, test cases covering the same number of code paths are sorted by their index.

[0038] In an example embodiment of the current technology, the code path test selection system generates an updated target vector by subtracting the test case code path vector from the test target code path vector. If the updated target vector is different from the test target code path vector, the test case is added to the test case set. If the updated target vector does not contain positive elements, the process terminates; otherwise, the updated target vector is used to replace the test target code path vector to continue iterative selection.

[0039] In an exemplary embodiment of the present technology, the code path test selection system terminates when all required code paths are covered by a set of test cases.

[0040] Figure 1 A computer network (also referred to herein as an information processing system) 100 configured according to an illustrative embodiment is shown. Computer network 100 includes a continuous integration / continuous delivery (CI / CD) pipeline system 105, a code path test selection system 106, a test system 102, and a code repository 101. In the example embodiment, CI / CD, as a term used herein, generally refers to continuous integration, continuous deployment, and / or continuous delivery. Such functionality, or portions thereof, is considered an example of the widely used term "software development process" herein. In other embodiments, various other types of software development processes may be utilized, illustratively relating to the integration, deployment, and / or other aspects of software development of one or more source codes executed on other systems, such as test system 102 (or multiple test systems 102-N, not shown). CI / CD pipeline system 105, test system 102, code path test selection system 106, and code repository 101 are coupled to network 104, wherein in this embodiment it is assumed that network 104 represents a subnetwork or other relevant portion of a larger computer network 100. Therefore, elements 100 and 104 are both referred to as examples of the "network" in this paper, but in Figure 1 In the context of this embodiment, it is assumed that the latter is a component of the former. Also coupled to network 104 is a code path test selection system 106 that can reside on the storage system. Such a storage system can include any of various types of storage devices, including network-attached storage (NAS), storage area network (SAN), direct-attached storage (DAS), and distributed DAS, as well as combinations of these and other storage types, including software-defined storage devices.

[0041] Each of the CI / CD pipeline system 105, test system 102, code path test selection system 106, and code repository 101 may include, for example, a portion of a server and / or one or more server systems, as well as devices such as mobile phones, laptops, tablets, desktop computers, or other types of computing devices. Such devices are examples of what are generally referred to herein as “processing devices.” Some of these processing devices are also generally referred to herein as “computers.”

[0042] In some embodiments, the CI / CD pipeline system 105, the testing system 102, the code path test selection system 106, and the code repository 101 include corresponding computers associated with a specific company, organization, or other enterprise. Furthermore, at least a portion of the computer network 100 may also be collectively referred to herein as including an "enterprise network." As those skilled in the art will understand, many other operational scenarios involving various types and arrangements of processing equipment and networks are possible.

[0043] Furthermore, it should be understood that the term "user" in the context of this document and elsewhere is intended to be interpreted broadly as including, for example, human, hardware, software, or firmware entities, as well as various combinations of these entities.

[0044] Assuming network 104 includes portions of a global computer network such as the Internet, although other types of networks may be portions of computer network 100, including wide area networks (WANs), local area networks (LANs), satellite networks, telephone or wired networks, cellular networks, wireless networks such as Wi-Fi or WiMAX networks, or various portions or combinations of these and other types of networks. Therefore, in some embodiments, computer network 100 includes a combination of multiple different types of networks, each including processing devices configured to communicate using Internet Protocol (IP) or other relevant communication protocols.

[0045] Also associated with the code path test selection system 106 are one or more input / output devices, which illustratively include a keyboard, a display, or any other combination of input / output devices. Such input / output devices can be used, for example, to support one or more user interfaces to the code path test selection system 106, and to support communication between the code path test selection system 106 and other related systems and devices not explicitly shown. For example, a dashboard can be provided to the user to view the execution progress of the code path test selection system 106. One or more input / output devices may also be associated with any CI / CD pipeline system 105, test system 102, code path test selection system 106, and code repository 101.

[0046] Furthermore, it is assumed that at least one processing device is used to implement this. Figure 1 The code path test selection system 106 in this embodiment. Each such processing device typically includes at least one processor and associated memory, and implements one or more functional modules for controlling certain features of the code path test selection system 106.

[0047] More specifically, the code path test selection system 106 in this embodiment may include a processor coupled to the memory and network interface.

[0048] Processors illustratively include microprocessors, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other types of processing circuitry, as well as portions or combinations of these circuitry elements.

[0049] Memory illustratively includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory and other memory disclosed herein can be considered as examples of storage of executable computer program code or other types of software programs, more commonly referred to as "processor-readable storage media".

[0050] One or more embodiments include articles of manufacture, such as computer-readable storage media. Examples of articles of manufacture include, but are not limited to, storage devices such as storage disks, storage arrays, or integrated circuits containing memory, as well as various other types of computer program products. The term “article of manufacture” as used herein should be understood to exclude transient propagation signals. These and other references to “disk” herein are intended to generally refer to storage devices, including solid-state drives (SSDs), and therefore should not be construed as being limited in any way to rotating magnetic media.

[0051] The network interface allows the code path test selection system 106 to communicate with the CI / CD pipeline system 105, the test system 102, the code path test selection system 106, and the code repository 101 via network 104, and illustratively includes one or more conventional transceivers.

[0052] The code path test selection system 106 can be implemented, at least in part, as software stored in memory and executed by a processor, and can reside in any processing device. The code path test selection system 106 can also be a standalone plugin that can be included within the processing device.

[0053] It should be understood that Figure 1The specific set of CI / CD pipeline system 105, test system 102, code path test selection system 106, and code repository 101 shown for computer network 100 is presented merely as an illustrative example, and additional or alternative elements may be used in other embodiments. Thus, another embodiment includes additional or alternative systems, devices, and other network entities, as well as different arrangements of modules and other components. For example, in at least one embodiment, one or more code path test selection systems 106 may be on the same processing platform and / or on portions of the same processing platform. Reference will be made to, for example... Figure 2 The flowchart describes in more detail the exemplary process of the code path test selection system 106 in computer network 100.

[0054] Figure 2 This is a flowchart illustrating the process of executing the code path test selection system 106 in an illustrative embodiment. It should be understood that this particular method is merely an example, and additional or alternative processes may be performed in other embodiments.

[0055] At point 200, the code path test selection system 106 defines multi-level test coverage targets based on code paths associated with multiple test cases. In the example embodiment, the multiple levels correspond to different test hit rates. Figure 3 This illustrates the principle behind multi-level code path coverage test selection. Figure 3 The left side shows traditional code path coverage that covers all code paths. Figure 3 The right side shows code paths in different colors, where each color represents a path that meets different test intensity requirements. For example, a blue code path requires test cases to cover it once, while a red code path requires test cases to cover it multiple times to ensure that the code path is fully tested.

[0056] In the example embodiment, the code path test selection system 106 analyzes the code paths of multiple test cases at the module level into the smallest unit of invocation. A code path is a specific set of instructions executed during a single run of a program or program segment. Path-based code paths are very complex and numerous. The code path test selection system 106 analyzes modules into the smallest unit of invocation of code paths to ensure the logical integrity of the code paths and also to reduce the computational cost of code paths. In the example embodiment, multi-level test coverage objectives enable the numerical quantification of the test intensity of multiple test cases. In the example embodiment, multi-level test coverage objectives are defined according to requirements, and different levels (of the multi-level) correspond to different test hit rates.

[0057] At position 202, the code path test selection system 106 generates the test target code path vector V.t This represents the code path to be tested, including weighted vector elements based on the importance of each code path. In the example embodiment, the set of code paths used for multiple test cases is P. f , where P f ={P1, P2, P3, P4,…,P n}. Variable P i Indicates the function-based code path. The code path vector used for multiple test cases is represented as V. f A 1×n vector, where all items have the value 1, means that all code paths P are covered. i In the example embodiment, the test target code path vector is represented as V. t V t Each item in is mapped to P f The code path in the file. If the code path P i If it has been changed, then the code path test selects system 106 and sets the value to 1. If P i If the code path in the test is unchanged, then the code path test selects system 106 and sets the value to 0.

[0058] V t =[δ i (i = 1, 2, 3, ..., n)

[0059]

[0060] The code path test selection system 106 generates a test target code path vector by analyzing code push requests to identify modified code and related code paths associated with the modified code, where multiple test cases include the modified code. In an example embodiment, the test target code can be a bug fix code change, a feature code change, or any code change requirement to be tested. For example, the code path test selection system 106 generates a test target code path vector V. t V t = [10111…10]. In an example embodiment, the code path test selection system 106 merges the modified code path and related code paths into the test target code path vector. In an example embodiment, push requests can be analyzed to identify the modified test code, and then the related code paths can be obtained. In an example embodiment, the weighting of vector elements in the test target code path vector is based on factors including the complexity of the code modification, the impact on other modules, and the criticality of the module.

[0061] In the example embodiment, different weight levels, such as low, medium, and high, are assigned to code paths based on developer feedback. For example, a weight of "low" means that the code path should be tested fewer times on test system 102 than a weight of "medium," while a weight of "high" means that the code path should be tested more times than a code path with a weight of "low" or "medium." In the example embodiment, weights are assigned numerical values, where higher values ​​correspond to higher importance levels. For example, a weight of "low" could be assigned "1," a weight of "medium" could be assigned "5," and a weight of "high" could be assigned "10." In the example embodiment, code path test selection system 106 represents each test case code path vector as a 1×n vector, where n indicates the number of code paths in multiple test cases. In the example embodiment, if the corresponding code path has changed, code path test selection system 106 sets each element in the test target code path vector to 1, and sets it to 0 if the corresponding code path has not changed.

[0062] At position 204, the code path test selection system 106 generates a test case code path vector representing the code paths covered by multiple test cases. In an example embodiment, the code path test selection system 106 represents each test case code path vector as a 1×n vector, where n indicates the number of code paths in the multiple test cases. In an example embodiment, the test case code path vector V c This indicates the code path of the test case and represents the runtime code path of the test case. In V c In the code, the item with the value 1 indicates the code path P. i The item covered by the test case, and has a value of 0, indicates the code path P. i Not covered by test cases. For example, V c =[00110…00].

[0063] V c =[δ i (i = 1, 2, 3, ..., n)

[0064]

[0065] In the example embodiment, the code path test selection system 106 represents the test case code path vector as a matrix, where each row corresponds to a test case and each column corresponds to a code path. In the example embodiment, for multiple test cases M, the code path test selection system 106 represents the test case code path array A as follows. tc :

[0066] For example:

[0067] In the example embodiment, the code path test selection system 106 defines a multi-level test target code path vector V'. t As mentioned above, code paths can be assigned different weight levels, for example:

[0068]

[0069] V′ t =V t *W t =[0 1 0 5 ... 5 10 0]

[0070] Figure 4 The relationship between each vector of multi-level test coverage based on code path is shown.

[0071] In an example embodiment, the code path test selection system 106 sorts each of a plurality of test cases, wherein the plurality of test cases have connections to a given push request. The code path test selection system 106 defines a vector V. Ci Let i represent the runtime code path of test case i, where i represents the index among multiple test cases.

[0072] In an example embodiment, the code path test selection system 106 selects the test case code path vector V based on the total number of code paths covered by each test case. c1 V c2 , ..., V cn Sort the test cases. In the example implementation, if there are more than one test case covering the same number of code paths, these test cases are sorted by their index i.

[0073] At point 206, the code path test selection system 106 selects a test case set from multiple test cases based on the test case code path vector and the test target code path vector. In the example embodiment, the code path test selection system 106 iteratively selects the test case set by subtracting the test case code path vector from the test target code path vector to generate an updated target vector. If the updated target vector is different from the test target code path vector, the test case is added to the test case set. If the updated target vector does not contain positive elements, the process terminates; otherwise, iteratively selecting an updated target vector to replace the test target code path vector continues. Figure 5 A flowchart for selecting a test case set is shown, where the test case set is represented by the selected list S.

[0074] In an example embodiment, the code path test selection system 106 generates a one-dimensional vector V' representing the code paths involved. tThe items in the vector are weighted based on factors such as the complexity of code modifications, whether the modifications affect other modules, and whether the module is critical. All code paths are also identified. The sorted group A is used. tc The code path test selects system 106 from V' t Subtract V from the middle ci To obtain V tnew (Where all negative values ​​are specified as 0). If in V t and V tnew If any distinct elements exist, the code path test selection system 106 will select V. ci The test cases are assigned to the selected list S (i.e., the test case set). In the example embodiment, the code path test selection system 106 terminates the process when the test case set covers all required code paths. In other words, if in V tnew If there are no positive elements, the code path test will terminate the process if system 106 is selected (otherwise V tnew Assigned to V' t (Used as another loop in the iterative process).

[0075] In the example implementation, 15 test cases (T1 to T15) are listed below, each covering 8 target code paths (P1 to P8).

[0076]

[0077]

[0078] In an example embodiment, the code path test selection system 106 creates a one-dimensional vector V′t with eight elements, each element representing one of the eight code paths connected to a given pull request. The value of each element is weighted, with 1 representing a low-importance code path, 5 representing a medium-importance code path, and 10 representing the most important code path.

[0079] V′ t =[0 1 5 1 5 1 10 5]

[0080] In the example embodiment, the code path test selection system 106 models the test case code path vector table as two matrices. The first matrix represents the test case index, and the second matrix represents the relationship between test cases and code paths (where 1 indicates that the test case and the code path are connected). Figure 6 The use case index and test case code path vector are shown.

[0081] In an exemplary embodiment, the code path test selection system 106 generates a test case code path vector A by sorting the test case code path vectors in descending order according to the number of code paths.tc If the same number of code paths are found, the code path test selection system 106 sorts the test case code path vectors in ascending order by test case index. Figure 7 The sorted test case code path vectors are shown.

[0082] In an example embodiment, the code path test selection system 106 generates an updated target vector by subtracting the test case code path vector from the test target code path vector. In other words, the code path test selection system 106 selects the target vector from V′. t Subtract V from the middle ci To obtain V tnew ,like Figure 8 As shown below:

[0083] V tnew =V′ t -V c1 =[0 0 4 0 4 0 10 4]

[0084] V tnew <>V′ t => T c1 Add to the selected list S

[0085] S = [TC1]

[0086] V tnew >0=> V tnew Assigned to V′ t Then switch to TC5

[0087] In the example embodiment, the code path test selection system 106 performs this step iteratively, such as... Figure 9 As shown, and as follows:

[0088] V tnew =V′ t -V c5 =[0 0 3 0 3 0 9 3]

[0089] V tnew ·<>·V′ t => Add TC5 to the selected list S

[0090] S = [TC1, TC5]

[0091] In the example embodiment, the code path test selection system 106 continues to iteratively execute this step, such as... Figure 10 As shown, and as follows:

[0092] V′ t >0=> Assign V tnew Give V′ tAnd switch to TC6

[0093] V tnew ·=·V′ t • => Do not add TC13 to the selected list S.

[0094] S=[TC1, TC5, TC6, TC14, TC2, TC3, TC8, TC12, TC10, TC11]

[0095] V′ t >0=> Assign V tnew Give V t And transferred to TC15

[0096] In the example embodiment, the code path test selection system 106 iteratively performs this step until V. t and V C6 There is no difference between them, such as Figure 11 And as shown below:

[0097] V tnew =V t -V c6 =[0 0 0 0 0 0 0 0]

[0098] V tnew ·<>·V′ t • => Add TC5 to the selected list SS = [TC1, TC5, TC6, TC14, TC2, TC3, TC8, TC12, TC10, TC11, TC15]V′ t =0=> Selection complete

[0099] The test case set S is determined as follows:

[0100] S=[TC1, TC5, TC6, TC14, TC2, TC3, TC8, TC12, TC10, TC11, TC15]

[0101] At point 208, CI / CD pipeline system 105 executes test case set S on test system 102 to provide multi-level test coverage. In an example embodiment, CI / CD pipeline system 105 triggers a code path test selection system to select a test case set to ensure multiple tests of critical code paths. For example, code path test selection system 106 may be incorporated into CI / CD pipeline system 105 as a phase within CI / CD pipeline system 105. In an example embodiment, test cases are committed to codebase 101, such as GitLab, when they are created and / or updated.

[0102] Therefore, with Figure 2 process Figure 1The specific processing operations and other functions described herein are presented by way of illustrative example only and should not be construed as limiting the scope of this disclosure in any way. For example, in other embodiments, the order of processing steps may be changed, or certain steps may be performed simultaneously rather than sequentially.

[0103] The illustrative embodiments described above offer significant advantages over conventional methods. For example, some embodiments are configured to provide methods and systems for providing a code path test selection system. Compared to conventional methods, these and other embodiments can effectively achieve multi-level test code path coverage to optimize test quality. For example, the embodiments disclosed herein select test cases to test source code change requirements, such as code path-based source code updates, new features, and bug fixes, to ensure that the selected test cases accurately target the required test points. The embodiments disclosed herein select test cases to test the source code by considering the test intensity of various test requirements. The embodiments disclosed herein recognize that important test code paths require more focused and repeated testing and apply quantitative methods to achieve multiple coverages of different code paths. The embodiments disclosed herein support multi-level quantitative test coverage of critical code paths, ensuring that critical code paths of critical test code modules are covered multiple times to achieve better overall test quality. The embodiments disclosed herein achieve hierarchical test coverage of test objectives through multi-level test code path coverage to ensure multiple tests of critical modules.

[0104] It should be understood that the specific advantages described above and elsewhere in this document are associated with specific illustrative embodiments and are not necessarily present in other embodiments. Furthermore, the specific types of information processing system features and functions shown in the accompanying drawings and described above are merely exemplary, and many other arrangements may be used in other embodiments.

[0105] As previously described, at least a portion of the information processing system 100 can be implemented using one or more processing platforms. A given such processing platform includes at least one processing device comprising a processor coupled to memory. In some embodiments, the processor and memory include corresponding processor and memory elements using virtual machines or containers provided by one or more underlying physical machines. The term "processing device" as used herein is intended to be broadly interpreted to include various arrangements of physical processors, memory, and other device components, as well as virtual instances of these components. For example, in some embodiments, a "processing device" may include or execute across one or more virtual processors. Thus, a processing device can be physical or virtual and can execute on one or more physical or virtual processors. It should also be noted that a given virtual device may be mapped to a portion of a physical device.

[0106] Some illustrative embodiments of a processing platform for implementing at least a portion of an information processing system include a cloud infrastructure comprising virtual machines implemented using a hypervisor running on a physical infrastructure. The cloud infrastructure also includes a set of applications running on the respective virtual machines under the control of the hypervisor. Multiple hypervisors may also be used, each providing a set of virtual machines using at least one underlying physical machine. Different sets of virtual machines provided by one or more hypervisors can be used to configure multiple instances of various components of the system.

[0107] These and other types of cloud infrastructure can be used to provide content, also known here as a multi-tenant environment. One or more system components or portions thereof are illustratively implemented for use by tenants in such a multi-tenant environment.

[0108] As previously stated, cloud infrastructures disclosed herein may include cloud-based systems. Virtual machines provided in these systems may be used to implement at least a portion of the computer systems in the illustrative embodiments.

[0109] In some embodiments, the cloud infrastructure additionally or alternatively includes multiple containers implemented using container host devices. For example, as described in detail herein, a given container in the cloud infrastructure illustratively includes Docker containers or other types of Linux containers (LXC). Containers run on virtual machines in a multi-tenant environment, although other arrangements are also possible. Containers are used to implement various different types of functionality within the information processing system 100. For example, containers can be used to implement corresponding processing devices that provide computing and / or storage services for cloud-based systems. Similarly, containers can be used in conjunction with other virtualization infrastructures such as virtual machines implemented using hypervisors.

[0110] Now refer to Figure 12 and Figure 13 An illustrative embodiment of the processing platform is described in more detail. Although described in the context of information processing system 100, these platforms may also be used to implement at least a portion of other information processing systems in other embodiments.

[0111] Figure 12An example processing platform including cloud infrastructure 1200 is illustrated. Cloud infrastructure 1200 includes a combination of physical and virtual processing resources for implementing at least a portion of information processing system 100. Cloud infrastructure 1200 includes multiple virtual machines (VMs) and / or container sets 1202-1, 1202-2, ... 1202-L implemented using virtualization infrastructure 1204. Virtualization infrastructure 1204 runs on physical infrastructure 1205 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructures. Operating system-level virtualization infrastructure illustratively includes the kernel control group of Linux operating system or other types of operating system.

[0112] Cloud infrastructure 1200 also includes application sets 1210-1, 1210-2, ... 1210-L, running on a corresponding VM / container set within VM / container sets 1202-1, 1202-2, ... 1202-L, under the control of virtualization infrastructure 1204. VM / container set 1202 includes a corresponding VM, one or more corresponding container sets, or a corresponding set of one or more containers running within a VM. Figure 12 In some implementations of the embodiments, the VM / container set 1202 includes a corresponding VM implemented using a virtualization infrastructure 1204 including at least one hypervisor.

[0113] The hypervisor platform can be used to implement the hypervisor within the virtualization infrastructure 1204, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines include one or more distributed processing platforms, which include one or more storage systems.

[0114] exist Figure 12 In other implementations of the embodiments, VM / container set 1202 includes corresponding containers implemented using virtualization infrastructure 1204, which provides operating system-level virtualization capabilities, support for Docker containers running on a metal host or on a VM. The containers are illustratively implemented using various kernel control groups of the operating system.

[0115] As can be clearly seen from the above, one or more processing modules or other components of the information processing system 100 can each run on a computer, server, storage device, or other processing platform element. The given example of such an element is considered to be more commonly referred to herein as a "processing device". Figure 12 The cloud infrastructure 1200 shown can represent at least a portion of a processing platform. Another example of such a processing platform is... Figure 13 The processing platform shown is 1300.

[0116] The processing platform 1300 in this embodiment includes a portion of the information processing system 100 and includes multiple processing devices 1302-1, 1302-2, 1302-3, ... 1302-K, which communicate with each other through network 1304.

[0117] Network 1304 includes any type of network, such as global computer networks including the Internet, WANs, LANs, satellite networks, telephone or wired networks, cellular networks, wireless networks such as Wi-Fi or WiMAX networks, or various parts or combinations of these and other types of networks.

[0118] The processing device 1302-1 in the processing platform 1300 includes a processor 1310 coupled to a memory 1312.

[0119] Processor 1310 includes a microprocessor, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other types of processing circuitry, as well as portions or combinations of these circuitry elements.

[0120] Memory 1312 includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory 1312 and other memories disclosed herein should be considered as illustrative examples of what is more commonly referred to as a "processor-readable storage medium" storing executable program code for one or more software programs.

[0121] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. Such articles of manufacture include, for example, any memory array, memory disk, or integrated circuit containing RAM, ROM, or other electronic memory, or a variety of other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagation signals. Many other types of computer program products including processor-readable storage media can be used.

[0122] The processing device 1302-1 also includes a network interface circuit 1314 for connecting the processing device to the network 1304 and other system components, and may include a conventional transceiver.

[0123] Assume that the other processing devices 1302 of the processing platform 1300 are configured in a similar manner to the processing device 1302-1 shown in the figure.

[0124] Furthermore, the specific processing platform 1300 shown in the figure is presented only as an example, and the information processing system 100 may include additional or alternative processing platforms, as well as multiple different processing platforms in any combination, each of which includes one or more computers, servers, storage devices or other processing devices.

[0125] For example, other processing platforms used to implement the illustrative embodiments may include different types of virtualization infrastructure instead of virtualization infrastructure that includes virtual machines, or may include different types of virtualization infrastructure in addition to virtualization infrastructure that includes virtual machines. Such virtualization infrastructure illustratively includes container-based virtualization infrastructure configured to provide Docker containers or other types of LXC.

[0126] As another example, in some embodiments, the various parts of a given processing platform may include convergence infrastructure.

[0127] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used. At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0128] Furthermore, numerous other arrangements of computers, servers, storage products or devices, or other components are possible within the information processing system 100. These components can communicate with other elements of the information processing system 100 via any type of network or other communication medium.

[0129] For example, specific types of storage products that can be used to implement the distributed processing system in the illustrative embodiments include all-flash and hybrid flash storage arrays, scaled-down all-flash storage arrays, scaled-down NAS clusters, or other types of storage arrays. Combinations of these and several other storage products can also be used to implement the given storage system in the illustrative embodiments.

[0130] It should be emphasized again that the above embodiments are for illustrative purposes only. Many variations and other alternative embodiments can be used. Furthermore, in other embodiments, the specific configurations of the system and device elements schematically shown in the drawings, as well as the associated processing operations, can be changed. Thus, for example, the specific types of processing devices, modules, systems, and resources deployed in a given embodiment, and their corresponding configurations, can be changed. Moreover, the various assumptions made above in describing exemplary embodiments should be considered exemplary rather than requirements or limitations of this disclosure. Many other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.

Claims

1. A method comprising: The code path test selection system defines multi-level test coverage targets based on code paths associated with multiple test cases, where the multi-levels correspond to different test hit rates; The code path test selection system generates a test target code path vector representing the code path to be tested, including vector elements weighted based on the importance of the corresponding code path; The code path test selection system generates a test case code path vector representing the code paths covered by the multiple test cases; The code path test selection system selects a set of test cases from the plurality of test cases based on the test case code path vector and the test target code path vector. as well as The test case set is executed on the test system by a continuous integration / continuous delivery (CI / CD) pipeline system to provide the multi-level test coverage, wherein the method is implemented by at least one processing device including a processor coupled to memory.

2. The method according to claim 1, further comprising: The CI / CD pipeline system triggers the code path test selection system to select the test case set to ensure multiple tests on critical code paths.

3. The method of claim 1, wherein the multi-level test coverage objective enables digital quantification of the test intensity of the plurality of test cases.

4. The method according to claim 1, wherein defining the multi-level test coverage target by the code path test selection system includes: The code paths of the multiple test cases are analyzed at the module level as the smallest unit of invocation.

5. The method of claim 1, wherein the test target code path vector is generated by analyzing code push requests to identify modified code and related code paths associated with the modified code, wherein the plurality of test cases include the modified code.

6. The method according to claim 5, further comprising: The modified code path and the relevant code path are merged into the test target code path vector.

7. The method of claim 1, wherein the weighting of the vector elements in the test target code path vector is based on factors including the complexity of the code modification, the impact on other modules, and the criticality of the modules.

8. The method of claim 7, wherein the weighting of the vector elements is based on low importance level, medium importance level, and high importance level.

9. The method of claim 8, wherein the weights are assigned numerical values, with higher values ​​corresponding to higher importance levels.

10. The method of claim 1, wherein generating the test target code path vector by the code path test selection system comprises: The test target code path vector is represented as a 1×n vector, where n indicates the number of code paths in the plurality of test cases.

11. The method of claim 10, wherein if the corresponding code path has changed, each element in the test target code path vector is set to 1, and if the corresponding code path has not changed, it is set to 0.

12. The method of claim 1, wherein generating the test case code path vector by the code path test selection system comprises: Each test case code path vector is represented as a 1×n vector, where n indicates the number of code paths in the plurality of test cases.

13. The method of claim 12, wherein if the corresponding code path is covered by the test case, then each element in the code path vector of each test case is set to 1, and if the corresponding code path is not covered, then each element in the code path vector of each test case is set to 0.

14. The method of claim 1, wherein the test case code path vector is represented as a matrix, with each row corresponding to a test case and each column corresponding to a code path.

15. The method according to claim 1, further comprising: The test case code path vector is sorted based on the total number of code paths covered by each test case.

16. The method of claim 15, wherein the test cases covering the same number of code paths are sorted by their index.

17. The method of claim 1, wherein selecting the test case set by the code path test selection system comprises: The test case set is selected iteratively using the following method: Subtract the test case code path vector from the test target code path vector to generate the updated target vector; If the updated target vector is different from the test target code path vector, then the test case is added to the test case set; as well as If the updated target vector does not contain positive elements, the process terminates; otherwise, the iterative selection continues by replacing the test target code path vector with the updated target vector.

18. The method of claim 17, wherein the code path test selection system terminates when all required code paths are covered by the test case set.

19. A system comprising: At least one processing device, including a processor coupled to a memory; The at least one processing device is configured to: The code path test selection system defines multi-level test coverage targets based on the code paths associated with the multiple test cases, wherein the multi-levels correspond to different test hit rates; The code path test selection system generates a test target code path vector representing the code path to be tested, including vector elements weighted based on the importance of the corresponding code path; The code path test selection system generates a test case code path vector representing the code paths covered by the multiple test cases; The code path test selection system selects a set of test cases from the plurality of test cases based on the test case code path vector and the test target code path vector. as well as The test case set is executed on the test system by the continuous integration / continuous delivery (CI / CD) pipeline system to provide the multi-level test coverage.

20. A computer program product comprising a non-transitory processor-readable storage medium wherein program code of one or more software programs is stored, wherein the program code, when executed by at least one processing device, causes the at least one processing device to: The code path test selection system defines multi-level test coverage targets based on the code paths associated with the multiple test cases, wherein the multi-levels correspond to different test hit rates; The code path test selection system generates a test target code path vector representing the code path to be tested, including vector elements weighted based on the importance of the corresponding code path; The code path test selection system generates a test case code path vector representing the code paths covered by the multiple test cases; The code path test selection system selects a set of test cases from the plurality of test cases based on the test case code path vector and the test target code path vector. as well as The test case set is executed on the test system by the continuous integration / continuous delivery (CI / CD) pipeline system to provide the multi-level test coverage.

Images