Method and system for testing software application components

Abstract

The present invention relates to a method and system for testing software application components. Aspects of the present invention disclose a method, computer program product, and system for performing testing on a portion of an application. The method includes one or more processors identifying a test configuration for testing the application. The application includes a plurality of components. The test configuration includes an indication of at least one component of the application to be tested. The method further includes one or more processors testing the indicated at least one component of the application. The method further includes one or more processors determining a verification result of the indicated at least one component of the application.

Description

Technical Field

[0001] The present invention generally relates to the field of integration testing, and more particularly to testing components of software applications. Background Art

[0002] Software testing is an investigation that is performed to provide stakeholders with information about the quality of a software product or service being tested. Software testing can also provide an objective, independent view of the software to allow an enterprise to understand and appreciate the risks associated with the software implementation. Testing techniques include executing a program or application to find software bugs (errors or other defects), the process of verifying that a software product is fit for use, and other techniques. Software testing involves executing software components or system components to evaluate one or more properties of interest. Typically, these properties indicate the extent to which the component or system being tested: meets the requirements that guided its design and development; responds correctly to various inputs; performs its functions within an acceptable time; can be adequately utilized; can be installed and run in its intended environment; and achieves the overall results expected by its stakeholders.

[0003] Integration testing (sometimes called integration and testing, abbreviated as I&T) is a phase in software testing where individual software modules are combined and tested as a group. Integration testing is performed to evaluate the compliance of a system or component with specified functional requirements. Integration testing occurs after unit testing and before validation testing. Integration testing takes the inputs of the modules that have been unit tested, groups the modules into larger aggregates, applies the tests defined in the integration test plan to the aggregates, and delivers the output as an integrated system in preparation for system testing.

[0004] Continuous delivery (CD) is a software engineering approach in which a team produces software in short cycles, ensuring that the software can be reliably released at any time, and when the software is released, this is done manually. The aim of continuous delivery is to build, test, and release software at a faster pace and frequency. This approach helps to reduce costs, time, and the risk of delivering changes by allowing more incremental updates to the applications in production. For continuous delivery, a simple and repeatable deployment process is important. Continuous delivery is contrasted with continuous deployment, which is a similar approach where software is also produced in short cycles, but through automated deployment rather than manual deployment. Summary of the Invention

[0005] Aspects of the present invention disclose a method, computer program product, and system for performing tests on a portion of an application. The method includes: one or more processors identifying a test configuration for testing the application. The application includes a plurality of components. The test configuration includes an indication of testing at least one component of the application. The method further includes: one or more processors testing the indicated at least one component of the application. The method further includes: one or more processors determining a verification result of testing the indicated at least one component of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a functional block diagram of a data processing environment in accordance with an embodiment of the present invention;

[0007] Figure 2A depicts an example application including a plurality of software components in accordance with an embodiment of the present invention;

[0008] Figure 2B depicts an example application including a plurality of software components in accordance with an embodiment of the present invention;

[0009] Figure 3 is a flowchart depicting operational steps of a program for performing tests on a portion of an application in accordance with an embodiment of the present invention; and

[0010] Figure 4 depicts components of a computing system representative of Figure 1 a computing device and a test management system in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0011] Embodiments of the present invention are directed to providing a software integration test framework that works with software change management and continuous delivery systems to automatically detect, test, and verify changes to software components, thereby reducing the need to repeatedly run full end-to-end integration tests. Embodiments of the present invention recognize the advantages of the corresponding process, particularly when end-to-end integration tests are long, taking several hours or even longer (e.g., for machine learning training tasks).

[0012] Some embodiments of the present invention recognize that performing analysis involving machine learning models requires multiple steps involving feature generation, feature selection, model training, and other steps. The entire process can be highly complex and require a long duration to fully complete. Additionally, while the examples discussed are for machine learning, embodiments of the present invention recognize that many different implementations and functions utilize long-running tests that require multiple prior steps to be executed before the steps that have changed can be verified. Embodiments of the present invention recognize that solving problems in the later part of the process depends on executing all earlier steps to build the necessary information for verification.

[0013] Accordingly, embodiments of the present invention recognize that a process that may take hours may need to be performed before a (even relatively simple) change to an application can be verified to determine if the change is correct and has no adverse effect on downstream processing of the application. Such a limitation may typically result in a delayed verification of a fix or update because steps (before and after the changed section) within the process or application must be completed to ensure that the process or application is not regressive.

[0014] In various aspects, embodiments of the present invention provide a process for testing / executing a set of operations (e.g., software components) of an application without having to re-run all previous operations (e.g., software components) in order to verify the change at a later step / operation. Some of the following examples relate to integration testing, but the corresponding concepts and patterns can also extend to any multi-step test scenario.

[0015] Embodiments of the present invention are used to perform a complete and verified end-to-end test on a system (e.g., an application including multiple software components), which includes capturing the state (e.g., metadata and data) of each step (e.g., software component) of the system, including inputs and outputs. Store the verified end-to-end test. Then, when the system needs to be re-verified (e.g., in response to a change to one or more software components of the application), embodiments of the present invention are capable of reloading the system to the state from a valid end-to-end test and executing the steps affected by the change that caused the re-verification. Embodiments of the present invention are then capable of verifying that the metadata and data obtained from the execution of the step(s) are the same as the metadata and data obtained from the verified end-to-end test.

[0016] Accordingly, embodiments of the present invention recognize that existing solutions are capable of providing the ability to restart system operation (set checkpoints) at a fault point. Additionally, embodiments of the present invention recognize that other alternative solutions are capable of providing the ability to capture a state and then restart at the time point corresponding to that state. However, embodiments of the present invention recognize that such solutions are not integrated with the source system to understand the steps (and corresponding components) that need to be re-tested, but leave the determination / identification as a manual exercise. Additionally, embodiments of the present invention recognize that other solutions are capable of providing the ability to use the previous output state of a component as a way to verify that a change has not occurred. However, embodiments of the present invention recognize that such current solutions do not provide a complete end-to-end process for solving the problem.

[0017] Implementations of embodiments of the present invention may take various forms, and exemplary implementation details are subsequently discussed with reference to the accompanying drawings.

[0018] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1is a functional block diagram showing a distributed data processing environment (generally designated 100) in accordance with one embodiment of the present invention. Figure 1 Only an illustration of one implementation is provided and does not imply any limitation to the environments in which different embodiments may be implemented. Those skilled in the art may make many modifications to the depicted environment without departing from the scope of the present invention as recited in the claims.

[0019] Embodiments of the data processing environment 100 include a computing device 110 and a test management system 120, both interconnected by a network 105. In an example embodiment, the test management system 120 represents a computing device (e.g., one or more management servers) that may provide a software integration testing framework that works with software change management and continuous delivery systems to automatically detect, test, and verify changes to software components (e.g., to an application such as application 200 on computing device 110), thereby reducing the need to repeatedly run full end-to-end integration tests. In other embodiments, in accordance with various embodiments of the present invention, the data processing environment 100 may include additional instances of computing devices (not shown) that may interface with the test management system 120.

[0020] The network 105 may be, for example, a local area network (LAN), a telecommunications network, a wide area network (WAN) such as the Internet, or any combination of the three, and includes wired, wireless, or fiber optic connections. Generally, in accordance with embodiments of the present invention, the network 105 may be any combination of connections and protocols that support communication between the computing device 110 and the test management system 120. In various embodiments, the network 105 facilitates communication between multiple networked computing devices (e.g., the computing device 110 and the test management system 120, and other devices not shown), corresponding users (e.g., users associated with the computing device 110 and the test management system 120, etc.), and corresponding management services (e.g., the test management system 120).

[0021] In various embodiments of the present invention, in accordance with embodiments of the present invention, the computing device 110 may be a workstation, a personal computer, a personal digital assistant, a mobile phone, or any other device capable of executing computer-readable program instructions. Generally, the computing device 110 represents any electronic device or combination of electronic devices capable of executing computer-readable program instructions. In accordance with embodiments of the present invention, the computing device 110 may include components Figure 4 further depicted and described in detail.

[0022] Computing device 110 includes application 200. In an example embodiment, computing device 110 is a personal workstation or a mobile device that includes an application having multiple software components. In various embodiments, computing device 110 represents a device that hosts application 200 (i.e., application 200 is installed on computing device 110) and receives updates for application 200. In one example, according to various embodiments of the present invention, computing device 110 hosts an instance of application 200 that is updated and tested (e.g., not shown before being deployed to other devices). In one example scenario, computing device 110 can be a device that hosts application 200 and is a component of a software change management system. In another example scenario, computing device 110 can be a device that hosts application 200 and is a component of a continuous delivery system.

[0023] Figure 2A Application 200 is depicted according to various embodiments of the present invention, which represents an application having multiple software components. In Figure 2A the example shown, application 200 includes data input 202, software component 204, software component 206, software component 208, software component 210, output 212, software component 214, software component 216, and output 218. In various embodiments, the software components of application 200 represent processes, functions, steps, modules, etc. that make up application 200. According to various embodiments of the present invention, the software components of application 200 are updated (e.g., as part of a continuous delivery process), and test management system 120 operates to verify the updates.

[0024] Additionally, Figure 2A the example depiction of application 200 in Figure 2A includes test bots 220, 221, 222, 223, 224, 225, 226, and 227. In various embodiments, the test bots of application 200 capture the inputs and outputs of the corresponding software components of application 200. In an example embodiment, a test bot is a component of a test integration system (e.g., a test integration system associated with test management system 120) that is responsible for interfacing with application components. In additional embodiments, a test bot can also operate to execute test segments using the captured data (e.g., according to the operations of test management system 120 and segment test program 300). In other embodiments, based on the specific operations of application 200 and test management system 120, application 200 can include more or fewer test bots than

[0025] In an example embodiment, the test management system 120 can be a desktop computer, a computer server, or any other computer system known in the art. In certain embodiments, the test management system 120 represents a computer system that utilizes clustered computers and components (e.g., database server computers, application server computers, etc.), which act as a single seamless resource pool when accessed by units of the data processing environment 100 (e.g., the computing device 110 and other devices not shown). Generally, the test management system 120 represents any electronic device or combination of electronic devices capable of executing computer-readable program instructions. According to an embodiment of the present invention, the test management system 120 may include components Figure 4 depicted and described in more detail.

[0026] The test management system 120 includes a fragment test program 300 and a storage device 122, and the storage device 122 includes a test data set 124 and a complete end-to-end test 126. In various embodiments of the present invention, the test management system 120 operates as a computing system that provides application test services (for one or more deployments and / or enterprises) according to an embodiment of the present invention. In an example embodiment, the test management system 120 represents a computing device (e.g., one or more management servers) that can provide a software integration test framework that works with a software change management and continuous delivery system to automatically detect, test, and verify changes to software components (e.g., applications such as the application 200 on the computing device 110), thereby reducing the need to repeatedly run complete end-to-end integration tests.

[0027] In an example embodiment, according to various embodiments of the present invention, the fragment test program 300 performs tests on a portion of an application. The fragment test program 300 can determine that a component of the application (e.g., a software component of the application 200) has been changed, and in response, the fragment test program 300 can identify the components of the application to be tested and the corresponding configurations for testing the changed components. In an example embodiment, according to various embodiments of the present invention, the fragment test program 300 can operate to test the changed software component without having to run all the previous components of the application, thereby reducing processing resources and execution time / complexity.

[0028] The storage device 122 can be implemented with any type of storage device, e.g., a permanent storage device 405 capable of storing data that can be accessed and utilized by the test management system 120, such as a database server, a hard disk drive, or a flash memory. In other embodiments, the storage device 122 can represent multiple storage devices and data sets within the test management system 120. In various embodiments, the test management system 120 can utilize the storage device 122 to store data associated with testing a software product (such as the application 200 of the computing device 110).

[0029] According to an embodiment of the present invention, the test data set 124 represents a data set used by the test management system 120 to test the application 200. In various embodiments, the test data set 124 includes defined and known data sets (e.g., sample data inputs to the application), and the test management system 120 (and the fragment test program 200) utilizes this data set to test the application 200. In one example, the test data set 124 is input into Figure 2A the data input 202 depicted in the example of the application 200 in. In an example embodiment, according to an embodiment of the present invention, the test management system 120 utilizes the test data set 124 to generate a complete end-to-end test 126 corresponding to the application 200 for use with the fragment test program 300.

[0030] The complete end-to-end test 126 represents the result of a complete and verified test of the application utilizing the test data set 124 (i.e., a complete and verified end-to-end data set). In various embodiments, the execution of the fragment test program 200 by the test management system 120 utilizes the complete end-to-end test 126 (which is the result of an initial end-to-end test of the application 200 utilizing a known data set, e.g., the test data set 124) to capture all inputs and outputs between the software components (e.g., Figure 2A the software components depicted in ) that make up the application 200. In additional embodiments, when generating the complete end-to-end test 126, the test management system 120 utilizes a fragment test robot (i.e., Figure 2A the test robot depicted in ) to capture the inputs and outputs from the software components of the application 200.

[0031] During a test run for establishing the complete end-to-end test 126, the test management system 120 feeds a known data set (e.g., the test data set 124) into the application 200. As the fed data flows through the application 200 (i.e., the process utilizing the corresponding software components), the test robot captures the input and output data and captures information corresponding to each respective component of the application 200. The test management system 120 can then store the captured input / output data and information as the complete end-to-end test 126 corresponding to the application 200 in the storage device 122. In various embodiments, according to various embodiments of the present invention, the test management system 120 (and the fragment test program 200) is capable of utilizing the data of the complete end-to-end test 126 and the test robot of the application 200 to test fragments or parts (e.g., individual software components or modules).

[0032] In some embodiments, the test framework of the test management system 120 can be integrated with a software change management system to record changes to software components of an application during the software development process. For example, in accordance with various embodiments of the present invention, in response to detecting a change, the test framework of the test management system 120 can store details of the change for test verification.

[0033] In an example embodiment, the test framework of the test management system 120 can be integrated with a software change management system using a test fragment plug-in (not shown). For example, a plug-in on the computing device 110 that notifies the test management system 120 of changes to the application 200. In an example scenario, when a change to the application 200 (e.g., a source file) is committed to the software change management system, the test fragment plug-in detects the change and sends a component change packet and a corresponding timestamp indicating when the change was made to the test management system 120. In additional embodiments, in accordance with embodiments of the present invention, the test management system 120 can map the received change indication to an integration test suite (e.g., and store it in the storage device 122) for testing the changed components of the application (using the fragment test program 300).

[0034] In other embodiments, the test framework of the test management system 120 can be integrated with a continuous delivery system to run integration test fragments on software components of an application that have changed since the previous build of the application. In an example embodiment, the test fragment is a subset of a complete integration test suite (e.g., a complete end-to-end test 126) for testing one or more software components of the application.

[0035] In an exemplary embodiment, the test framework of the test management system 120 can be integrated with a continuous delivery system to run one or more integration test suites when the continuous delivery system builds and deploys an application. For example, the test framework of the test management system 120 can be integrated with the continuous delivery system using a test fragment plug-in (as described above). In other embodiments, when an application is built and deployed, the continuous delivery system (and the test management system 120) can use the test fragment plug-in to facilitate the execution of tests on the application. For example, in accordance with embodiments of the present invention, the test management system 120 can interact with the test fragment plug-in to determine / provide the required test suite configuration and instantiate a test fragment robot to test the application.

[0036] In another embodiment, the test framework of the test management system 120 can be manual or semi-manual, with a software change management system augmented by a software integration test framework to allow testing of an application to skip one or more steps (e.g., software components) when performing an end-to-end test of the application.

[0037] Figure 2AIllustrates application 200 according to various embodiments of the present invention, which represents an application including multiple software components. In Figure 2A the illustrated example of, application 200 includes data input 202, software component 204, software component 206, software component 208, software component 210, output 212, software component 214, software component 216, and output 218. In various embodiments, the software components of application 200 represent the processes, functions, steps, modules, etc. that make up application 200. Additionally, Figure 2A the example of application 200 in depicts including test robots 220, test robots 221, test robots 222, test robots 223, test robots 224, test robots 225, test robots 226, and test robots 227. In various embodiments, the test robots of application 200 capture the inputs and outputs of the corresponding software components of application 200.

[0038] Figure 2B Illustrates application 250 according to various embodiments of the present invention, which represents an application including multiple software components. In an example embodiment, application 250 represents application 200 with at least one change or update to the software components that make up application 200 (e.g., a change to software component 210) ( Figure 2A depicted in). In Figure 2A the illustrated example of, application 250 includes data input 202, software component 204, software component 206, software component 208, software component 210, output 212, software component 214, software component 216, and output 218. In various embodiments, the software components of application 250 represent the processes, functions, steps, modules, etc. that make up application 250. Additionally, Figure 2B the example description of application 250 in includes test robots 223 and test robots 224. In various embodiments, according to an example embodiment of the present invention, the test robots of application 250 capture the inputs and outputs of the corresponding software components of application 250, and the fragment test program 300 can utilize the inputs and outputs to test software component 210.

[0039] Figure 3FIG. 0 is a flowchart showing the operation steps of a fragment test program 300 (a program for performing tests on a part of an application) according to an embodiment of the present invention. In one embodiment, the fragment test program 300 starts and operates to test an application (e.g., application 200) or a software product. In an example embodiment, the fragment test program 300 starts in response to identifying or receiving an indication of a change to one or more components of the application. In various embodiments, the fragment test program 300 operates to test a part (fragment) of the application. Thus, the fragment test program 300 can operate to automatically detect, test, and verify changes to software components, thereby reducing the need to repeatedly run full (possibly long and resource-intensive) end-to-end integration tests for the application (e.g., application 200).

[0040] In step 302, the fragment test program 300 identifies a test configuration. In one embodiment, the fragment test program 300 extracts a test configuration that includes an indication of one or more components of the application 200 to be tested (e.g., software components that have changed since a previous test). In an example embodiment, a change management system can track changes to the components of the application 200 and aggregate the changes into a test suite or test configuration. Once the test suite or test configuration is generated and stored, the fragment test program 300 can operate to execute one or more test fragments on the application 200 without performing a full end-to-end integration test. In various embodiments, the test configuration can include a configuration to run all test fragments, a subset of all test fragments, or only the test fragments corresponding to the software components of the application 200 that have changed since the most recent test of the application 200.

[0041] In an example scenario, a test management system 120 (e.g., from a plug-in on a computing device 110) receives an indication that a change has been made to a software component 210 of the application 200. Thus, the test management system 120 generates a test suite that includes a test configuration for testing the software component 210 of the application 200. In this example, in response to starting to test the application 200, the fragment test program 300 identifies the test configuration that includes an indication of the software component 210 of the application 200 to be tested. In various embodiments, the fragment test program 300 can advantageously operate to test the changed part of the application 200, thus reducing the resource usage of the test management system 120 by not running a full end-to-end test of the application 200.

[0042] In step 304, the fragment test program 300 identifies the components to be tested. In one embodiment, the fragment test program 300 parses the identified test configuration (from step 302) to identify the components of the application 200 to be tested. In another embodiment, the fragment test program 300 may identify multiple components of the application 200 to be tested based on the test suite of the test configuration (i.e., corresponding to the amount of change to the application 200). The fragment test program 300 may be executed to concurrently test multiple components, or separate instances of the fragment test program 300 may be executed to test multiple components (e.g., based on client / customer specifications).

[0043] In the example scenario discussed previously, the fragment test program 300 identifies a test configuration (in step 302) that includes an indication of the software component 210 of the test application 200. Thus, the fragment test program 300 identifies that the software component 210 is a component of the application 200 to be tested. In another embodiment, the fragment test program 300 may be operable to automatically analyze the application 200 (e.g., at automatic intervals, in response to receiving a request, etc.) to determine when changes are made to the application 200. Then, in accordance with an embodiment of the present invention, the fragment test program 300 may identify changes to components (e.g., to the software component 210), and determine the changed software components of the test application 200.

[0044] In step 306, the fragment test program 300 initiates a fragment test of the identified components. In one embodiment, the fragment test program 300 initiates a fragment test of the identified components of the application 200 (from step 304). In an example embodiment, the fragment test program 300 instantiates and initiates a fragment test robot for the component. The fragment test program 300 may utilize the fragment test robot to send appropriate input data (from the full end-to-end test 126) into the identified component. In another example embodiment, the fragment test program 300 may directly interact with the application 200 to test the identified changed component.

[0045] In another embodiment, the fragment test program 300 identifies data in the full end-to-end test 126 that corresponds to the input to the identified component to be tested. In various embodiments, the full end-to-end test 126 includes verified inputs and outputs corresponding to the components of the application 200 (utilizing the test data set 124). Thus, the fragment test program 300 identifies data (e.g., metadata, information, etc.) in the full end-to-end test 126 that corresponds to the input of the identified component of the application 200.

[0046] In the example scenario discussed previously, the fragment test program 300 initiates a fragment test of the software component 210 of the application 200. In this example scenario, the fragment test program 300 identifies the data corresponding to the input to the software component 210 in the complete end-to-end test 126. As Figure 2B depicted in the example of

[0047]

[0048] Figure 2B In the example scenario discussed previously, the fragment test program 300 receives the data output from the software component 210. As Figure 2B depicted in the example of

[0049] In step 310, the fragment test program 300 determines the verification result of the test fragment. In one embodiment, the fragment test program 300 verifies the output (from step 308) received from the identified component (from step 304) of the test application 200. In various embodiments, the fragment test program 300 determines whether the received result for the test fragment executed on the component of the application 200 matches the expected (i.e., verified) output result (which corresponds to the output from the software component 210) included in the complete end-to-end test 126. In an example embodiment, the fragment test program 300 may determine whether the output data corresponds to (e.g., matches within a defined threshold) the corresponding output result included in the complete end-to-end test 126. Thus, the fragment test program 300 can determine and generate a test / visual indication of the verification result (e.g., "yes" or "no", "pass" or "fail", etc.).

[0050] In another embodiment, the fragment test program 300 may utilize a test fragment robot to receive the output and perform the verification result. In Figure 2BIn the example of, the fragment test program 300 can utilize the test robot 224 to receive the data output from the software component 210 of the execution (e.g., test) application 200, and perform verification of the received output. In this example, the fragment test program 300 can receive an indication of the verification result from the test robot 224.

[0051] In step 312, the fragment test program 300 outputs the verification result. In one embodiment, the fragment test program 300 sends the determined verification result (from step 310) to one or more users associated with the application 200, or to the associated test of the application 200. In another embodiment, the fragment test program 300 can store the determined verification result of the fragment test in the storage device 122 associated with the application 200.

[0052] In additional embodiments, the test framework of the test management system 120 can be configured to handle and process the expected changes to the output caused by one or more code changes. In an exemplary scenario, the test management system 120 can verify the changes made to the application 200 that change the input and output values included in the full end-to-end test 126. For example, a user associated with the application 200 can receive the verification result and authorize the test management system 120 to update the full end-to-end test 126 to correspond to the updated instance of the application 200. In this case, the test management system 120 can run a subset of the full end-to-end integration test on the application 200, starting from the first changed component, and capture all input and output data throughout the remaining test flow. Accordingly, the test management system 120 can update the instance of the full end-to-end test 126 with the newly captured (and verified data) for use in future operations of the fragment test of the application 200 (i.e., the execution of the fragment test program 300).

[0053] Figure 4 FIG. shows a computer system 400 according to an exemplary embodiment of the present invention, which represents the computing device 110 and the test management system 120. It should be understood that Figure 4Only an implementation is provided, without implying any limitation to the environment in which different embodiments can be implemented. Many modifications can be made to the described environment. Computer system 400 includes a processor 401, a cache 403, a memory 402, a permanent storage device 405, a communication unit 407, one or more input / output (I / O) interfaces 406, and a communication fabric 404. The communication fabric 404 provides communication between the cache 403, the memory 402, the permanent storage device 405, the communication unit 407, and the input / output (I / O) interfaces 406. The communication fabric 404 can be implemented with any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripherals, and any other hardware components within the system. For example, the communication fabric 404 can be implemented with one or more buses or crossbar switches.

[0054] The memory 402 and the permanent storage device 405 are computer-readable storage media. In this embodiment, the memory 402 includes random access memory (RAM). Generally, the memory 402 can include any suitable volatile or non-volatile computer-readable storage media. The cache 403 is a fast memory that enhances the performance of the processor 401 by storing recently accessed data and data near the recently accessed data.

[0055] The program instructions and data (e.g., software and data 410) for practicing the embodiments of the present invention can be stored in the permanent storage device 405 and the memory 402 for execution by one or more corresponding processors 401 via the cache 403. In one embodiment, the permanent storage device 405 includes a magnetic hard disk drive. As an alternative or supplement to the magnetic hard disk drive, the permanent storage 405 can include a solid state drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0056] The medium used by the permanent storage device 405 can also be removable. For example, a removable hard disk drive can be used for the permanent storage device 405. Other examples include optical discs and disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of the permanent storage device 405. Software and data 410 can be stored in the permanent storage device 405 for access and / or execution by one or more corresponding processors 401 via the cache 403. With respect to the computing device 110, the software and data 310 include the application 200. With respect to the test management system 120, the software and data 410 include the test data set 124, the complete end-to-end test 126, and the fragment test program 300.

[0057] In these examples, the communication unit 407 provides communication with other data processing systems or devices. In these examples, the communication unit 407 includes one or more network interface cards. The communication unit 407 can provide communication by using one or both of physical and wireless communication links. Program instructions and data (e.g., software and data 410) for implementing embodiments of the present invention can be downloaded to the permanent storage device 405 via the communication unit 407.

[0058] The I / O interface 406 allows input and output of data with other devices that can be connected to each computer system. For example, the I / O interface 406 can provide a connection to external devices 408 such as a keyboard, keypad, touch screen, and / or some other suitable input device. The external device 408 can also include a portable computer-readable storage medium, such as a thumb drive, portable optical disc or disk, and memory card. Program instructions and data (e.g., software and data 410) for practicing embodiments of the present invention can be stored on such a portable computer-readable storage medium and can be loaded onto the permanent storage device 405 via the I / O interface 406. The I / O interface(s) 406 is also connected to the display 409.

[0059] The display 409 provides a mechanism for displaying data to a user and can be, for example, a computer monitor.

[0060] The programs described herein are identified based on the applications in which they are implemented in particular embodiments of the present invention. However, it should be understood that any specific program terms herein are used for convenience only, and thus the present invention should not be limited to use in any particular application identified and / or implied by such terms.

[0061] The present invention can be a system, method, and / or computer program product at any possible level of integration of technical details. The computer program product can include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0062] A computer-readable storage medium can be a tangible device that is capable of storing and retaining instructions for use by an instruction execution device. A computer-readable storage medium can be, by way of example and not limitation, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punched card or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed to be a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0064] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, in order to perform aspects of the present invention, an electronic circuit, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0065] Aspects of the present invention are described herein with reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0066] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions executed via the processor of the computer or other programmable data processing apparatus create a means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct a computer, a programmable data processing apparatus, and / or other devices to work in a particular manner, such that the computer-readable storage medium in which the instructions are stored comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0067] The computer-readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0068] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be implemented as one step, executed simultaneously, substantially simultaneously, partially or wholly in time overlapping fashion, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0069] The description of the various embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The terms used herein were chosen to best explain the principles of the embodiments, the practical application, or technical improvement over technologies found in the marketplace, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: identifying, by one or more processors, an integration test configuration for testing an application, the application including a plurality of components, wherein the integration test configuration includes an indication to test at least one component of the application; receiving, by one or more processors, a data output from the indicated at least one component of the application; testing, by one or more processors, the indicated at least one component of the application, wherein the testing is at least partially based on a verified data output associated with a verified end-to-end integration test data set corresponding to the application program and the data output; and determining, by one or more processors, a verification result of testing the indicated at least one component of the application; and updating, by one or more processors, the verified data output using the verification result of the indicated at least one component of the application, wherein the updated verified data output is added to the complete verified end-to-end integration test data set, and wherein the complete verified end-to-end integration test data set is used to test at least one new component added to the application program after the at least one component.

2. The method according to claim 1, wherein, Identifying an integration test configuration for testing an application further includes: determining, by one or more processors, at least one component of the application that has changed since the previous integration test of the application; and generating, by one or more processors, the integration test configuration for testing the application to include a test suite for the application, the test suite including the indication to test the at least one component of the application.

3. The method according to claim 1, further comprising: running, by one or more processors, a complete end-to-end integration test on the application, wherein running the complete end-to-end integration test further includes: feeding, by one or more processors, an integration test data set into the application; capturing, by one or more processors, corresponding inputs and outputs of each component of the application; and storing, by one or more processors, the captured corresponding inputs and outputs of each component of the application as the verified end-to-end integration test data set.

4. The method according to claim 1, wherein Testing the indicated at least one component of the application further includes: identifying, by one or more processors, a verified data input from the verified end-to-end integration test data set corresponding to the application; and feeding, by one or more processors, the verified data input into the indicated at least one component of the application.

5. The method according to claim 1, wherein, Testing the indicated at least one component of the application further includes: comparing, by one or more processors, the received data output with the verified data output from the verified end-to-end integration test data set corresponding to the application.

6. The method according to claim 1, further comprising: sending, by one or more processors, the determined verification result to a user associated with the application, the determined verification result indicating whether the indicated at least one component of the application passes the test verification.

7. The method according to claim 2, wherein Determining the at least one component of the application that has changed since the previous test of the application further includes: Receiving, by one or more processors, an indication of a change to the application from a software change management plug-in associated with the application.

8. A computer program product comprising: One or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions executable to perform the steps of the method according to any one of claims 1 to 7.

9. A computer system comprising: One or more computer processors; One or more computer-readable storage media; And Program instructions stored on the computer-readable storage media to be executed by at least one of the one or more processors, the program instructions executable to perform the steps of the method according to any one of claims 1 to 7.

Images