Computer-implemented method, system, and computer program product for code coverage for a test system for testing source code (Code coverage measurement for a test system)

Abstract

To provide a computer-implemented method, system and computer program product for code coverage measurement for test systems.SOLUTION: The invention provides a method and system for code coverage for test systems for testing a source code. The method comprises: obtaining a trace with trace records written at an entry and an exit to one or more test modules for function calls to the test modules; comparing response codes for functions extracted from the trace with a finite list of response codes retrieved from metadata for the source code for the functions; and analyzing a code coverage based on the comparison.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to code coverage measurement, and more particularly to code coverage measurement for test systems. [Background technology]

[0002] Test completeness is traditionally measured using code coverage, a measure used to describe how much of a program's source code is exercised when a particular test suite is run.

[0003] A program with high test coverage has a high amount of source code executed during testing. A program with high test coverage is less likely to contain undetected software bugs than a program with low test coverage. Many different metrics are used to calculate test coverage; some of the most basic are the percentage of a program's subroutines and the percentage of program statements that are called during the execution of a test suite.

[0004] Code coverage is typically achieved through low-level instrumentation, such as machine instructions, which are reported at the module level and then aggregated into total values. Existing solutions focus on whether a code path was executed or not, or whether it was executed for certain external input criteria.

[0005] In reality, the problem is more complex because execution depends on values ​​passed to various modules. These can result in a large number of error flows that are not captured by code coverage metrics. For example, if a module performs various input validations and then branches to code that sets an invalid return code and returns for each of these checks, the code involved in setting the return code and returning may be very small. Therefore, even if a module has very high coverage, it may not actually test many of the possible error flows. This needs to be determined by looking at line-by-line coverage data. Summary of the Invention [Problem to be solved by the invention]

[0006] A computer-implemented method, system, and computer program product for code coverage measurement for a test system are provided. [Means for solving the problem]

[0007] According to one aspect of the present invention, there is provided a computer-implemented method for code coverage for a test system for testing source code, the method including obtaining trace records written at the entry and exit to one or more test modules for function calls to the test modules, comparing response codes of the functions extracted from the trace with a finite list of response codes retrieved from metadata of the source code of the functions, and analyzing code coverage based on the comparison.

[0008] According to another aspect of the present invention, there is provided a system for code coverage for a test system for testing source code, the system including: a processor; and a memory configured to provide computer program instructions for causing the processor to perform functions of components including: a trace acquisition component for acquiring trace records written at entry and exit to one or more test modules for function calls to the test modules; a metadata comparison component for comparing response codes of the functions extracted from the trace with a finite list of response codes retrieved from metadata of the source code for the functions; and a code coverage analysis component for analyzing code coverage based on the comparison.

[0009] According to a further aspect of the present invention, there is provided a computer program product for code coverage for a test system for testing source code, the computer program product including a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to obtain trace records written at entry and exit to one or more test modules for function calls to the test modules; compare response codes of the functions extracted from the trace with a finite list of response codes retrieved from metadata of the source code for the functions; and analyze code coverage based on the comparison.

[0010] The computer-readable storage medium may be a non-transitory computer-readable storage medium, and the computer-readable program code may be executable by a processing circuit.

[0011] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of this specification. The invention, both as to organization and method of operation, together with its objects, features and advantages, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings.

[0012] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings: [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic diagram of a method according to the present invention. [Figure 2] FIG. 1 is a flow diagram illustrating an example of an embodiment of a method according to the present invention. [Figure 3] 1 is a block diagram illustrating an example of an embodiment of a system according to the present invention. [Figure 4] FIG. 1 is a block diagram of one embodiment of a computer system or cloud server in which the present invention may be implemented. [Figure 5] 1 is a schematic diagram of a cloud computing environment in which the present invention may be implemented; [Figure 6] FIG. 1 is a diagram of abstraction model layers of a cloud computing environment in which the present invention may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0014] It will be understood that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous features.

[0015] A method and system are provided for analyzing code coverage under test using trace data that is based on input and output combinations rather than code paths.

[0016] Source code may be tested by a test module, whose inputs and outputs are traced. The inputs may include the function by which the test module was called and other possible inputs. Other possible inputs may include numerical values ​​required by the function, such as how much storage to allocate, or other input parameters required by the function. The outputs may include a response code indicating either that the call was successful or the reason the call failed. Because each test module may call other test modules, the traces have a form in which each input trace is linked to an output trace.

[0017] The term "function" is used to refer to any function or method that may be invoked to execute, including, for example, an application programming interface (API) call, a Java method (Java is a trademark of Oracle Corporation), or an IBM Customer Information Control System (CICS) function (IBM CICS is a trademark of International Business Machines Corporation).

[0018] For the function being entered, metadata is searched for listing its inputs and outputs, including the values ​​of the function code and response code data items, which are a finite set. The metadata may, for example, be a documented list of functions in the source code, or it may be obtained by scanning the source code. The inputs are required in addition to the outputs, since they can affect the code path of execution within the program.

[0019] The described method and system compares trace response data with metadata of functions executed by a test module to determine the coverage achieved by the test module.

[0020] Referring to Figure 1, a schematic diagram 100 illustrates the described method and system. As part of a test system, input 110 is made to a test module 120 in the form of function input 111 and other input 112 (e.g., function parameters or values), and output 130 is returned in the form of response codes, such as return code 131 and reason code 132. A test module 120 may be an executable module for testing source code invocations. A test module 120 may call other test modules 120 as part of its execution, and output 130 may be from the resulting series of test modules 120.

[0021] Metadata codes 140 are provided as a finite list of all response codes, such as in the form of function return and reason codes. Return codes 131 can indicate that the call was successful, and reason codes 132 can provide the reason for the call's failure.

[0022] A code coverage system 150 described herein receives trace data 151 of inputs 110 and outputs 130 for functions executed by one or more test modules 120. The code coverage system 150 also retrieves metadata 152 associated with the inputs 110 and outputs 130 in the form of a finite list of possible response codes for the functions executed by the test module(s) 120. The code coverage system 150 analyzes coverage 153 of the test module(s) 120 by comparing the trace data 151 with the retrieved metadata code 152.

[0023] Referring to FIG. 2, a flow diagram 200 illustrates an exemplary embodiment of the described method for analyzing code coverage of source code in a test system.

[0024] The method may, at 201, obtain trace records written at the entry and exit to one or more test modules for function calls to the test modules. This may include running the test to generate a trace with trace records written at the entry and exit for functions being executed. As the trace is run, it captures all input and output function calls for each test module. This may include collecting module input traces for functions called by the test module and collecting test module output traces including function response codes. The response codes may include success return codes and reason codes for unsuccessful calls.

[0025] At 202, the entry and exit trace records are paired to generate a single record. The inputs to the function are provided by the entry trace record, and the outputs from the function are provided by the exit trace record, and the pairing of the entry and exit trace records provides information about the code path that was executed.

[0026] At 203, response codes may be extracted for executed functions and sorted, including removing duplicates. Also, input and output traces of each module may be used to create a record of the list of detected response codes for each function.

[0027] The method may retrieve metadata codes for all response codes for each function in each test module executed in the test, at 204. The method may compare the trace record records with the metadata codes for the functions to determine coverage, at 205.

[0028] The coverage of the functions is analyzed at 206. At 207, the code is grouped into levels and comparison data from each level may be aggregated to a higher level.

[0029] In one embodiment, looking at a specific function, and taking the simplest case of a function call and response code combination, the levels may be aggregated as follows: Function coverage may be determined as the percentage of possible response codes that appear in the trace compared to the metadata of a finite list of responses the function can return. Module coverage can be determined in the same way, but looking at all possible functions and all possible response codes for each function. Additionally, overall code coverage can be determined in the same way, but looking at all possible modules along with all possible functions and all possible response codes.

[0030] In another embodiment, the code may be grouped into multiple levels, for example, classes, methods, functions, etc. Data from each level may be aggregated to a higher level.

[0031] At 208, missing code paths (i.e., functions or response codes) that are listed in the definitive list of metadata codes but not recorded in the trace code may be analyzed to provide feedback regarding missing tests that may be required to be performed.

[0032] At a basic level, this method can look at functions and the response codes from them. For example, the function can be READ or WRITE, and the response can be OK, NOTFOUND, NOTAUTH, etc. This may be expanded to include input and output values ​​for the function and response code. For example, the input and output values ​​may include options for a READ, such as EXCLUSIVE with an option YES or NO, indicating that the READ should gain exclusive access to something.

[0033] To illustrate this method, an example is provided. A trace record is written at the entry of a function, which may be a function, method, or API call. A trace record is written at the exit of the function and a response code, which may be in the form of one or more of a return code, reason code, or exception. Metadata is searched that lists all functions that have a response code. A function may also describe its location within a higher level hierarchy in the source code.

[0034] The following general terms are used in the trace records: Entry: Function Exit: Function, Response Code

[0035] In the example of an application programming interface (API), the metadata might be a list of documented functions, and the trace records might be in the form: Entrance:API name Exit: API name, return code

[0036] In the IBM CICS example, the metadata is the API description CDURUN and the trace records may be in the following format: Entrance: Domains, Gates, Functions Exit: Domain, Gate, Function, Return Code, Response Code

[0037] In a Java example, the metadata could be Javadoc or code scans, and the trace records could be in the form: Entrance:API name Exit: API name, return code

[0038] IBM CICS terminology is used in the following examples: the hierarchy of levels in the source code is given by Domain / Gate / Function, and the metadata is the list of CDURUNs and their contents.

[0039] In this example, the gate "Business Application Container" (BACR) of the domain "Business Applications" (BA) is tested.

[0040] An example of trace data can be obtained as follows: BA / BACR / DELETE_CONTAINER / OK BA / BACR / DELETE_CONTAINER / CONTAINER_NOT_FOUND BA / BACR / GET_CONTAINER_INTO / OK BA / BACR / GET_CONTAINER_SET / OK BA / BACR / PUT_CONTAINER / OK BA / BACR / PUT_CONTAINER / INVALID_CONTAINER_NAME BA / BACR / COPY_CONTAINER / OK

[0041] The finite list of metadata codes for functions in gate BACR is as follows: BA / BACR / DELETE_CONTAINER / OK BA / BACR / DELETE_CONTAINER / CONTAINER_NOT_FOUND BA / BACR / DELETE_CONTAINER / CONTAINER_READONLY BA / BACR / DELETE_CONTAINER / INVALID_CONTAINER_NAME BA / BACR / GET_CONTAINER_INTO / OK BA / BACR / GET_CONTAINER_INTO / CONTAINER_NOT_FOUND BA / BACR / GET_CONTAINER_SET / OK BA / BACR / GET_CONTAINER_SET / CONTAINER_NOT_FOUND BA / BACR / PUT_CONTAINER / OK BA / BACR / PUT_CONTAINER / INVALID_CONTAINER_NAME BA / BACR / GET_CONTAINER_LENGTH / OK BA / BACR / GET_CONTAINER_LENGTH / CONTAINER_NOT_FOUND BA / BACR / MOVE_CONTAINER / OK BA / BACR / MOVE_CONTAINER / CONTAINER_NOT_FOUND BA / BACR / MOVE_CONTAINER / CONTAINER_READONLY BA / BACR / COPY_CONTAINER / OK BA / BACR / COPY_CONTAINER / CONTAINER_NOT_FOUND BA / BACR / COPY_CONTAINER / CONTAINER_READONLY

[0042] Comparing the trace data with the metadata will result in the following, with the trace code underlined: TIFF0007807355000001.tif111129

[0043] So the coverage results at the function level are: BA / BACR / DELETE_CONTAINER 2 out of 4 covered 50% BA / BACR / GET_CONTAINER_INTO 1 of 2 covered 50% BA / BACR / GET_CONTAINER_SET 1 of 2 covered 50% BA / BACR / PUT_CONTAINER 2 out of 2 covered 100% BA / BACR / GET_CONTAINER_LENGTH 0 out of 2 covered 0% BA / BACR / MOVE_CONTAINER 0 out of 3 covered 0% BA / BACR / COPY_CONTAINER 1 out of 3 covered 33%

[0044] Coverage results may be aggregated at the method level, in which case the coverage for gate BACR would be: 7 out of 18 BA / BACR covered 39%

[0045] Coverage results may be aggregated at the class level, in which case the coverage for domain BA would be: 27 of 96 BAs covered 28%

[0046] Coverage results may be aggregated at the program level, in which case the coverage for CICS code would look like this: 564 of CICS 7812 covered 7%

[0047] The method and system uses trace data to collect code coverage information. The trace data provides the actual path traversed, not just the appearance, which has the advantage of not requiring tools. Because code coverage is determined from the trace, no instrumentation is required.

[0048] Code coverage, as used here, is a measure of which functions are executed and what outputs they may return, regardless of the length of those paths - this is more relevant to the complexity of the code and the tests.

[0049] 3, a block diagram illustrates an exemplary embodiment of the described code coverage system 150 provided on a computing system 300. The computing system 300 or a separate computing system may also include a test system 310 and a trace system 320 that run in conjunction with or are incorporated into the code coverage system 150.

[0050] The computing system 300 includes at least one processor 301, circuitry for performing the functions of the described components, which may be a hardware module or a software unit running on at least one processor. Multiple processors executing parallel processing threads may be provided, allowing for parallel processing of some or all of the functions of the components. The memory 302 may be configured to provide computer instructions 303 to the at least one processor 301 to cause the functions of the components to be performed.

[0051] The code coverage system 150 may include a trace acquisition component 331 for acquiring traces having trace records written at the entry and exit to one or more test modules for function calls to the test modules. The trace acquisition component 331 may acquire trace records where a test module calls another test module as part of executing the function call.

[0052] The trace acquisition component 331 may include a trace processing component 332 for pairing entry and exit trace records to generate trace records, sorting the trace records, and removing duplicate trace records.

[0053] The code coverage system 150 may include a metadata retrieval component 333 for retrieving metadata of the source code of the functions, and a metadata comparison component 334 for comparing the response codes of the functions extracted from the trace with a finite list of response codes retrieved from the metadata of the source code of the functions. The metadata comparison component 334 may provide a ratio of the metadata response codes provided in the trace record.

[0054] The code coverage system 150 may include a code coverage analysis component 335 for analyzing code coverage based on the comparison results of the metadata comparison component 334. The code coverage analysis component 335 may include a level aggregation component 336 for grouping functions into higher levels and aggregating the code coverage of each level into a higher-level analysis. A higher level may be a module that aggregates all possible functions and all possible response codes for each function. An even higher level may be the total source code that aggregates all possible modules. Alternatively, a higher level may be a higher level in the function hierarchy.

[0055] The code coverage system 150 may include a missing test analysis component 337 for analyzing response codes for missing metadata from trace records to provide feedback related to required tests.

[0056] 4 illustrates a block diagram of components of a computing system 300 used in the code coverage system 150, in accordance with one embodiment of the present invention. It should be understood that FIG. 4 is provided only as an illustration of one embodiment and is not intended to imply limitation with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

[0057] The computing system may include one or more processors 402, one or more computer-readable RAMs 404, one or more computer-readable ROMs 406, one or more computer-readable storage media 408, device drivers 412, read / write drives or interfaces 414, and network adapters or interfaces 416, all interconnected via a communications fabric 418. The communications fabric 418 may be implemented in any architecture designed to pass data and / or control information between processors (such as microprocessors, communications processors, network processors), system memory, peripheral devices, and any other hardware components in the system.

[0058] One or more operating systems 410 and application programs 411, such as code coverage system 150, are stored on one or more computer-readable storage media 408 and executed by one or more processors 402 through one or more of the respective RAMs 404 (which typically include cache memory). In the illustrated embodiment, each of computer-readable storage media 408 may be an internal hard drive magnetic disk storage device, a CD-ROM, a DVD, a memory stick, magnetic tape, a magnetic disk, an optical disk, a semiconductor storage device such as RAM, ROM, EPROM, flash memory, or any other computer-readable storage medium capable of storing computer programs and digital information, in accordance with an embodiment of the present invention.

[0059] The computing system may also include a R / W drive or interface 414 for reading from and writing to one or more portable computer-readable storage media 426. Application programs 411 on the computing system are stored on one or more of the portable computer-readable storage media 426, read via the respective R / W drive or interface 414, and loaded into the respective computer-readable storage media 408.

[0060] The computing system may also include a network adapter or interface 416, such as a TCP / IP adapter card or a wireless communication adapter. Application programs 411 on the computing system may be downloaded to the computing device from an external computer or external storage device via a network (e.g., the Internet, a LAN, other WAN, or a wireless network) and the network adapter or interface 416. From the network adapter or interface 416, the programs may be loaded into the computer-readable storage medium 408. The network may include copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0061] The computing system may also include a display screen 420, a keyboard or keypad 422, and a computer mouse or touchpad 424. The device driver 412 interfaces to the display screen 420 for image formation, the keyboard or keypad 422, the computer mouse or touchpad 424, and / or the display screen 420 for alphanumeric character entry and pressure sensing of user selections. The device driver 412, the R / W drive or interface 414, and the network adapter or interface 416 may include hardware and software stored in the computer-readable storage medium 408 and / or the ROM 406.

[0062] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0063] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, and suitable combinations thereof. As used herein, a computer-readable storage device should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computer / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computer / processing device for storage.

[0065] The computer-readable program instructions for carrying out the operations of the present invention can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions can execute entirely on the user's computer as a stand-alone software package, or partially on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer readable program instructions by utilizing state information of the computer readable program instructions to customize the electronic circuitry for carrying out aspects of the present invention.

[0066] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] The computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, create means for performing the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner. The computer-readable storage medium having instructions stored thereon thereby constitutes an article of manufacture including instructions for performing aspects of the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0068] Also, computer-readable program instructions may be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0069] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing a specific logical function. In some implementations, the functions depicted in the blocks may be performed in an order different from that depicted in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, may be executed simultaneously or substantially simultaneously, may be executed in a partially or fully overlapping manner, or may even be executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0070] Cloud Computing

[0071] Although this disclosure includes detailed descriptions of cloud computing, implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the invention can be practiced in conjunction with any other type of computing environment now known or later developed.

[0072] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0073] The characteristics are as follows:

[0074] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider.

[0075] Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, PDAs).

[0076] Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers generally have no control or knowledge of the exact location of the resources they are provided with, so there is a sense of location independence. However, consumers may be able to identify their location at a higher level of abstraction (e.g., country, state, data center).

[0077] Rapid Elasticity: Computing capacity can be provisioned quickly and elastically, sometimes automatically, to instantly scale out and quickly released to instantly scale in. To the consumer, the computing power available to provision often appears unlimited, and can be purchased at any time and in any quantity.

[0078] Metered Services: Cloud systems leverage measurement capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user count) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of utilized services.

[0079] The service model is as follows:

[0080] Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure. The applications can be accessed from a variety of client devices through a thin-client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for limited user-specific application configuration settings.

[0081] Platform as a Service (PaaS): The capability offered to consumers is to deploy applications they create or acquire onto a cloud infrastructure using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of their hosting environment.

[0082] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provisioning of processors, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, and deployed applications, and in some cases partial control over some network components (e.g., host firewalls).

[0083] The deployment model is as follows:

[0084] Private Cloud: This cloud infrastructure is dedicated to a specific organization and can be managed by that organization or a third party, and can exist on-premise or off-premise.

[0085] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by those organizations or a third party and can exist on-premises or off-premises.

[0086] Public cloud: This cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0087] Hybrid cloud: This cloud infrastructure combines two or more cloud models (private, community, or public), each of which retains its inherent nuances but is bound by standards or specific technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0088] A cloud computing environment is a service-oriented environment that emphasizes statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0089] FIG. 5 illustrates an exemplary cloud computing environment 50. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers (e.g., PDAs or cell phones 54A, desktop computers 54B, laptop computers 54C, or automobile computer systems 54N, or combinations thereof) can communicate. The nodes 10 can communicate with each other. The nodes 10 can be physically or virtually grouped (not shown) in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software as a service, or combinations thereof, for which cloud consumers are not required to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N illustrated in FIG. 5 are merely exemplary, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network-addressable connection (e.g., using a web browser), or both.

[0090] Next, a set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 5) is shown in FIG. 6. It should be understood in advance that the components, layers, and functions shown in FIG. 6 are merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0091] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage device 65, and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0092] The virtualization layer 70 provides an abstraction layer from which virtual entities such as virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75 can be provided.

[0093] By way of example, the management layer 80 may provide the following functionality: Resource provisioning 81 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. By way of example, these resources may include application software licenses. Security enables identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 enables advance arrangement and procurement of anticipated future cloud computing resources required in accordance with SLAs.

[0094] The workload layer 90 provides examples of functionality available in a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and code coverage processing 96.

[0095] The computer program product of the present invention comprises one or more computer-readable hardware storage devices having computer-readable program code stored thereon, said program code being executable by one or more processors to perform the methods of the present invention.

[0096] The computer system of the present invention comprises one or more processors, one or more memories, and one or more computer-readable hardware storage devices, the one or more hardware storage devices containing program code executable by the one or more processors via the one or more memories, and is capable of implementing the method of the present invention.

[0097] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments described herein.

[0098] Improvements and modifications can be made to the foregoing without departing from the scope of the present invention.

Claims

1. 1. A computer-implemented method for code coverage for a test system for testing source code, comprising: obtaining trace records written at the entry and exit to one or more test modules for function calls to said test modules; comparing the response code of the function extracted from the trace record to a finite list of response codes retrieved from metadata of the source code of the function; analyzing code coverage based on the comparison; and A method comprising:

2. The method of claim 1 , wherein comparing response codes to metadata response codes provides a ratio of the metadata response codes provided in the trace records.

3. Analyzing the code coverage includes: grouping the functions into higher levels and aggregating the code coverage at each level into a higher level of analysis; The method of claim 1 , comprising:

4. The method of claim 3 , wherein the higher level is a module that aggregates all possible functions and all possible response codes for each function.

5. The method of claim 4 , wherein the higher level is the total source code that aggregates all possible modules.

6. The method of claim 3 , wherein the higher level is a higher level in a function hierarchy.

7. analyzing response codes of the metadata missing from the trace records to provide feedback related to required tests; The method of claim 1 , comprising:

8. pairing entry and exit trace records to generate trace records, sorting the trace records, and removing duplicate trace records; The method of claim 1 , comprising:

9. The method of claim 1 , wherein the response code comprises one or more of the group: a success return code, a failure reason code, and an exception.

10. The method of claim 1 , wherein a test module calls another test module as part of executing a function call.

11. 1. A system for code coverage for a test system for testing source code, comprising: a processor and a memory configured to provide computer program instructions for causing said processor to perform the functions of a component, said component comprising: a trace capture component for capturing trace records written at the entry and exit to one or more test modules for function calls to said test modules; a metadata comparison component for comparing the response codes of the function extracted from the trace records with a finite list of response codes retrieved from metadata of the source code of the function; a code coverage analysis component for analyzing code coverage based on the comparison; Including, the system.

12. The system of claim 11 , wherein the metadata comparison component that compares response codes to metadata response codes provides a ratio of the metadata response codes provided in the trace records.

13. The code coverage analysis component a level aggregation component for grouping the functions into higher levels and aggregating the code coverage at each level into a higher level of analysis; The system of claim 11 , comprising:

14. 14. The system of claim 13, wherein the higher level is a module that aggregates all possible functions and all possible response codes for each function.

15. 15. The system of claim 14, wherein the higher level is the total source code that aggregates all possible modules.

16. The system of claim 13 , wherein the higher level is a higher level of a function hierarchy.

17. a missing test analysis component for analyzing response codes of the metadata missing from the trace records to provide feedback related to required tests; The system of claim 11 , comprising:

18. a trace processing component for pairing entry and exit trace records to generate trace records, sorting the trace records, and removing duplicate trace records; The system of claim 11 , comprising:

19. The system of claim 11 , wherein the trace capture component captures trace records of test modules calling other test modules as part of executing function calls.

20. 1. A computer program for code coverage for a test system for testing source code, the computer program comprising: obtaining trace records written at the entry and exit to one or more test modules for function calls to said test modules; comparing the response code of the function extracted from the trace record to a finite list of response codes retrieved from metadata of the source code of the function; analyzing code coverage based on the comparison; and Execute Computer program.

Images