Automated generation of test coverage metrics for microservice systems

Abstract

Methods and systems for the automated generation of test coverage metrics for microservice systems are disclosed. An example method for generating test coverage metrics for a test suite comprising multiple tests for a microservice system containing multiple microservices includes generating a first set of endpoints associated with the multiple microservices based on a static analysis of the corresponding microservice source code and a second set of endpoints based on generating traces and capturing interactions during operation of the corresponding microservice. The method further includes integrating the first and second sets of endpoints to generate a combined set of endpoints, and determining, based thereon, the test coverage metrics for the multiple tests. The method then reconfigures, based on the test coverage metrics, at least one of the multiple tests or at least one of the multiple microservices. An example system includes one or more processors configured to implement the above-described method.

Description

AUTOMATED GENERATION OF TEST COVERAGE METRICS FOR MICROSERVICE SYSTEMSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This patent document claims priority to and benefits of U.S. Provisional Patent 63 / 612,462 entitled “AUTOMATED GENERATION OF END-TO-END TEST COVERAGE METRICS FOR MICROSERVICE SYSTEMS” and filed on December 20, 2024. The entire content of the before-mentioned patent application is incorporated by reference as part of the disclosure of this patent document.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

[0002] This invention was made with government support under Grant No. 2245287 awarded by the National Science Foundation. The government has certain rights in the invention.TECHNICAL FIELD

[0003] This patent document is generally related to microservice systems, and more particularly, to methods and systems for the automated generation of test coverage metrics for microservice systems.BACKGROUND

[0004] Cloud-native architecture, which leverages the framework of microservice systems and other cloud-based technologies, has transformed software development, providing engineers with tools for agile and scalable solutions. However, it also introduces challenges when dealing with system maintenance and quality assurance. End-to-end testing is commonly applied to verify all decentralized system components can run under real-world scenarios. However, it is challenging to assess how comprehensive the test coverage is in these systems.SUMMARY

[0005] Embodiments of the disclosed technology are related to comprehensive end-to-end testing and a holistic approach covering the execution paths of software applications to ensure their reliability and quality assurance. The described embodiments are directed to coverage of end-to-end testing, relevant test metrics forevaluation, and strategic automation to improve testing efficiency in the context of cloudnative systems.

[0006] In an example aspect, the described methods and systems are directed to multiple end-to-end testing coverage metrics relevant to microservices, and providing a process that combines static and dynamic analysis to derive such metrics from cloudnative systems and their respective test suites. The efficacy of the process is validated using a proof of concept tool tailored to the Java platform, and applied to a third-party microservice benchmark.

[0007] In another example aspect, a method for generating test coverage metrics for a test suite comprising multiple tests for a microservice system containing multiple microservices is described. In this example, the multiple microservices interact to perform an overall application function, with each microservice being associated with at least one endpoint and configured to perform a partial function of the overall application function, and the at least one endpoint of a corresponding microservice enabling a user or an alternative microservice to interact with the corresponding microservice. Herein, the method includes extracting at least one endpoint from each corresponding microservice of the multiple microservices, and determining, based on the at least one endpoint from each corresponding microservice, test coverage metrics. The method further includes reconfiguring, based on the test coverage metrics, at least one test of the multiple tests or at least one of the multiple microservices.

[0008] In yet another example aspect, a method for generating test coverage metrics for a test suite comprising multiple tests for a microservice system containing multiple microservices is described. The method includes generating a first set of endpoints associated with the multiple microservices based on a static analysis of the corresponding microservice source code and a second set of endpoints based on generating traces and capturing interactions during operation of the corresponding microservice. The method further includes integrating the first and second sets of endpoints to generate a combined set of endpoints, and determining, based thereon, the test coverage metrics for the multiple tests. The method then reconfigures, based on the test coverage metrics, at least one of the multiple tests or at least one of the multiple microservices.

[0009] In yet another example aspect, a system for generating test coverage metrics for a test suite comprising a plurality of tests is described. The system includesa microservice system comprising a plurality of microservices that interact to perform an overall application function, and one or more processors. Herein, each microservice of the plurality of microservices is (a) associated with at least one endpoint and at least one method, and (b) configured to perform a partial function of the overall application function, the at least one endpoint of a corresponding microservice enables a user or an alternative microservice to interact with the corresponding microservice, and the at least one method of the corresponding microservice enables the user or the alternative microservice to utilize one or more functions of the corresponding microservice. In this example system, the one or more processors is configured to extract the at least one endpoint and the at least one method from each corresponding microservice of the plurality of microservices, determine, based on the at least one endpoint and the at least one method from each corresponding microservice, the test coverage metrics, and reconfigure, based on the test coverage metrics, at least one of the plurality of tests or at least one of the plurality of microservices.

[0010] In yet another example aspect, an apparatus comprising a memory and a processor that implements the above-described methods is disclosed.

[0011] In yet another example aspect, the above-described methods may be embodied as processor-executable code and may be stored on a non-transitory computer-readable program medium.

[0012] The above and other aspects and features of the disclosed technology are described in greater detail in the drawings, the description and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIGS. 1A and 1 B illustrate examples of an end-to-end (E2E) testing suite and an application programming interface (API) testing suite, respectively.

[0014] FIG. 2 is a block diagram illustrating an example process for extracting test coverage metrics based on static and dynamic analysis of microservice source code.

[0015] FIG. 3 is a block diagram illustrating an example static analyzer that is configured to generate a control flow graph.

[0016] FIG. 4 is a block diagram illustrating an example process for dynamic analysis to generate a control flow graph.

[0017] FIG. 5 is a block diagram illustrating the cloud-native architecture of the Train-Ticket System, which is used to evaluate the efficacy of the disclosed technology.

[0018] FIGS. 6 and 7 are bar graphs illustrating method coverage and data entity coverage, respectively, in the benchmark test cases.

[0019] FIG. 8 is a flowchart illustrating an example method for generating test coverage metrics for a test suite comprising multiple tests for a microservice system containing multiple microservices.

[0020] FIG. 9 is a flowchart illustrating another example method for generating test coverage metrics for a test suite comprising multiple tests for a microservice system containing multiple microservices.

[0021] FIG. 10 is a block diagram illustrating an example system configured to implement embodiments of the disclosed technology.DETAILED DESCRIPTION

[0022] The advent of cloud-native architecture has brought about significant changes in software development and deployment. Cloud-native applications are intentionally designed to be flexible, scalable, and adaptable, in contrast to traditional monolithic systems. These unique attributes make them well-suited to thrive in the modern digital landscape, thanks to the support of cloud computing, microservices, containerization, and DevOps methodologies, enabling rapid application development, continuous integration, and seamless delivery. However, testing poses a distinctive set of challenges.

[0023] Conventional testing methods face difficulties when it comes to effectively handling the intricacies introduced by cloud-native systems. These systems are known for their distributed nature and dynamic infrastructure provisioning, making traditional testing strategies challenged as applications are disassembled into smaller, interconnected microservices and deployed across diverse cloud environments. This paradigm shift necessitates a thorough reassessment of testing methodologies to ensure that these modern distributed systems meet the expected performance, reliability, and security standards.

[0024] Amid these challenges, one current concern is how to assess whether End- to-End (E2E) testing aligns with what the cloud-native system under test has to offer. E2E is a software testing technique that involves evaluating a system from end to end, covering every component and process, from the user interface to the database. The goal of end-to-end testing is to verify the system's functionality, performance, reliability, and security, among other aspects. These systems are designed to fulfill a wide rangeof requirements, and these are implemented across various methods, components, and even branching scenarios. Ongoing research initiatives are in the early stages of formalizing E2E testing, focusing on endpoint evaluation and comprehensive analysis of metrics. However, the effectiveness of E2E testing largely relies on its capacity to efficiently capture executed procedures and identify the classes, methods, services, and components activated. To attain this level of thoroughness, a thorough examination of the complete execution path, encompassing internal and inter-microservice interactions, is imperative. Ensuring the robustness of a system necessitates all- encompassing E2E testing. The incorporation of dynamic test coverage metrics is essential for accurately assessing the breadth and depth of E2E test suite coverage. This approach allows for a rigorous evaluation of the testing process, conducting a comprehensive assessment of crucial elements and pathways to meet the demanding standards of E2E testing.

[0025] In view of the approach discussed above, embodiments of the disclosed technology are related to:

[0026] - A comprehensive metric suite for End-to-End testing coverage tailored to complexities of microservice systems;

[0027] - A methodology for applying the metric suite in cloud-native systems;

[0028] - A tool to automatically evaluate the testing coverage in Java-basedMicroservice Systems; and

[0029] - An empirical evaluation of all the metrics in the metric suite

[0030] Section headings are used in the present document to improve readability of the description and do not in any way limit the discussion or the embodiments (and / or implementations) to the respective sections only.

[0031] 1 Testing Coverage in Software Development

[0032] Software quality assurance, pivotal in development practices, relies on the foundation of effective testing methodologies. In addition to E2E testing discussed above, other testing methodologies include unit testing and integration testing.

[0033] Unit testing is the process where individual units of code are tested in isolation. The individual code units are small, functional components, typically at the level of functions, methods, or classes, that each have a corresponding test unit. Unit testing affords developers the ability to detect and rectify defects early in the development lifecycle, thereby mitigating the risk of downstream complications. Thisensures they function correctly according to predetermined specifications, ensuring code quality and enhancing the maintainability of software systems.

[0034] Integration testing is a type of software testing where components of the software are gradually integrated and then tested as a unified group. Usually, these components operate correctly in isolation, but they may break when they interact or are combined with other components. This ensures that the integrated software functions as expected, resolving any bugs that may not be evident when the individual components are tested in isolation.

[0035] Existing research contends comprehensive test coverage metrics are vital for the efficacy of testing strategies, and underlines the significance of black-box testing, using formal software requirements to thoroughly evaluate test suite effectiveness.

[0036] With the evolution of software development towards cloud-native architectures and microservices, new complexities have emerged within the testing domain. Testing coverage has long been regarded by developers as an indicator of their confidence level in their software. To ease the analysis of code coverage, programmers have employed various metrics to test their monolithic applications. These metrics are utilized in numerous code testing coverage tools, aimed at assisting researchers, practitioners, and end-users in comprehending the software testing process. These tools typically employ well-established metrics for monolithic systems like path, branch, line, statement, and method coverage to assess the thoroughness of testing efforts and pinpoint areas for enhancement.

[0037] For example, path coverage ensures that every possible execution path through the code is tested, helping to identify potential logic errors. Branch coverage focuses on testing every possible outcome of conditional branches in the code, ensuring comprehensive evaluation of decision points. Line coverage measures the percentage of lines of code executed by the test cases, providing a basic measure of code execution. Statement coverage, on the other hand, ensures that each executable statement in the code is executed at least once during testing, offering amore granular assessment. Finally, method coverage measures the percentage of methods or functions invoked by the test cases, identifying unused or untested methods.

[0038] Existing research has also emphasized the importance of system-level coverage, spotlighting the complexity and dynamism of modern software architectures. This is complemented by insights into effective microservices testing tools andmethodologies, some of which focus on unit and E2E testing to ensure comprehensive coverage and quality.

[0039] In this evolving context, static analysis and anti-pattern detection have emerged as pivotal tools for identifying potential issues, notably within monolithic systems. The utility of static analysis extends beyond its traditional bounds, playing a crucial role in the dynamic, multilingual landscape of cloud-native systems. Its efficiency in detecting common anti-patterns underpins a comprehensive test-generation approach, leveraging code coverage to unravel the structural and functional intricacies of software, as posited by recent research. However, the inherent challenges of cloudnative environments characterized by their dispersed nature and dynamic microservices architecture — call for a paradigm shift. Traditional bytecode analysis tools, once effective, now grapple with the complexity of dependencies and behaviors prevalent in these settings. This necessitates a deeper exploration into bytecode analysis within microservice architectures, recognizing the imperative for specialized tools that can pinpoint microservice-specific code smells.

[0040] Dynamic analysis further complements static analysis, employing instrumentation to capture and analyze the runtime behavior of programs. This approach is crucial for detecting property violations and gaining insights into program behavior. Recent systems have extended the capabilities of dynamic binary instrumentation, fostering customizable error checking, bug discovery, and performance evaluation. This methodology is especially pertinent in cloud-native systems, where the real-time tracking of endpoints and components in distributed architectures is essential for maintaining system integrity and performance.

[0041] To encapsulate, the landscape of software testing is undergoing a profound transformation, driven by the adoption of cloud-native architectures and microservices. This transition necessitates a holistic approach that integrates both static and dynamic analysis, underpinning the development of comprehensive coverage metrics tailored to the unique complexities of these systems. Embodiments of the disclosed technology bridge this gap, therefor providing metrics and methodologies designed to enhance the thoroughness and reliability of E2E testing in dynamic, distributed environments.

[0042] In some examples, the described embodiments leverage predefined coverage metrics, which empowers testers to develop and manage testing suites with known confidence and clarity regarding system coverage. Central to the adoptedapproach is the recognition of the dynamic and decentralized evolution nature of cloudnative systems, where the current system version may diverge from assumptions. Thus, the metrics suite serves as a safeguard against misinformation, offering testers a comprehensive perspective on their test suites’ alignment with the evolving system landscape. For example, the metrics suite aims to illuminate blind spots within test coverage while equipping testers with actionable insights to refine their testing strategies. By integrating assessment measures that account for system evolution, testers are empowered to navigate the complexities of cloud-native testing with precision and effectiveness. Through this cohesive framework, testers can navigate the testing process, ensuring that their test suites remain robust and aligned with the everchanging dynamics of cloud-native architectures. Moreover, the disclosed metrics suite can be used by practitioners, and can be automatically extracted without manual input.

[0043] 2 Example Embodiments of Test Coverage Metrics

[0044] Embodiments of the disclosed technology are directed to Microservice Test Coverage (MSTC) metrics and testing processes. The MSTC testing process relies on well-designed test suites comprising detailed test cases. These cases serve as comprehensive blueprints, outlining events and stages and establishing specific test scenarios.

[0045] The disclosed technology delves into the limitations of existing methodologies, particularly in understanding the effectiveness of testing suites in cloudnative systems. The described embodiments are related to assessing E2E and API testing suites using a generalized methodology, and using several coverage metrics.

[0046] 2.1 End point Coverage

[0047] Calculating the test coverage for endpoint components in a microservice system involves retrieving information on both the static endpoints declared in the system's source code and the dynamic endpoints actually tested during test suite execution. Subsequently, these two sets of information are compared in order to derive various test coverage metrics. Thus, this methodology employs both static analysis techniques (further detailed in Section 3.2) and dynamic analysis techniques (further detailed in Section 3.3) to extract the necessary information for the two testing approaches.

[0048] Herein, three metrics to assess the coverage of endpoints in microservice systems are introduced: microservice endpoint coverage, test case endpoint coverage, and complete test suite coverage.

[0049] Microservice endpoint coverage

[0050] This determines the tested endpoints within each microservice. It is obtained by dividing the number of tested endpoints from all tests by the total number of endpoints in that microservice. This metric offers insights into the comprehensiveness of coverage for individual microservices. The formula for microservice endpoint coverage is shown below.

[0051] Herein,the coverage per microservice / , E^^dis the set of tested endpoints in microservice / , and Ems(i)is the set of all endpoints in microservice / .

[0052] Test case endpoint coverage

[0053] This metric provides the percentage of endpoints covered by each test case. It is calculated by dividing the number of endpoints covered by each test by the total number of endpoints in the system. This metric allows for insights into the effectiveness of individual tests in covering the system's endpoints. The formula for test case endpoint coverage is shown below.

[0054] Herein, Ctest^ is the coverage per test / , E^f^ is the set of tested endpoints from test / , and m_total is the total number of microservices in the system. In addition, the set of all endpoints in the system is given by:

[0055] Complete test suite endpoint coverage

[0056] This metric determines the test suite overall coverage of the system by dividing the total number of unique endpoints covered by all test cases in the test suite by the total number of endpoints in the system. It provides insights into the completeness of the test suite in covering all endpoints within the system. The formula for complete test suite endpoint coverage is shown below.

[0057] Herein, Csuiteis the complete test suite coverage, m_total is the total number of microservices in the system, and t_total is the total number of tests in the test suite. In addition, (i) the set of all tested endpoints from all tests and (ii) the set of all endpoints in the system are given, respectively, by:

[0058] Examples of Endpoint Coverage in E2E and API Testing

[0059] Consider a system consisting of three microservices (MS-1 , MS-2, MS-3), each with two endpoints, and a test suite composed of two tests (Test-1 , Test-2), as depicted in FIG. 1A for an E2E test suite. In the example, the tests interact with the endpoints through the user interface, which triggers the initiation of endpoint requests passed through the API gateway component. The example demonstrates that Test-1 calls two endpoints, one from MS-1 (E1.1 ) and one from MS-2 (E2.1 ). On the other hand, Test-2 calls two endpoints from MS-2 (E2.1 , E2.2), and E2.2 has an inter-service call to endpoint E3.1 in MS-3. The identical illustration is depicted in FIG. 1 B for the API testing suite, showcasing the same interactions; however, in this case the calls are made directly through the API gateway component instead of through the user interface.

[0060] By applying the metrics on both test suites, it is possible to calculate the microservice endpoint coverage (Cms^) for each microservice. For MS-1 and MS-3, only one of the two endpoints is tested throughout all tests, resulting in a coverage of 50% (CmsW= CmS(3= 1 / 2) for each. However, for MS-2, both endpoints are tested at least one time, leading to a coverage of 100%= 2 / 2).

[0061] Next, the test case endpoint coverageper each test can be calculated. Test-1 covers two of the six endpoints in the system, resulting in a coverage of approximately 33.3%= 2 / 6). Test-2 covers three distinct endpoints, resulting in a coverage of 50% (Ctest(2^ = 3 / 6). It is important to highlight that Test-2 contains an inter-service call to endpoint E3.1.

[0062] Finally, calculate the complete test suite endpoint coverage (Csuite) of the system can be calculated. Of the six endpoints in the system, four distinct endpoints are tested during the two tests. This results in ~ 66.6% coverage (Csuite= 4 / 6).

[0063] 2.2 Method Coverage

[0064] Method Coverage (MC) assesses the extent of test coverage for methods, crucial for identifying potential bugs and ensuring system reliability. As discussed earlier, employing method coverage to assess test coverage in a monolithic system offers a comprehensive understanding of the codebase's behaviors. Recent research has underscored the significance of higher-level coverage (compared to line and statement coverage) in testing microservice-architected systems.

[0065] Method coverage translates well to microservices because it ensures thorough testing of individual service components, verifying their correctness and reliability in isolation. By focusing on testing at the method level, testers can ensure that service boundaries are well-defined and that each service behaves as expected independently. Moreover, method coverage complements other testing metrics, providing a granular view of internal logic alongside broader system-level testing. This approach helps identify and resolve issues early, ensuring the overall reliability of the microservices-based system.

[0066] To optimize microservice architecture software testing, the goal is to ascertain the extent of the test suite's coverage in covering methods across the entire system. This evaluation allows an understanding of the effectiveness of the tests in covering both individual microservices and the system as a whole. Consider a scenario where there is already a test suite in place, which is seeking to gauge its coverage depth. Two metrics are introduced for this purpose: the internal Method Calls Coverage (IMCall) and the overall Method Coverage (MCall).

[0067] Internal Method Calls Coverage (iMCall)

[0068] iMCall examines method calls within each microservice against its total available methods, offering insights into microservice-level coverage. It is defined as:

[0069] Herein, iMCallmsWis the method coverage per microservice instance i,Mml(C)disnumber of methods being tested in each microservice i, and Mms^ is the number of methods available for microservice i.

[0070] Overall Method Coverage (MCall)

[0071] MCall evaluates method coverage across the entire cloud-native system by comparing the tested methods across all microservices to the total available methods.These metrics are instrumental in ensuring thorough code coverage, reducing the risk of overlooked bugs, and ultimately boosting the reliability of our system.

[0072] Herein, MCallcnWis the method coverage for the whole cloud-native system i, is the number of methods being tested in each microservice i, andMms(o is the number of methods available for microservice i.

[0073] 2.3 Path Coverage

[0074] Path Coverage (PC) assesses the completeness of traversed paths, optimizing reliability and stability. The aim is to gauge the extent of path coverage within individual microservice instances and across the entire system to optimize the cloudnative systems. In previous research, various software testing criteria were compared, shedding light on the effectiveness and efficiency of different testing approaches, particularly those based on path coverage evaluation. Particularly, the study examined graph-based criteria such as control flow analysis and data flow analysis, foundational for path coverage evaluation.

[0075] In a microservices architecture, where software is broken down into smaller, loosely connected services, conventional testing methods like path coverage must adapt to the decentralized nature of the system. Graph-based criteria, such as control flow analysis and data flow analysis, serve as fundamental techniques in software testing that can be effectively applied in microservices environments.

[0076] For instance, in a monolithic system, control flow analysis scrutinizes how control flows through the code. Conversely, in microservices, each service possesses its unique flow, detailing its response operations to requests. This analysis highlights critical pathways within each microservice, aiding testers in identifying dependencies and potential failures.

[0077] Likewise, data flow analysis investigates the movement of data within a program. In microservices, concentrates on how data is exchanged between services, with each service having its distinct data flow, illustrating internal data movements. This analysis unveils data dependencies and potential discrepancies, ensuring testers can verify data transmission and processing across the system.

[0078] By prioritizing testing efforts on critical paths identified through control and data flow analysis, testers can ensure comprehensive coverage while effectively managing the inherent complexity of microservices architectures.

[0079] In the context of the described testing approach, two metrics are introduced: Internal Path Calls (IPC) and Overall Path Calls (PC).

[0080] IPC and PC metrics align with the concept of prime path coverage, ensuring comprehensive coverage of critical paths without requiring an exhaustive number of tests. These metrics are depicted in the following illustration and are pivotal in testing diverse interactions to ensure comprehensive coverage of all potential paths. This comprehensive coverage ultimately enhances the reliability and stability of the system.

[0081] Internal Path Calls (iPC)

[0082] iPC evaluates path coverage within each microservice by calculating the ratio of paths derived from test-generated call graphs to the total available paths. ptested pr_rms(i) lrms(i) p

[0083] Herein, iPCmsWis the path coverage per microservice instance,is the number of paths obtained from the call graphs generated from test suites for each microservice i, and PmsV)is the number of paths available for microservice i.

[0084] Overall Path Call (PC)

[0085] PC extends to the entire system, assessing paths from dynamic call graphs across all microservice instances compared to the total paths.

[0086] Herein, PCcnWis the path coverage for the whole cloud-native system, Pms r)dis the number of paths obtained from the call graphs generated from test suites for each microservice i, and Pms^ is the number of paths available for microservice i.

[0087] 2.4 Data Entity Coverage

[0088] The initial exposition of coverage metrics within this section adheres to established conventions. Nevertheless, within the complex landscape of systems across varying contexts, conventional coverage metrics may demonstrate inadequacies in obtaining substantive insights. Consequently, to address this exigency, the disclosed metrics are tailored to the nuances of service-oriented architectures.

[0089] In this subsection, the focus lies on the coverage of data entities, aiming to scrutinize their representation within the cloud-native system to augment the efficacy of testing procedures. Picture a testing scenario where there is a need to ensure the inclusion of crucial data points. To achieve this, two metrics are introduced.

[0090] Internal Data Entity Coverage (iDEC)

[0091] iDEC compares data entities in dynamic testing call graphs with static analysis in microservice instances, delineating the scope of coverage. This comparison helps prevent the omission of critical data points and minimizes data-related risks, ensuring accuracy for quality assurance.

[0092] Herein, iDECms^ is the data entity coverage per microservice instance, DE s is the number of data entities found in call graphs generated from test suites for each microservice i, and DEms^ is the number of data entities available for microservice i.

[0093] Overall Data Entity Coverage (DEC)

[0094] DEC provides comprehensive coverage of entities, which is vital for regulatory compliance, as it mitigates data-related issues. It is computed from dynamic call graphs across all microservice instances as follows:

[0095] Herein, DECcnWis the data entity coverage for the cloud-native system, DE is the number of data entities found in call graphs generated from test suites for each microservice i, and DEms^ is the number of data entities available for microservice i.

[0096] 2.5 Data Functionality Coverage

[0097] Maintaining data consistency is a primary goal in cloud-native system testing. To achieve this, various aspects of data management and manipulation are examined, with a particular focus on CRUD (Create, Read, Update, Delete) operations, which are foundational in many systems, especially those using microservices architectures. When assessing data functionality coverage, the aim is to ensure a comprehensive evaluation of all aspects of data processing in complex cloud-nativeenvironments. This is crucial for guaranteeing the reliable functionality of the system.To this end, the following metrics are introduced.

[0099] Herein, iDFCms^ is the data functionality coverage per microservice instance, DF^^dis the number of data operations found in call graphs generated from test suites for each microservice i, and DFmsWis the number of data operations from the static analyzer for microservice i.

[0100] Overall Data Functionality Coverage (DFC)

[0101] Herein, DFCcn(N^ is the data functionality coverage for the cloud-native system, DF^^dis the number of data operations found in call graphs generated from test suites for each microservice i, and DFmsWis the number of data operations from the static analyzer for microservice i.

[0102] In this context, recent research has highlighted the challenges associated with comprehending the entirety of microservices architectures due to their decentralized and rapidly evolving nature. This research introduces the concept of the Microservice Intermediate Representation Language (MIRL) to bridge this gap, facilitating a more nuanced understanding of system dynamics.

[0103] Furthermore, in the context of the data functionality coverage metric, the disclosed technology underscores the importance of having comprehensive insights into data processing within complex systems. By leveraging frameworks like MIRL, which provide a structured approach to understanding service interactions and dependencies, the assessment of data functionality coverage within cloud-native environments can be enhanced. Essentially, MIRL aids in articulating the service view perspective, thereby enabling a more robust evaluation of data processing aspects crucial for ensuring the reliable functionality of the system.

[0104] 2.6 Microservice Invocation Frequency

[0105] Determining the frequency of microservices metrics usage in cloud-native system testing is crucial for optimizing testing efforts and ensuring system reliability.Here, the goal is to prioritize testing on frequently used microservices to achieve thorough coverage and identify potential bottlenecks. By calculating both the highest and lowest usage frequencies, system performance can be assessed and workload allocation optimized. These metrics serve as a guide for testers, enabling them to prioritize testing on essential microservices, thus ensuring efficient testing strategies and evaluating system scalability for reliable functionality. The usage frequency metrics are depicted below:

[0106] Herein, C is the number of times each microservice instance is invoked in testing for microservice i, HcnWis the highest frequency usage of the microservice instance in cloud-native system, and LcnWis the lowest frequency usage of the microservice instance in cloud-native system.

[0107] 3 Example Embodiments for Extracting Test Coverage Metrics

[0108] FIG. 2 is a block diagram illustrating an example process for extracting and calculating test coverage metrics. As shown therein, the process comprises the following two main stages:

[0109] 1. Static Analysis. This stage focuses on understanding the distributed system code to discern the involved structures and flows. This step addresses the need to comprehend the system's architecture and behavior comprehensively, and is further detailed in Section 3.2

[0110] 2. Dynamic Analysis. This stage centers around collecting information about the actual execution of the system. For this purpose, listeners are integrated into the system structures to capture relevant data during runtime. The dynamic analysis is further detailed in Section 3.3

[0111] This approach is in response to the gaps identified earlier, emphasizing the need for a tailored MSTC testing framework. These gaps include uncertainties regarding the adequacy of test coverage for all methods within the tests' call graph and the assurance of exhaustiveness in paths obtained through dynamic analysis. By structuring the approach in this way, the initial comprehension of system structures and flows are prioritized before delving into the specifics of static and dynamic analysis, which is discussed in subsequent subsections.

[0112] Subsequent sections elaborate on strategies for comprehensive coverage within the intricate microservices architecture. The approach employs a combined approach to extract structural properties and operational insights, thereby evaluating the system under test thoroughly. Structural analysis focuses on path extraction, enhancing call graphs with pertinent information. Concurrently, operational insights are gathered through executing test suites and collecting traces to identify crucial data, forming the basis for constructing system route graphs.

[0113] The described evaluation process relies on the synergy between these two analyses, utilizing recommended metrics to gain detailed insights into the system's behavior, structure, and performance across various circumstances. This holistic approach fosters a robust understanding of the system's complexities, empowering testers to make informed decisions regarding their MSTC test suites.

[0114] 3.1 Systematic Analysis of Cloud-Native Architecture

[0115] In some embodiments, a systematic analysis of cloud-native architecture includes extracting structural properties, augmenting the system structures to trace their usage in operation, collecting operational insights from system usage, and analyzing operational traces. Each of these are discussed below.

[0116] Extracting Structural Properties

[0117] In the static analysis stage in FIG. 2, the focus is on comprehending the system's architecture and functionalities without being hindered by language-specific details. This comprehensive static analysis aims to uncover communication pathways among system components, placing a specific emphasis on REST components like RestTemplate in Java or their equivalents in other languages. The primary goal of this phase is to extract the structural properties of the system under test. By initiating the static analysis process, a deep understanding of the system's behavior and interactions is achieved through code / bytecode examination. Consequently, this enables the extraction of execution paths, paving the way for insights and performance enhancements through systematic analysis of the compiled code.

[0118] Augmenting the System Structures to Trace Usage in Operation

[0119] Instrumentation serves as a fundamental component in augmenting system structures to trace their usage in operation. In the context of existing tools, dynamic binary instrumentation tailored for NVIDIA GPUs allows for the modification of precompiled binaries and libraries at runtime. This augmentation empowers developers toinsert code dynamically, facilitating error checking, bug discovery, performance evaluation, and selective profiling.

[0120] By augmenting system structures with existing tools, developers can gain essential insights into the behavior of GPU-accelerated applications. This enhances their understanding of performance characteristics, potential bottlenecks, and areas for optimization.

[0121] Collecting Operational Insights from System Usage

[0122] One of the objectives of the disclosed technology is to collect operational insights from the usage of the system. To this end, techniques to proactively locate execution paths containing methods called at runtime for every test case in a suite are employed. This involves monitoring the real-time behavior and transactional operations of the target system to generate traces during sequential test execution. Integration of OpenTelemetry facilitates the creation of a distributed tracing system for cloud-native applications, enabling insights into system operations.

[0123] The instrumentation process includes adding trace spans to the system dynamically and configuring Kubernetes Deployments to trace microservices and methods. Traces collected by Jaeger are organized into a hierarchical tree, enabling in- depth analysis of microservices interactions. A trace filtration mechanism is applied to sift through the data, ensuring that only relevant traces are retained for analysis. This meticulous approach enhances the traceability and observability of cloud-native systems, ultimately contributing to the complete testing techniques required to confirm their resilience and dependability. This approach is the dynamic analysis phase.

[0124] Analyzing Operational Traces in Cloud-Native Architecture

[0125] At the last step in FIG. 2 (MSTC Metrics), the MSTC metrics are calculated, thereby demonstrating the application of the devised metrics and methodologies for evaluating testing completeness in microservice architectures. These metrics are strategically employed to assess the effectiveness and comprehensiveness of testing processes within cloud-native systems and serve as instruments to achieve our overarching goal of analyzing operational traces and deriving valuable insights.

[0126] For example, method coverage metrics provide visibility into the extent of testing coverage for individual microservices and the system as a whole, aiding in identifying potential gaps in testing scenarios. Path coverage metrics offer detailedanalysis on the efficiency of test scenarios in traversing diverse routes across the system, revealing potential areas of weakness or redundancy.

[0127] The inclusion of data entity coverage metrics ensures thorough testing of critical data points, mitigating risks associated with data-related issues. Additionally, by examining microservice invocation frequency, we gain valuable insights into the frequency and distribution of interactions among microservices, further enhancing our understanding of system behavior under test conditions.

[0128] Through the calculation of MSTC metrics, a holistic approach to evaluating testing completeness in microservice architectures is provided. By focusing on the extraction and analysis of operational traces, the aim is to drive improvements in quality assurance practices and enhance the overall reliability of cloud-native systems.

[0129] 3.2 Example Embodiments for Static Analysis

[0130] The Static Analysis (Source Code Analysis) portion of FIG. 2 aims to comprehend the offerings of the system implementation concerning the declared endpoints ready for consumption. The approach applies a static analysis approach to the system's source code to extract the paths (or equivalently, endpoints or any other metric) employed in each microservice. Static analysis refers to the process of analyzing the syntax and structure of code without executing it in order to extract information about the system. As shown in the Static Analysis portion of FIG. 2, microservices can be divided and detected from the system codebase. Each microservice's codebase is then processed by the path extraction process, which produces the paths corresponding to each microservice. As noted earlier, an endpoint extraction process could be employed to produce the endpoints corresponding to each microservice.

[0131] In the example of endpoints, the identification of API endpoints typically relies on specific frameworks or libraries (for example, the Java Spring framework uses annotations such as @RestController and @RequestMapping). This ensures consistency in metadata identification. Code analysis extracts metadata attributes about each endpoint, including the path, HTTP method, parameters, and return type. However, identification of endpoints can be performed across platforms or accomplished by frameworks such as Swagger.

[0132] As a result of this process, a list of endpoints is generated and organized according to the respective microservice that each belongs to. This comprehensive listof endpoints becomes one of the inputs for our coverage calculation process, where it is combined with the output of the dynamic analysis flow.

[0133] FIG. 3 is a block diagram showing an example architecture for the static analysis procedure. The static analysis process includes generation call graphs with paths containing all the paths for each microservice instance and the communication paths from one instance to another. These can be mapped to generate the complete path for the whole system. In the example embodiments shown in FIG. 3, this can be implemented by defining the cloud-native system’s location as part of the process. The specified path then is followed to extract JAR files for each microservice instance. For every JAR file, we get a consistent pool of classes. Each class is traversed with a code iterator to extract method call opcodes for analysis. Finally, the aim is to find RestTemplate-based techniques that enable extraction of endpoints or calls for intermicroservice communication. Combining the method calls for each microservice and the inter-microservice calls results in the creation of a meticulous path based on this extensive data. Finally, the outcomes are aggregated into a JSON file, which offers an organized depiction of the detected paths and method calls.

[0134] In some embodiments, the static analysis stage includes syntactic parsing of the source code of each corresponding microservice. Herein, performing the syntactic parsing of the source code includes inspecting an intermediate code representation of the source code. For example, the source code is Java or C++, and the intermediate code representation is bytecode. In some embodiments, inspecting the intermediate code representation includes separating a control flow and a data flow, and performing an instrumentation of instructions to the source code. In some examples, instrumenting instructions to the source code includes adding additional code (e.g., in the same language as the source code is written) to the source code to enable logging and / or tracing. More generally, code instrumentation is performed without changing the original source code. The additional code monitors the behavior of a specific component and collects data such as metrics, events, logs, and traces. Code instrumentation provides access to performance metrics, software debugging, and security analysis. It also enables profiling, which is the measurement of dynamic behavior during a test run.

[0135] 3.3 Example Embodiments for Dynamic Analysis

[0136] The Dynamic Analysis (Test Execution) portion of FIG. 2 aims to identify the endpoints invoked by the test suites during runtime. Dynamic analysis is utilized toidentify the endpoints called during the execution of each test case in test suites. In addition, dynamic analysis identifies the microservices containing these tested endpoints. The analyzed system is executed to observe its runtime behavior and transactions. This analysis involves running multiple tests and capturing the traces that occur, as illustrated in Dynamic Analysis portion of FIG. 2.

[0137] The dynamic analysis flow sketched in FIG. 2 has two main responsibilities. First, it takes the tests (e.g., E2E tests and API tests shown in FIGS. 1A and 1 B, respectively) and executes them sequentially. During the execution of the tests, traces are generated to capture the interactions with the system. These traces are sent to a configured centralized logging system (e.g., SkyWalking, Jaeger), which stores them either in its own storage or in an externally configured data storage solution (e.g., Elasticsearch), enabling analysis and further processing. Second, the process calculates the delta of the produced traces in order to identify the traces relevant to each executed test. This can be achieved in various ways, such as by recording a timestamp from the start of a test's execution to its completion, retrieving the traces after each test execution and calculating the difference based on the latest track record, or sending a dynamically generated trace before and after the execution of each test to mark the start and end. An advantage of the first strategy is that it avoids unnecessary processing and complexity at this stage.

[0138] The extracted test trace sequences corresponding to each test undergo a trace filtration process that filters and identifies the traces related to endpoints. This may involve queries to the trace storage to return specific trace indexes in the data. For instance, the SkyWalking tool marks the traces involving endpoint calls and makes them accessible under an index (in particular, sw_endpoint_relation_server_side index). Additionally, centralized logging systems encode the data records using, for example, Base64 when sending them to external storage like Elasticsearch. Therefore, this step may include an additional decoding process if needed to detect the endpoints. These endpoint-related trace records contain information about the source and destination endpoints involved in the call relationship. As a result, a list of endpoints is generated and organized according to the respective test suite they belong to. This list of endpoints becomes the second input for the coverage calculation process, where it is combined with the output of the static analysis stage.

[0139] FIG. 4 is a block diagram showing an example architecture for the dynamic analysis procedure. The dynamic analysis process includes integrating OpenTelemetry (Otel) and Jaeger to create a distributed tracing system for cloud-native applications. The tracing system helps provide insights into the application’s operations. The process starts with the Otel Library, which is integrated into the application to capture trace data. This data is then sent to the Otel-collector, which can be deployed in various configurations within a Kubernetes environment, such as a deployment. The Otel- collector sends the collected trace data to the Jaeger-collector, which processes and stores it in a database that could be Elasticsearch, Kafka, Cassandra, or an in-memory solution. The Jaeger-query service retrieves the data and presents it through the Jaeger Ul for querying and visual inspection of the traces. The distributed tracing system helps monitor and troubleshoot microservices-based applications by clearly visualizing the transactions and workflows.

[0140] Instrumentation Process

[0141] Using the OpenTelemetry library for instrumentation enables a distributed tracing library to be included in the targeted cloud-native solutions. This process entails adding trace spans to the system on the fly. To be more precise, spans are made for every method to ensure they are traceable and connected correctly.

[0142] Filtration Process

[0143] To carefully sift through the gathered data, a trace filtration mechanism is actively integrated in the trace. This process involves skillfully applying a whitelist, a crucial part of the setup. In this case, the explicit names of the libraries that have been identified and are intended to be retained inside the gathered traces are included in the whitelist. Traces produced by entities not part of these approved libraries are carefully and systematically filtered out, with the primary goal being preserving the overall graph structure. For this particular application, the end goal is to keep only the traces from the method library. This trace filtration process is crucial in ensuring that the trace data produced suits the analytical needs and perfectly matches the predetermined standards. This meticulous procedure guarantees that the trace data provides a more targeted, significant, and pertinent viewpoint, allowing for a deep understanding of the system’s behavior.

[0144] 3.4 Example Implementations in Java Projects

[0145] The efficacy of the disclosed technology is evaluated using the Java platform due to its widespread usage and extensive ecosystem of frameworks and standards. For tracing purposes, OpenTelemetry, which is a common choice atop Kubernetes environments, has been adopted. Additionally, to ensure thorough end-to- end testing coverage, the Selenium framework has been integrated, thereby enabling automated testing of user interactions and functionalities. This comprehensive toolset enables analyzing, tracing, and testing the cloud-native systems with precision and efficiency.

[0146] Analysis of Java-based Cloud-Native Systems

[0147] The disclosed embodiments provide a robust approach to conducting static analysis on Java-based cloud-native systems, with a particular focus on Spring Boot applications. The methodology encompasses a series of steps aimed at comprehensively understanding the intricacies of these systems.

[0148] Firstly, the tool embarks on a thorough examination of the Java source code. This initial phase involves scrutinizing the structure and logic of the codebase to identify essential elements crucial for system functionality. While the prototype project is tailored for Java, the described methodology is adaptable to any Java-based codebase, providing flexibility for developers across various projects.

[0149] One of the pivotal aspects of the approach lies in bytecode analysis. By dissecting the Java bytecode, the tool gains insight into the control flow, data flow, and specific instructions within the program. This deep dive into the bytecode level allows for a granular understanding of the program's behavior, enabling developers to identify potential bottlenecks or areas for optimization.

[0150] In the realm of cloud-native systems, inter-microservice communication plays a crucial role. The tool places significant emphasis on analyzing these communication channels, particularly leveraging Spring's RestTemplate for HTTP communication between microservices. By tracking method invocations utilizing RestTemplate, the tool uncovers dependencies and communication patterns, shedding light on the intricate network of interactions within the system.

[0151] Furthermore, the tool facilitates the generation of call graphs and paths, providing a visual representation of the system's architecture and execution flows.These visualizations offer invaluable insights into the relationships between different components, facilitating better understanding and decision-making for developers.

[0152] The culmination of the tool's analysis efforts results in the aggregation of findings into structured JSON files. This organized representation allows developers to easily navigate and interpret the data.

[0153] Enhancing Observability and Traceability through Instrumentation

[0154] In some embodiments, the instrumentation process in the described tool is designed to enhance the observability and traceability of cloud-native systems. This process involves several specific steps and techniques aimed at enabling proactive monitoring, analysis, and optimization of application behavior.

[0155] One of the key aspects of the instrumentation process is the integration of the OpenTelemetry library within the targeted applications. This integration entails adding trace spans to the system on the fly, ensuring that each method invocation is accurately captured and traced. By systematically instrumenting methods with trace spans, detailed monitoring of application behavior at a granular level is enabled, thereby ensuring comprehensive coverage during testing and runtime operation.

[0156] Moreover, the instrumentation process includes the generation of control flow graphs using the OpenTelemetry methods library. These graphs provide a structured visualization of the sequence of method calls and their interdependencies within the architecture. By configuring the OpenTelemetry library to construct control flow graphs for methods, valuable knowledge of the execution flow of the application is gained. This visualization aids in identifying performance bottlenecks, inefficiencies, and potential points of failure, facilitating targeted optimization efforts.

[0157] Additionally, the instrumentation process involves configuring the system to collect and store trace data effectively. This includes setting up Jaeger collectors and storage solutions such as Elasticsearch, Kafka, Cassandra, or in-memory databases to store the collected trace data. Jaeger's capabilities are leveraged to ensure that trace data is efficiently organized and stored, enabling comprehensive analysis and visualization.

[0158] Moreover, the instrumentation process includes creating a Java project specifically intended for trace collection using Jaeger. This project automates the retrieval of traces from registered microservice instances within a given time frame.Leveraging Jaeger's hierarchical tree structure to organize gathered traces optimizes its trace collecting and in-depth analysis capabilities.

[0159] Furthermore, the instrumentation process incorporates a filtration mechanism to sift through the gathered trace data effectively. By integrating a trace filtration mechanism within the trace collection process, whitelists can be skillfully applied to retain only approved traces from specific libraries. This meticulous filtration process ensures that the trace data aligns with analytical needs and standards, providing a more targeted and pertinent viewpoint of the system's behavior.

[0160] Finally, the instrumentation process, closely integrated with Jaeger and OpenTelemetry, enhances the observability and traceability of cloud-native applications. By leveraging the capabilities of both tools for trace collection, organization, and visualization, a deeper understanding of application behavior is acquired, thereby empowering proactive monitoring, analysis, and optimization efforts.

[0161] Real-time Profiling and Traceability with Dynamic Analysis

[0162] In some embodiments, the described dynamic analysis tool (e.g., implemented in the dynamic analysis stage in FIG. 2) is specifically tailored for Java programs within cloud-native environments. It focuses on capturing runtime behavior, execution paths, and method invocations to provide insights into the system's functionality and performance. Using dynamic analysis techniques, the tool proactively collects traces of execution paths containing methods invoked at runtime for each test case in a suite. This approach allows capture real-time interactions with the system to be captured, enabling comprehensive understanding and troubleshooting.

[0163] Unlike traditional approaches that solely identify paths, the tool actively locates microservices associated with the tested methods. By dynamically tracing method invocations, it provides a detailed map of microservices involved in the execution paths. The tool integrates with OpenTelemetry and Jaeger to establish a distributed tracing system for cloud-native Java applications. OpenTelemetry captures trace data within the application, while Jaeger facilitates the processing, storage, and visualization of traces.

[0164] In some embodiments, for Java programs, the tool employs the OpenTelemetry Java library for instrumentation. This library dynamically adds trace spans to methods, ensuring they are traceable and connected correctly. The setup and integration adhere to the OpenTelemetry standard, enhancing observability andtraceability. To create control flow graphs for Java programs, the tool generates a Java command-line project. This project configures the OpenTelemetry method library to instrument methods within the architecture. It sets up Kubernetes deployments for each microservice, incorporating the OpenTelemetry Java Agent for instrumentation. Jaeger's trace collection project automatically retrieves traces from microservice instances, organizing them into a hierarchical tree for analysis. The tool integrates a trace filtration mechanism to sift through gathered data, ensuring that only relevant traces are retained for analysis.

[0165] In some embodiments, the workflow of the dynamic analysis tool involves adding dependencies, creating a command-line project, configuring Kubernetes deployments, running test suites, and recording traces using Jaeger. These traces undergo filtration, resulting in comprehensive control flow graphs tailored for Java programs within cloud-native environments. Overall, our dynamic analysis tool provides developers with a greater understanding of the behavior and performance of Java programs within cloud-native architectures, enabling optimization and troubleshooting.

[0166] Analyzing Traces for Comprehensive Metric Derivation

[0167] The process of E2E testing relies on test suites containing detailed test cases. These cases serve as comprehensive guides, detailing events and stages while establishing specific test scenarios. In some embodiments, and to ensure a thorough evaluation, several metrics that precisely measure method, path, and data coverage within cloud-native systems are utilized. These metrics help assess the comprehensiveness and effectiveness of the testing approach.

[0168] The process to derive the coverage metrics from the traces obtained through dynamic analysis is as follows. Firstly, the traces are queried (structured in JSON format) to extract method invocations and executions, providing a detailed record of the system's runtime behavior. These method invocations are then mapped to the corresponding methods defined within the system architecture, facilitating a structured representation of the system's interactions.

[0169] Once mapped, the unique methods identified through dynamic analysis are counted, thereby tallying the distinct methods executed during system operation. Subsequently, these methods are compared with the ground truth, a reference for the complete set of methods within the system. By comparing the dynamic analysis results,represented in JSON, with the ground truth, also structured in JSON, the accuracy and completeness of method coverage can be assessed.

[0170] Similarly, for path coverage calculation, query the traces (structured in JSON format) to retrieve the sequence of method calls representing different system paths. Mapping these paths to architectural components enables evaluation of how comprehensively the system's behavior is captured during dynamic analysis. Comparison with the ground truth, also structured in JSON, helps assess the accuracy and completeness of path coverage.

[0171] For data entity coverage, identify data entities involved in method invocations or operations from the traces (structured in JSON format). Mapping these entities to the system's data model allows the counting of unique data entities. Comparing with the ground truth (also structured in JSON), evaluates the accuracy and completeness of data entity coverage.

[0172] Lastly, for data functionality coverage, detect CRLID operations or other data-related functionalities from the traces (structured in JSON). Mapping these functionalities to the system's functionalities enables the counting of unique data functionalities. Comparison with the ground truth also represented in JSON, assesses the accuracy and completeness of data functionality coverage.

[0173] Overall, this process involves querying the traces (structured in JSON format), mapping the information to the system's architecture, and comparing with the ground truth (also represented in JSON) to determine the accuracy and completeness of coverage metrics.

[0174] 4 Examples of Empirical Evaluation in Microservice Systems

[0175] The efficacy of the described embodiments is evaluated using the TrainTicket system, shown in FIG. 5, as a benchmark. The TrainTicket system, featuring 40+ microservices and over 60,000 lines of code, replicates real-world complexities in operational environments. The evaluation included an open-source benchmark with 11 Selenium test cases, which aligns with the specific version of the TrainTicket system. This meticulous approach ensured a rigorous assessment of our methodology's reliability across diverse scenarios. The study addressed specific metrics identified and applied five key metrics - MC, PC, DEC, DFC, and MIF. These metrics were systematically applied and compared with manual ground truth to validate the accuracy and practicality of our methodology for implementation.

[0176] 4.1 Analyzing System Dynamics for Test Completeness

[0177] To configure the static analyzer within the TrainTicket system's architecture, the source code path is specified. Initially, the microservice source code is compiled into Java Executable Files or Jar files. These Jar files are then assessed by the static analyzer to create call graphs forthe system. The resulting logs offer a detailed breakdown of specified paths, capturing critical information on code paths, dependencies, methods, and structure.

[0178] For instance, a closer look at these paths reveals a structured format, exemplified in the t s -admin-ba s ic-inf o-service . It outlines the Admin- Ba s icinf oControiier class, prefixed with the package name edu . fudanselab . trainticket , adminba s ic . cont rolle r, specifically specifying the getAHConf igs method. This method correlates intricately with the same-named method within the AdminBa s ici nfoService class, distinguished by colons in the call graph.

[0179] The analysis extends beyond single microservice paths, encompassing inter-microservice calls. For example, the AdminBa s i cinf oService impi class triggers the deletestation method, which then invokes the 'delete' method within the stationcontrol ler class in the t s - statlon-s ervlce. This recursive method execution across microservices highlights the complexities of inter-microservice communication, showcasing the depth of the static analysis captured in the logs.

[0180] Subsequently, the analyzer generates a JSON file containing these paths identified within the call graphs, formatted in a specific structure. The JSON structure was purposefully crafted to facilitate a seamless evaluation of the proposed metrics. It systematically captures components and their associated method invocations within the labeled JSON array 'paths,' incorporating crucial attributes like package name, class layer (indicating class type), class name, method name, and a node field encompassing the entire path from the organizational package to the specified method. When a method calls other methods, they are categorized as subnodes. If these called methods further invoke additional methods, the same hierarchical structure is replicated. This approach ensures a comprehensive and structured representation of the system's call graph, enabling efficient metrics assessment and analysis.

[0181] 4.2 Enhancing Test Coverage through Dynamic Instrumentation

[0182] The TrainTicket system was tested by incorporating OpenTelemetry and Jaeger into the source code, which involved adding dependencies to the project's pom. xml file, enabling automatic download and execution during application runtime. Subsequently, a command-line project utilized static analysis principles to extract essential information from the source code, facilitating OpenTelemetry with a YML configuration file for Jaeger tracing across all microservice instances. Furthermore, to configure the TrainTicket system, metadata for microservice instances, Otel Agent definitions, instrumentation methods, and resource specifications were specified. This configuration ensured effective data collection and transmission from Otel Agent to Jaeger, establishing a foundation for robust monitoring and traceability.

[0183] During the phase of executing Selenium tests, crafted in Java, these tests covering MSTC scenarios in the TrainTicket system benefited from the implemented configuration. This allowed for in-depth method instrumentation and tracing. The Otel agent collected these traces, enabling Jaeger to visually represent the method details, microservice instances, classes, and inter-service communication intricacies.

[0184] Upon obtaining traces in Jaeger, the Jaeger API service is utilized to retrieve and organize traces into a tree structure, filtering based on a predefined whitelist. This filtration ensured the creation of a graph mirroring the static analyzer's JSON output, illustrating dynamic runtime interactions within the TrainTicket system. This approach offers valuable insights into its behavior and performance across different scenarios.

[0185] 4.3 Quantitative Evaluation ofCoverage Metrics

[0186] Herein, the developed metrics are evaluated using two JSON datasets: one from static analysis and the other from dynamic analysis. Applying mathematical formulas detailed earlier to these datasets allows valuable conclusions and measurements to be drawn. This case study validates the methodology's ability to accurately identify and capture specified methods, pathways, entities, functions, and microservice instances, showcasing the technique's comprehensiveness and accuracy in describing the underlying system components.

[0187] Method Coverage (MC)

[0188] To comprehensively assess MC, the unique methods from the static analyzer's JSON output were tallied. These were compared with the ground truth,offering insight into the accuracy of our static analysis in identifying system-defined methods. Extending our assessment, we then considered unique methods from the dynamic analysis JSON. FIG. 6 synthesizes the findings, presenting a horizontal bar chart that contrasts the number of unique methods detected by static and dynamic analyses in the TrainTicket system. Light grey bars denote methods identified solely by static analysis, while dark grey bars represent those identified during code execution by dynamic analysis. The chart illustrates that the static analyzer detects a total of 2143 unique methods, with dynamic analysis covering 1067 of these. Additionally, there is a higher count for the t s -common- servi ce category in both analyses. From these values, coverage metrics can be derived, such as Cms(t s-cornmon), indicating coverage for the t s- common-service microservice instance at 69.85%. The chart serves as an evaluation tool, assessing the comprehensiveness and accuracy of both analysis methods in capturing the full spectrum of methods within the system.

[0189] Path Coverage (PC)

[0190] The unique paths within the static analyzer's output JSON were isolated with the aim to cover all potential paths in the analysis. These paths were then compared against the ground truth, assessing the precision of the static analysis. Additionally, the unique paths extracted from the dynamic analysis JSON were assessed and contrasted with the static analysis. The results indicated a higher count for the ts -common- servi ce category in both analyses. Utilizing these values, the PC metric is derived, like PCms(ts -admin-ba s ic-info- service), indicating a coverage of 36.91 % for the ts -admin-ba s ic- info- service category. The overall PC for the benchmark system, CbenchmarkC7), stands at 7.98%. This assessment not only aids in comprehending the system's comprehensiveness but also sheds light on the accuracy of the described analysis techniques.

[0191] Data Entity Coverage (DEC)

[0192] To comprehensively evaluate DEC, the static analyzer output JSON was analyzed to compute unique data entities. A stringent comparison with the detailed ground truth enabled an accurate assessment of the static analysis's identification of data entities in the TrainTicket architecture. Subsequently, unique data entities extracted from the dynamic analysis JSON were evaluated. This critical analysis offers insights into the correlation between static and dynamic analyses in terms of DEC, depicted visually in FIG. 7. The visualization illustrates that static analyzer detects 92unique data entities, while dynamic analysis covers 63. Notably, more entities are observed in the ts -common-service category in both analyses: 43 unique data entities in static and 29 in dynamic. Specifically, for the t s-ins ide-payment- service category, the coverage metric computes to 50%, represented as DECms(ts- ins ide-payment- service) = 48 ~ 50%. Assessing the DEC of the benchmark system reveals approximately 56.45% coverage (Z)£Cbenchn k( / V) = 63 / 92 ~ 56.45%).

[0193] Data Functionality Coverage (DFC)

[0194] DFC was assessed by identifying and scrutinizing unique data functionalities within the static analyzer output JSON. These functionality were categorized and quantified, thereby laying the groundwork for a detailed assessment. Data functionalities encompass diverse operations associated with data management, transformation, validation, storage, presentation, and manipulation — collectively shaping software capabilities and user experience. The system's unique data functionalities were compared with both the ground truth and the static analysis. Extending the evaluation, data functionalities from the dynamic analysis JSON were incorporated. The comprehensive analysis highlights the alignment between static and dynamic analyses regarding DFC. The results indicates that the static analyzer identifies 103 unique data functionalities, including CRUD operations, whereas dynamic analysis covers 66. Notably, there's a higher frequency for the "ts-admin-basic-info- service" category in both analyses. Specifically, for the "ts-admin-basic-info" microservice, a coverage metric of 100% (£>FCms(ts-admin-basic-info) = 20 / 20 = 100%) is ascertained. The overarching assessment of DFC in the benchmark system yields a coverage metric of 64.07% (D Cbenciimark(N) = 66 / 103 ~ 64.07%).

[0195] Microservice Invocation Frequency (MIF)

[0196] Understanding the usage frequency of microservices is pivotal for optimizing the TrainTicket system. Table 1 showcases notable frequency variations among microservices. For instance, the "ts-common-service" instance emerges as the most frequently utilized, appearing 2552 times in static analysis and 204 times in dynamic analysis. Conversely, the "ts-payment" instance exhibits the lowest frequency. A comprehensive understanding of their usage patterns is gained by computing the lower quartile, median, and upper quartile frequencies of these microservice instances. This detailed analysis furnishes crucial statistical insights into the distribution of instancefrequencies, offering a deeper comprehension of their prevalence and impact within the TrainTicket system.Table 1: Microservice Invocation Frequency (MIF) in the Benchmark Test Cases

[0197] 5 Additional Embodiments of the Disclosed Technology

[0198] As discussed in this patent document, the described embodiments are directed to a structured approach to assess test completeness in cloud-native systems. By concentrating on a standard architecture and utilizing the TrainTicket system as a benchmark, the adaptability of the methodology across various scenarios was illustrated. The case study illuminated how the methodology accurately pinpointed and captured vital system components, pathways, entities, functions, and microservice instances, directly addressing the research questions outlined at the study's inception.

[0199] Moreover, the case study facilitated comparison with manual ground truth, showcasing the efficiency and precision of our methodology. By systematically applying metrics and leveraging automated analysis tools, substantial time savings compared to manual assessment methods were achieved. This streamlined approach enables more frequent and thorough evaluations of test completeness, ultimately elevating the overall quality and dependability of cloud-native systems.

[0200] Through the automated methodology, a better understanding of the depth of inter-microservice communication and its impact on test coverage was garnered, as well as the efficiency and reliability of the test suite under varying circumstances. These revelations, made possible through automated analysis, underscore the significance of modern software development practices and highlight the transformative potential of techniques like those described in this patent document.

[0201] A non-exhaustive summary of the advantages provided by embodiments of the disclosed technology include:

[0202] - An assessment of MSTC test suite coverage, which involved evaluating various aspects such as method coverage, path coverage, data entity coverage, data functionality coverage, and microservice frequency coverage. Method coverage ensures thorough examination of all system methods, reducing the likelihood of unnoticed bugs and improving overall system reliability. Path coverage guarantees testing of diverse execution paths, crucial for cloud-native systems to ensure robustness and stability. Data entity coverage maintains data integrity and functionality alignment, preventing errors across various scenarios. Data functionality coverage ensures comprehensive testing of data processing functionalities, contributing to reliable system performance. Microservice frequency coverage offers insights into the distribution and frequency of microservice invocations, guiding testers to focus on heavily used services while not neglecting less frequently used ones, thus ensuring a balanced and comprehensive testing approach. By examining these metrics, testers can determine the comprehensiveness of their testing approach, identify areas needing further attention, and ensure reliable system functionality.

[0203] - The deployment of automated tools to automatically gather test metrics for MSTC testing in cloud-native systems. These tools track system activities, organize and visualize collected data, and are configured to collect detailed tracks and automate the process of capturing and analyzing test metrics. This enables testers to efficiently evaluate MSTC testing coverage, continuously monitor and improve their testing approach.

[0204] - The introduction of a robust MSTC testing methodology for cloud-native microservice architectures, setting a new standard for performance evaluation validated in real-world scenarios like the TrainTicket system.

[0205] The described embodiments further include, as shown in the flowchart in FIG. 8, a method 800 for generating test coverage metrics for a test suite comprising multiple tests for a microservice system containing multiple microservices. In method 800, the multiple microservices interact to perform an overall application function, with each microservice being associated with at least one endpoint and configured to perform a partial function of the overall application function, and the at least one endpoint of a corresponding microservice enabling a user or an alternative microservice to interact with the corresponding microservice. The method 800 includes (810) extracting at least one endpoint from each corresponding microservice of the multiplemicroservices, (820) determining, based on the at least one endpoint from each corresponding microservice, test coverage metrics, and (830) reconfiguring, based on the test coverage metrics, at least one test of the multiple tests or at least one of the multiple microservices.

[0206] The described embodiments further include, as shown in the flowchart in FIG. 9, a method 900 for generating test coverage metrics for a test suite comprising multiple tests for a microservice system containing multiple microservices. The method 900 includes (910) generating a first set of endpoints associated with the multiple microservices based on a static analysis of the corresponding microservice source code and (920) a second set of endpoints based on generating traces and capturing interactions during operation of the corresponding microservice. The method further includes (930) integrating the first and second sets of endpoints to generate a combined set of endpoints, and (940) determining, based thereon, the test coverage metrics for the multiple tests. The method then (950) reconfigures, based on the test coverage metrics, at least one of the multiple tests or at least one of the multiple microservices.

[0207] The described features and aspects can be implemented to further provide one or more of the following technical solutions:

[0208] A1. A system for generating test coverage metrics for a test suite comprising a plurality of tests, the system comprising: a microservice system comprising a plurality of microservices that interact to perform an overall application function, each microservice of the plurality of microservices being associated with at least one endpoint and configured to perform a partial function of the overall application function, wherein the at least one endpoint of a corresponding microservice enables a user or an alternative microservice to interact with the corresponding microservice; and one or more processors configured to: extract the at least one endpoint from each corresponding microservice of the plurality of microservices, determine, based on the at least one endpoint from each corresponding microservice, end-to-end test coverage metrics, and reconfigure, based on the end-to-end test coverage metrics, at least one end-to-end test of the plurality of tests or at least one of the plurality of microservices.

[0209] A2. The system of solution A1 , wherein an end-to-end coverage metric for a microservice (Cms) of the plurality of microservices is determined based on a number of tested endpoints associated with the microservice and a number of endpoints associated with the microservice.

[0210] A3. The system of solution A1 , wherein an end-to-end coverage metric for an end-to-end test (Ctest) of the test suite is determined based on a number of tested endpoints associated with the end-to-end test, a number of the plurality of microservices, and a total number of endpoints associated with all microservices of the plurality of microservices.

[0211] A4. The system of solution A1 , wherein an end-to-end coverage metric for the test suite (Csuite) is determined as:

[0212] Herein, t_total is a number of the at least one end-to-end test, wherein m_total is a number of the plurality of microservices, |utot“;£teesst(o l is a total number of tested endpoints associated with all of the at least one end-to-end test of the plurality of tests, and | \j™-totalEms(J)| is a total number of endpoints associated with all microservices of the plurality of microservices.

[0213] A5. The system of solution A1 , wherein each microservice is associated with at least one method, wherein the at least one method of the corresponding microservice enables the user or the alternative microservice to utilize one or more functions of the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one method from each corresponding microservice; and determine, based on the at least one method, a method coverage metric for a microservice iMCallms) of the plurality of microservices that is determined based on a number of tested methods in the microservice and a number of available methods in the microservice.

[0214] A6. The system of solution A1 , wherein each microservice is associated with at least one path, wherein the at least one path of the corresponding microservice enables the user or the alternative microservice to access one or more functions of the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one path from each corresponding microservice; and determine, based on the at least one path, a path coverage metric for a microservice (iPCms) of the plurality of microservices that is determined based on a number of paths retrieved from call graphs generated from the test suite and a number of paths available for the microservice.

[0215] A7. The system of solution A1 , wherein each microservice is associated with at least one data entity, wherein the at least one data entity of the corresponding microservice is data that is (a) accessible to the user or the alternative microservice and (b) owned and controlled by the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one data entity from each corresponding microservice; and determine, based on the at least one data entity, a data entity coverage metric for a microservice (iDECms) of the plurality of microservices that is determined based on a number of data entities retrieved from call graphs generated from the test suite and a number of data entities available to the microservice.

[0216] A8. The system of solution A1 , wherein each microservice is associated with at least one data operation, wherein the at least one data operation of the corresponding microservice enables the user or the alternative microservice to write, update, or delete data associated with a data entity, and wherein the one or more processors is configured to: extract the at least one data operation from each corresponding microservice; and determine, based on the at least one data operation, a data functionality coverage metric for a microservice (iDFCms) of the plurality of microservices that is determined based on a number of data operations retrieved from call graphs generated from the test suite and a number of data operations from a static analysis of the microservice.

[0217] A9. The system of any of solutions A1 to A8, comprising: a visualization model configured to generate a visualization of the microservice system.

[0218] A10. The system of solution A9, wherein the visualization comprises each endpoint of each of the plurality of microservices, and wherein an endpoint covered by the at least one end-to-end test is colored using a first color and an endpoint missed by each of the at least one end-to-end test is colored using a second color.

[0219] A11. The system of solution A9, wherein the visualization comprises a service dependency graph that represents the microservice system, and wherein each microservice is a node of the service dependency graph and a dependency between two microservices is an edge between two nodes corresponding to the two microservices.

[0220] A12. The system of solution A11 , wherein the node representing a particular microservice is colored based on a coverage percentage of the particular microservice.

[0221] A13. The system of any of solutions A1 to A8, wherein extracting the at least one endpoint is based on a syntactic parsing of source code of the corresponding microservice, wherein the syntactic parsing of the source code refrains from executing the source code, and wherein the source code comprises bytecode or binary code.

[0222] A14. The system of any of solutions A1 to A8, wherein extracting the at least one endpoint is based on generating traces and capturing interactions during an operation of the corresponding microservice.

[0223] A15. The system of solution A13 or A14, wherein extracting the at least one endpoint generates metadata attributes thereof, and wherein the metadata attributes comprise a path, a Hypertext Transfer Protocol (HTTP) method, one or more parameters, or a return type associated with the corresponding microservice.

[0224] A16. A method for generating end-to-end test coverage metrics for a test suite comprising a plurality of end-to-end tests for a microservice system that includes a plurality of microservices, the method comprising: generating, based on a syntactic parsing of source code of a corresponding microservice, a first plurality of endpoints associated with the plurality of microservices, wherein the syntactic parsing of the source code refrains from executing the source code, and wherein the source code comprises bytecode or binary code; generating, based on generating traces and capturing interactions during an operation of the corresponding microservice, a second plurality of endpoints associated with the plurality of microservices; integrating the first plurality of endpoints and the second plurality of endpoints to generate a combined plurality of endpoints; determining, based on the combined plurality of endpoints, the end-to-end test coverage metrics for the plurality of end-to-end tests; and reconfiguring, based on the end-to-end test coverage metrics, at least one of the plurality of end-to- end tests or at least one of the plurality of microservices.

[0225] A17. The method of solution A16, comprising: generating a visualization of the microservice system.

[0226] A18. The method of solution A16, wherein integrating the first plurality of endpoints and the second plurality of endpoints comprises: extracting, from the first plurality of endpoints, a first set of metadata attributes for the corresponding microservice; extracting, from the traces, a second set of metadata attributes; and comparing the first set of metadata attributes and the second set of metadata attributes to generate the combined plurality of endpoints.

[0227] A19. The method of solution A18, wherein the first set of metadata attributes comprise a path, a request type, and a parameter lists associated with the corresponding microservice.

[0228] A20. The method of solution A16, wherein the end-to-end test coverage metrics comprise a method coverage metric, a path coverage metric, a data entity coverage metric, a data functionality coverage metric, or a frequency coverage metric.

[0229] A21. The method of solution A16, wherein the syntactic parsing of the source code comprises a bytecode inspection and a call graph generation.

[0230] A22. The method of solution A21 , wherein the bytecode inspection separates a control flow, a data flow, and bytecode instructions from the source code, and wherein the bytecode inspection is application platform-independent.

[0231] A23. The method of solution A21 , wherein the call graph generation produces all paths for each microservice of the plurality of microservices and communication paths between different instances of two or more of the plurality of microservices.

[0232] A24. A system for generating test coverage metrics for a test suite comprising a plurality of tests, the system comprising: a microservice system comprising a plurality of microservices that interact to perform an overall application function, wherein each microservice of the plurality of microservices is (a) associated with at least one endpoint and at least one method, and (b) configured to perform a partial function of the overall application function, wherein the at least one endpoint of a corresponding microservice enables a user or an alternative microservice to interact with the corresponding microservice, wherein the at least one method of the corresponding microservice enables the user or the alternative microservice to utilize one or more functions of the corresponding microservice; and one or more processors configured to: extract the at least one endpoint and the at least one method from each corresponding microservice of the plurality of microservices, determine, based on the at least one endpoint and the at least one method from each corresponding microservice, the test coverage metrics, and reconfigure, based on the test coverage metrics, at least one of the plurality of tests or at least one of the plurality of microservices.

[0233] A25. The system of solution A24, wherein the plurality of tests comprise end-to-end (E2E) tests, and wherein the test suite interacts with the at least oneendpoint through a user interface (III) and an application programming interface (API) gateway.

[0234] A26. The system of solution A24, wherein the plurality of tests comprise application programming interface (API) tests, and wherein the test suite interacts with the at least one endpoint directly through an API gateway.

[0235] A27. The system of solution A25 or A26, wherein the test coverage metrics comprise a test coverage metric for a microservice (Cms) of the plurality of microservices that is determined based on a number of tested endpoints associated with the microservice and a number of endpoints associated with the microservice.

[0236] A28. The system of solution A25 or A26, wherein the test coverage metrics comprise a test coverage metric for a test (Ctest) of the test suite that is determined based on a number of tested endpoints associated with the test, a number of the plurality of microservices, and a total number of endpoints associated with all microservices of the plurality of microservices.

[0237] A29. The system of solution A25 or A26, wherein the test coverage metrics comprise a test coverage metric for the test suite (Csllite) that is determined as:

[0238] Herein, t_total is a number of the plurality of tests, wherein m_total is a number of the plurality of microservices,| is a total number of tested endpoints associated with all tests of the plurality of tests, andisatotal number of endpoints associated with all microservices of the plurality of microservices.

[0239] A30. The system of solution A25 or A26, wherein the test coverage metrics comprise a method coverage metric for a microservice (iMCallms) of the plurality of microservices that is determined based on a number of tested methods in the microservice and a number of available methods in the microservice.

[0240] A31. The system of solution A25 or A26, wherein each microservice is associated with at least one path, wherein a path of the corresponding microservice enables the user or the alternative microservice to access the one or more functions of the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one path from each corresponding microservice of the plurality of microservices; and determine, based on the at least one path, a path coverage metric.

[0241] A32. The system of solution A31 , wherein the path coverage metric for a microservice (iPCms) of the plurality of microservices is determined based on a number of paths retrieved from call graphs generated from the test suite and a number of paths available for the microservice.

[0242] A33. The system of solution A25 or A26, wherein each microservice is associated with at least one data entity, wherein a data entity of the corresponding microservice is data that is (a) accessible to the user or the alternative microservice and (b) owned and controlled by the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one data entity from each corresponding microservice of the plurality of microservices; and determine, based on the at least one data entity, a data entity coverage metric.

[0243] A34. The system of solution A33, wherein the data entity coverage metric for a microservice (iDECms) of the plurality of microservices is determined based on a number of data entities retrieved from call graphs generated from the test suite and a number of data entities available to the microservice.

[0244] A35. The system of solution A25 or A26, wherein each microservice is associated with at least one data operation, wherein a data operation of the corresponding microservice enables the user or the alternative microservice to write, update, or delete data associated with a data entity, and wherein the one or more processors is configured to: extract the at least one data operation from each corresponding microservice of the plurality of microservices; and determine, based on the at least one data operation, a data functionality coverage metric.

[0245] A36. The system of solution A35, wherein the data functionality coverage metric for a microservice (iDFCms) of the plurality of microservices is determined based on a number of data operations retrieved from call graphs generated from the test suite and a number of data operations from a static analysis of the microservice.

[0246] A37. The system of any of solutions A24 to A36, comprising: a visualization model configured to generate a visualization of the microservice system.

[0247] A38. The system of any of solutions A24 to A36, wherein extracting the at least one endpoint and at the at least one method is based on a syntactic parsing of source code of the corresponding microservice, wherein the syntactic parsing of the source code refrains from executing the source code, and wherein the source code comprises bytecode or binary code.

[0248] A39. The system of any of solutions A24 to A36, wherein extracting the at least one endpoint and the at least one method is based on generating traces and capturing interactions during an operation of the corresponding microservice.

[0249] A40. The system of solution A39, wherein the traces are organized into a hierarchical tree, thereby enabling an analysis of the interactions during the operation of the corresponding microservice.

[0250] A41. The system of solution A40, wherein the analysis comprises a trace filtration mechanism.

[0251] A42. The system of any of solutions A38 to A41 , wherein extracting the at least one endpoint generates metadata attributes thereof, and wherein the metadata attributes comprise a path, a Hypertext Transfer Protocol (HTTP) method, an HTTP with encryption and verification (HTTPS) method, one or more parameters, or a return type associated with the corresponding microservice.

[0252] A43. A method for generating test coverage metrics for a test suite comprising a plurality of tests for a microservice system that includes a plurality of microservices, the method comprising: generating, based on a static analysis of source code of a corresponding microservice, a first plurality of endpoints and a first plurality of coverage parameters associated with the plurality of microservices, wherein the source code comprises bytecode or binary code, and wherein the first plurality of coverage parameters comprises a first plurality of methods, a first plurality of paths, a first plurality of data entities, or a first plurality of data operations, wherein an endpoint of the corresponding microservice enables a user or an alternative microservice to interact with the corresponding microservice, wherein a method of the corresponding microservice enables the user or the alternative microservice to utilize one or more functions of the corresponding microservice, wherein a path of the corresponding microservice enables the user or the alternative microservice to access the one or more functions of the corresponding microservice, wherein a data entity of the corresponding microservice is data that is (a) accessible to the user or the alternative microservice and (b) owned and controlled by the corresponding microservice, and wherein a data operation of the corresponding microservice enables the user or the alternative microservice to write, update, or delete the data associated with the data entity; generating, based on a dynamic analysis of the corresponding microservice, a second plurality of endpoints and a second plurality of coverage parameters associated withthe plurality of microservices; integrating the first plurality of endpoints and the second plurality of endpoints to generate a combined plurality of endpoints; integrating the first plurality of coverage parameters and the second plurality of coverage parameters to generate a combined plurality of coverage parameters; determining, based on the combined plurality of endpoints and the combined plurality of coverage parameters, the test coverage metrics for the plurality of tests; and reconfiguring, based on the test coverage metrics, at least one of the plurality of tests or at least one of the plurality of microservices.

[0253] A44. The method of solution A43, comprising: generating a visualization of the microservice system.

[0254] A45. The method of solution A43, wherein the dynamic analysis comprises generating traces and capturing interactions during an operation of the corresponding microservice.

[0255] A46. The method of solution A45, wherein the dynamic analysis comprises injecting instrumentation code into the microservice system to enable log generation, error checking, bug discovery, performance evaluation, and selective checking.

[0256] A47. The method of solution A46, wherein injecting the instrumentation code into the microservice system further enables trace span insertion, control flow graph generation, trace data collection, or trace data filtration.

[0257] A48. The method of solution A45, wherein integrating the first plurality of endpoints and the second plurality of endpoints comprises: extracting, from the first plurality of endpoints, a first set of metadata attributes for the corresponding microservice; extracting, from the traces, a second set of metadata attributes; and comparing the first set of metadata attributes and the second set of metadata attributes to generate the combined plurality of endpoints.

[0258] A49. The method of solution A48, wherein the first set of metadata attributes comprise a path, a request type, and a parameter lists associated with the corresponding microservice.

[0259] A50. The method of solution A43, wherein the test coverage metrics comprise an end-to-end test coverage metric, a method coverage metric, a path coverage metric, a data entity coverage metric, a data functionality coverage metric, or a frequency coverage metric.

[0260] A51 . The method of solution A43, wherein the static analysis comprises a syntactic parsing of the source code of the corresponding microservice.

[0261] A52. The method of solution A51 , wherein the syntactic parsing comprises inspecting an intermediate code representation of the source code and generating a call graph.

[0262] A53. The method of solution A52, wherein inspecting the intermediate code representation comprises: separating a control flow and a data flow; and performing an instrumentation of instructions to the source code, wherein inspecting the intermediate code representation is application platform-independent.

[0263] A54. The method of solution A52, wherein generating the call graph produces all paths for each microservice of the plurality of microservices and communication paths between different instances of two or more of the plurality of microservices.

[0264] FIG. 10 shows an example of a hardware platform 1000 that can be used to implement some of the techniques described in the present patent document. For example, the hardware platform 1000 may implement the various modules and algorithms described herein. The hardware platform 1000 may include a processor 1002 that can execute code to implement a method. The hardware platform 1000 may include a memory 1004 that may be used to store processor-executable code and / or store data. The hardware platform 1000 may further include a static analyzer 1006, a dynamic analyzer 1008, and a metric calculator 1010, which can communicate with the processor 1002. In some embodiments, the processor 1002 may include one or more processors implementing at least a portion of the static analyzer 1006, the dynamic analyzer 1008, and / or the metric calculator 1008. The processor 1002 may be configured to implement instrumentation, trace generation, syntactic parsing and other software testing algorithms. In some embodiments, the memory 1004 may include multiple memories, some of which are exclusively used by the processor 1002 when implementing the static analyzer, metric analyzer, and metric calculator algorithms.

[0265] Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0266] Part of the disclosed subject matter in this specification can be implemented as one or more computer program products, i.e. , one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing unit” or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g. , code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0267] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0268] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. Processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0269] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0270] While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0271] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0272] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document

Claims

WHAT IS CLAIMED IS:

1. A system for generating test coverage metrics for a test suite comprising a plurality of tests, the system comprising: a microservice system comprising a plurality of microservices that interact to perform an overall application function, each microservice of the plurality of microservices being associated with at least one endpoint and configured to perform a partial function of the overall application function, wherein the at least one endpoint of a corresponding microservice enables a user or an alternative microservice to interact with the corresponding microservice; and one or more processors configured to: extract the at least one endpoint from each corresponding microservice of the plurality of microservices, determine, based on the at least one endpoint from each corresponding microservice, end-to-end test coverage metrics, and reconfigure, based on the end-to-end test coverage metrics, at least one end-to-end test of the plurality of tests or at least one of the plurality of microservices.

2. The system of claim 1 , wherein an end-to-end coverage metric for a microservice (Cms) of the plurality of microservices is determined based on a number of tested endpoints associated with the microservice and a number of endpoints associated with the microservice.

3. The system of claim 1 , wherein an end-to-end coverage metric for an end-to-end test (Ctest) of the test suite is determined based on a number of tested endpoints associated with the end-to-end test, a number of the plurality of microservices, and a total number of endpoints associated with all microservices of the plurality of microservices.

4. The system of claim 1 , wherein an end-to-end coverage metric for the test suite (Csuite) is determined as:wherein t_total is a number of the at least one end-to-end test, wherein m_total is a number of the plurality of microservices, wherein| 's a tota' number of tested endpoints associated with all of the at least one end-to-end test of the plurality of tests, and wherein | \j' -LotalEms(J| is a total number of endpoints associated with all microservices of the plurality of microservices.

5. The system of claim 1 , wherein each microservice is associated with at least one method, wherein the at least one method of the corresponding microservice enables the user or the alternative microservice to utilize one or more functions of the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one method from each corresponding microservice; and determine, based on the at least one method, a method coverage metric for a microservice (iMCallms) of the plurality of microservices that is determined based on a number of tested methods in the microservice and a number of available methods in the microservice.

6. The system of claim 1 , wherein each microservice is associated with at least one path, wherein the at least one path of the corresponding microservice enables the user or the alternative microservice to access one or more functions of the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one path from each corresponding microservice; and determine, based on the at least one path, a path coverage metric for a microservice (iPCms) of the plurality of microservices that is determined based on a number of paths retrieved from call graphs generated from the test suite and a number of paths available for the microservice.

7. The system of claim 1 , wherein each microservice is associated with at least one data entity, wherein the at least one data entity of the corresponding microservice is data that is (a) accessible to the user or the alternative microservice and (b) owned and controlled by the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one data entity from each corresponding microservice; anddetermine, based on the at least one data entity, a data entity coverage metric for a microservice (iDECms) of the plurality of microservices that is determined based on a number of data entities retrieved from call graphs generated from the test suite and a number of data entities available to the microservice.

8. The system of claim 1 , wherein each microservice is associated with at least one data operation, wherein the at least one data operation of the corresponding microservice enables the user or the alternative microservice to write, update, or delete data associated with a data entity, and wherein the one or more processors is configured to: extract the at least one data operation from each corresponding microservice; and determine, based on the at least one data operation, a data functionality coverage metric for a microservice (iDFCms) of the plurality of microservices that is determined based on a number of data operations retrieved from call graphs generated from the test suite and a number of data operations from a static analysis of the microservice.

9. The system of any of claims 1 to 8, comprising: a visualization model configured to generate a visualization of the microservice system.

10. The system of claim 9, wherein the visualization comprises each endpoint of each of the plurality of microservices, and wherein an endpoint covered by the at least one end-to-end test is colored using a first color and an endpoint missed by each of the at least one end-to-end test is colored using a second color.

11. The system of claim 9, wherein the visualization comprises a service dependency graph that represents the microservice system, and wherein each microservice is a node of the service dependency graph and a dependency between two microservices is an edge between two nodes corresponding to the two microservices.

12. The system of claim 1 1 , wherein the node representing a particular microservice is colored based on a coverage percentage of the particular microservice.

13. The system of any of claims 1 to 8, wherein extracting the at least one endpoint is based on a syntactic parsing of source code of the corresponding microservice, wherein the syntactic parsing of the source code refrains from executing the source code, and wherein the source code comprises bytecode or binary code.

14. The system of any of claims 1 to 8, wherein extracting the at least one endpoint is based on generating traces and capturing interactions during an operation of the corresponding microservice.

15. The system of claim 13 or 14, wherein extracting the at least one endpoint generates metadata attributes thereof, and wherein the metadata attributes comprise a path, a Hypertext Transfer Protocol (HTTP) method, one or more parameters, or a return type associated with the corresponding microservice.

16. A method for generating end-to-end test coverage metrics for a test suite comprising a plurality of end-to-end tests for a microservice system that includes a plurality of microservices, the method comprising: generating, based on a syntactic parsing of source code of a corresponding microservice, a first plurality of endpoints associated with the plurality of microservices, wherein the syntactic parsing of the source code refrains from executing the source code, and wherein the source code comprises bytecode or binary code; generating, based on generating traces and capturing interactions during an operation of the corresponding microservice, a second plurality of endpoints associated with the plurality of microservices; integrating the first plurality of endpoints and the second plurality of endpoints to generate a combined plurality of endpoints; determining, based on the combined plurality of endpoints, the end-to-end test coverage metrics for the plurality of end-to-end tests; and reconfiguring, based on the end-to-end test coverage metrics, at least one of the plurality of end-to-end tests or at least one of the plurality of microservices.

17. The method of claim 16, comprising: generating a visualization of the microservice system.

18. The method of claim 16, wherein integrating the first plurality of endpoints and the second plurality of endpoints comprises: extracting, from the first plurality of endpoints, a first set of metadata attributes for the corresponding microservice; extracting, from the traces, a second set of metadata attributes; and comparing the first set of metadata attributes and the second set of metadata attributes to generate the combined plurality of endpoints.

19. The method of claim 18, wherein the first set of metadata attributes comprise a path, a request type, and a parameter lists associated with the corresponding microservice.

20. The method of claim 16, wherein the end-to-end test coverage metrics comprise a method coverage metric, a path coverage metric, a data entity coverage metric, a data functionality coverage metric, or a frequency coverage metric.

21. The method of claim 16, wherein the syntactic parsing of the source code comprises a bytecode inspection and a call graph generation.

22. The method of claim 21 , wherein the bytecode inspection separates a control flow, a data flow, and bytecode instructions from the source code, and wherein the bytecode inspection is application platform-independent.

23. The method of claim 21 , wherein the call graph generation produces all paths for each microservice of the plurality of microservices and communication paths between different instances of two or more of the plurality of microservices.

24. A system for generating test coverage metrics for a test suite comprising a plurality of tests, the system comprising: a microservice system comprising a plurality of microservices that interact to perform an overall application function, wherein each microservice of the plurality of microservices is (a) associated with at least one endpoint and at least one method, and (b) configured to perform a partial function of the overall application function, wherein the at least one endpoint of a corresponding microservice enables a user or an alternative microservice to interact with the corresponding microservice,wherein the at least one method of the corresponding microservice enables the user or the alternative microservice to utilize one or more functions of the corresponding microservice; and one or more processors configured to: extract the at least one endpoint and the at least one method from each corresponding microservice of the plurality of microservices, determine, based on the at least one endpoint and the at least one method from each corresponding microservice, the test coverage metrics, and reconfigure, based on the test coverage metrics, at least one of the plurality of tests or at least one of the plurality of microservices.

25. The system of claim 24, wherein the plurality of tests comprise end-to-end (E2E) tests, and wherein the test suite interacts with the at least one endpoint through a user interface (III) and an application programming interface (API) gateway.

26. The system of claim 24, wherein the plurality of tests comprise application programming interface (API) tests, and wherein the test suite interacts with the at least one endpoint directly through an API gateway.

27. The system of claim 25 or 26, wherein the test coverage metrics comprise a test coverage metric for a microservice (Cms) of the plurality of microservices that is determined based on a number of tested endpoints associated with the microservice and a number of endpoints associated with the microservice.

28. The system of claim 25 or 26, wherein the test coverage metrics comprise a test coverage metric for a test Ctest) of the test suite that is determined based on a number of tested endpoints associated with the test, a number of the plurality of microservices, and a total number of endpoints associated with all microservices of the plurality of microservices.

29. The system of claim 25 or 26, wherein the test coverage metrics comprise a test coverage metric for the test suite (Csuite) that is determined as:wherein t_total is a number of the plurality of tests, wherein m_total is a number of the plurality of microservices, wherein| 's a tota' number of tested endpoints associated with all tests of the plurality of tests, and wherein | \j' -LotalEms(J| is a total number of endpoints associated with all microservices of the plurality of microservices.

30. The system of claim 25 or 26, wherein the test coverage metrics comprise a method coverage metric for a microservice (iMCallms) of the plurality of microservices that is determined based on a number of tested methods in the microservice and a number of available methods in the microservice.

31. The system of claim 25 or 26, wherein each microservice is associated with at least one path, wherein a path of the corresponding microservice enables the user or the alternative microservice to access the one or more functions of the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one path from each corresponding microservice of the plurality of microservices; and determine, based on the at least one path, a path coverage metric.

32. The system of claim 31 , wherein the path coverage metric for a microservice (iPCms) of the plurality of microservices is determined based on a number of paths retrieved from call graphs generated from the test suite and a number of paths available for the microservice.

33. The system of claim 25 or 26, wherein each microservice is associated with at least one data entity, wherein a data entity of the corresponding microservice is data that is (a) accessible to the user or the alternative microservice and (b) owned and controlled by the corresponding microservice, and wherein the one or more processors is configured to: extract the at least one data entity from each corresponding microservice of the plurality of microservices; and determine, based on the at least one data entity, a data entity coverage metric.

34. The system of claim 33, wherein the data entity coverage metric for a microservice (iDECms) of the plurality of microservices is determined based on a number of data entities retrieved from call graphs generated from the test suite and a number of data entities available to the microservice.

35. The system of claim 25 or 26, wherein each microservice is associated with at least one data operation, wherein a data operation of the corresponding microservice enables the user or the alternative microservice to write, update, or delete data associated with a data entity, and wherein the one or more processors is configured to: extract the at least one data operation from each corresponding microservice of the plurality of microservices; and determine, based on the at least one data operation, a data functionality coverage metric.

36. The system of claim 35, wherein the data functionality coverage metric for a microservice (iDFCms) of the plurality of microservices is determined based on a number of data operations retrieved from call graphs generated from the test suite and a number of data operations from a static analysis of the microservice.

37. The system of any of claims 24 to 36, comprising: a visualization model configured to generate a visualization of the microservice system.

38. The system of any of claims 24 to 36, wherein extracting the at least one endpoint and at the at least one method is based on a syntactic parsing of source code of the corresponding microservice, wherein the syntactic parsing of the source code refrains from executing the source code, and wherein the source code comprises bytecode or binary code.

39. The system of any of claims 24 to 36, wherein extracting the at least one endpoint and the at least one method is based on generating traces and capturing interactions during an operation of the corresponding microservice.

40. The system of claim 39, wherein the traces are organized into a hierarchical tree, thereby enabling an analysis of the interactions during the operation of the corresponding microservice.

41. The system of claim 40, wherein the analysis comprises a trace filtration mechanism.

42. The system of any of claims 38 to 41 , wherein extracting the at least one endpoint generates metadata attributes thereof, and wherein the metadata attributes comprise a path, a Hypertext Transfer Protocol (HTTP) method, an HTTP with encryption and verification (HTTPS) method, one or more parameters, or a return type associated with the corresponding microservice.

43. A method for generating test coverage metrics for a test suite comprising a plurality of tests for a microservice system that includes a plurality of microservices, the method comprising: generating, based on a static analysis of source code of a corresponding microservice, a first plurality of endpoints and a first plurality of coverage parameters associated with the plurality of microservices, wherein the source code comprises bytecode or binary code, and wherein the first plurality of coverage parameters comprises a first plurality of methods, a first plurality of paths, a first plurality of data entities, or a first plurality of data operations, wherein an endpoint of the corresponding microservice enables a user or an alternative microservice to interact with the corresponding microservice, wherein a method of the corresponding microservice enables the user or the alternative microservice to utilize one or more functions of the corresponding microservice, wherein a path of the corresponding microservice enables the user or the alternative microservice to access the one or more functions of the corresponding microservice, wherein a data entity of the corresponding microservice is data that is (a) accessible to the user or the alternative microservice and (b) owned and controlled by the corresponding microservice, and wherein a data operation of the corresponding microservice enables the user or the alternative microservice to write, update, or delete the data associated with the data entity;generating, based on a dynamic analysis of the corresponding microservice, a second plurality of endpoints and a second plurality of coverage parameters associated with the plurality of microservices; integrating the first plurality of endpoints and the second plurality of endpoints to generate a combined plurality of endpoints; integrating the first plurality of coverage parameters and the second plurality of coverage parameters to generate a combined plurality of coverage parameters; determining, based on the combined plurality of endpoints and the combined plurality of coverage parameters, the test coverage metrics for the plurality of tests; and reconfiguring, based on the test coverage metrics, at least one of the plurality of tests or at least one of the plurality of microservices.

44. The method of claim 43, comprising: generating a visualization of the microservice system.

45. The method of claim 43, wherein the dynamic analysis comprises generating traces and capturing interactions during an operation of the corresponding microservice.

46. The method of claim 45, wherein the dynamic analysis comprises injecting instrumentation code into the microservice system to enable log generation, error checking, bug discovery, performance evaluation, and selective checking.

47. The method of claim 46, wherein injecting the instrumentation code into the microservice system further enables trace span insertion, control flow graph generation, trace data collection, or trace data filtration.

48. The method of claim 45, wherein integrating the first plurality of endpoints and the second plurality of endpoints comprises: extracting, from the first plurality of endpoints, a first set of metadata attributes for the corresponding microservice; extracting, from the traces, a second set of metadata attributes; and comparing the first set of metadata attributes and the second set of metadata attributes to generate the combined plurality of endpoints.

49. The method of claim 48, wherein the first set of metadata attributes comprise a path, a request type, and a parameter lists associated with the corresponding microservice.

50. The method of claim 43, wherein the test coverage metrics comprise an end-to- end test coverage metric, a method coverage metric, a path coverage metric, a data entity coverage metric, a data functionality coverage metric, or a frequency coverage metric.

51. The method of claim 43, wherein the static analysis comprises a syntactic parsing of the source code of the corresponding microservice.

52. The method of claim 51 , wherein the syntactic parsing comprises inspecting an intermediate code representation of the source code and generating a call graph.

53. The method of claim 52, wherein inspecting the intermediate code representation comprises: separating a control flow and a data flow; and performing an instrumentation of instructions to the source code, wherein inspecting the intermediate code representation is application platformindependent.

54. The method of claim 52, wherein generating the call graph produces all paths for each microservice of the plurality of microservices and communication paths between different instances of two or more of the plurality of microservices.

Images