A testing framework that supports precise selection of microservices testing in cloud environments

Abstract

The present disclosure relates to computer-implemented methods, software, and systems for test selection performed by microservices in a cloud environment. Metadata of a set of changed files is obtained. The set of changed files is to be deployed in a software product and stored in a source code repository. The metadata of the set of changed files and the content of at least one of the changed files are analyzed based on a set of rules to determine a subset of tests of a default test plan to be executed. The subset of tests is executed at a test landscape that runs a set of software components associated with the set of changed files.

Description

Technical Field

[0001] The present disclosure relates to computer-implemented methods, software, and systems for managing the design and testing of software solutions. Background Art

[0002] Software complexity is increasing, leading to changes in the lifecycle management and maintenance of software applications, databases, and platform systems. In software development, testing is often associated with complex systems and solutions that require time and resources to ensure high-quality results. Quality engineers are often tasked with testing complex software solutions that include various combinations of components integrated in different ways. Summary of the Invention

[0003] Embodiments of the present disclosure generally relate to a computer-implemented method for selecting tests for execution on microservices in a cloud environment.

[0004] An example method may include operations such as: obtaining metadata for a set of changed files, wherein the set of changed files is to be deployed in a software product and stored at a source code repository; analyzing the metadata for the set of changed files and content of at least one of the changed files based on a rule set to determine a subset of tests of a default test plan to execute; and executing the subset of tests at a test landscape running a set of software components associated with the set of changed files.

[0005] The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations according to embodiments of the methods provided herein.

[0006] The present disclosure also provides a system for implementing the method provided herein. The system includes one or more processors and a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations according to embodiments of the method provided herein.

[0007] It should be understood that the method according to the present disclosure may include any combination of aspects and features described herein. That is, the method according to the present disclosure is not limited to the combination of aspects and features specifically described herein, but also includes any combination of aspects and features provided.

[0008] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the present disclosure will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram depicting an example computer-implemented system that may perform embodiments of the present disclosure according to embodiments of the present disclosure.

[0010] Figure 2A is a block diagram of an example computing environment including a testing framework that implements rules for selecting tests for execution on services running in a cluster on a cloud infrastructure, according to an embodiment of the present disclosure.

[0011] Figure 2B is a block diagram of an example computing environment in which selected tests for a service are performed in a temporary test landscape according to an embodiment of the present disclosure.

[0012] Figure 2C is a block diagram of an example computing environment in which the load of a test is distributed for execution on different types of landscapes in a temporary test landscape, according to an embodiment of the present disclosure.

[0013] Figure 3 is a flow chart of a method for selecting tests to execute on a service running in a cluster on a cloud infrastructure according to an embodiment of the present disclosure.

[0014] Figure 4 is a flow chart of a method for testing load distribution according to an embodiment of the present disclosure.

[0015] Figure 5 is a schematic diagram of an example computer system that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] This disclosure describes various tools and techniques for enabling selection and distribution of tests to run on microservices executed on ephemeral landscapes of various infrastructure landscape types.

[0017] In some embodiments, a landscape can be defined as a collection of clusters (e.g., a Kubernetes cluster), where each cluster can be associated with a type. In some instances, a cluster type can provide a defined set of services. For example, a cluster can be product-specific and include both microservices and general-purpose services used in the product. In another example, some clusters (e.g., general-purpose clusters) may host only general-purpose services that may not be tied to a product but may be used by product-specific features (such as those implemented in other clusters). In such an instance, a service from one cluster may invoke a command to be executed in another cluster (e.g., thereby performing an action in a separate cluster). Such remote commands requesting an action to be executed "outside" of a specific calling cluster may be slower and may provide fewer options for controlling security and scalability. Therefore, in some cases, a cluster can combine product-specific and general-purpose services to improve service performance. For example, logging is a fairly throughput-intensive operation, and outbound network throughput is a significant cost factor. Therefore, a logging service can be installed in each product cluster to reduce resource expenditure.

[0018] Software testing is an activity to ensure and improve the quality of software products during the software development process. When performing a test, a test set can be defined and configured based on the details of the software project. In some embodiments, a software project can combine multiple software components (e.g., services or microservices) to run together and provide resources to customers. In some cases, an existing software project can be modified, e.g., the source code can be changed. This change to the software project can be automatically and dynamically associated with the start of a test plan. In some embodiments, the modification of a part of a software project may not require all available tests for executing the project. In some instances, such execution may be too time-consuming, resource-intensive, and unnecessary for solving or identifying problems.

[0019] In some embodiments, if a production system is to be tested in a fully automated or substantially fully automated manner, a copy of the production system may be made, which may be a smaller copy to avoid interruptions or data loss. Setting up tests for full-scale production testing can be resource intensive and may require a significant amount of time to prepare the test effort and computing resources to execute. Therefore, it is desirable to minimize the test effort or at least reduce it to a meaningful set of test operations that can produce good results when identifying issues associated with changes made to a software project.

[0020] According to an embodiment of the present disclosure, when a change is made to a software product, metadata associated with the change can be created, for example, at a source code repository storing the code for the software product. The metadata can be evaluated based on an implemented set of rules to determine which parts of the software project are to be tested and to identify tests associated with the changes made.

[0021] In some implementations, an overall pipeline can be defined with tests that include landscape creation, instance provisioning, landscape upgrades, testing of landscapes and instances, instance de-provisioning, and landscape deletion in the correct order.

[0022] The pipeline can be used to determine which tests are relevant or which tests can be skipped from execution based on a notified change to a product or service. In some embodiments, a test framework comprising rules can be defined. When used, these rules can automatically decide which tests to skip based on default settings for interpreting flags added to changes (e.g., flags can be added by the user modifying the source code, by automatic execution of changes provided by the system, or in other ways), situational rules, and / or additional user-provided instructions for executing the tests (e.g., to override automatic decisions or redefine decisions through automatic rules (e.g., by setting additional flags for the changes made)). In some instances, a test pipeline can be constructed with a sequential initialization and unit test portion and a parallel integration test portion, where several landscape setups are tested in parallel. The landscape setup for each test in the integration test phase can execute a defined pipeline process, including landscape creation, testing, and deletion.

[0023] According to an embodiment of the present disclosure, components of a software product or a corresponding software product are defined as being relevant for testing, and a test set for each test unit is identified. Components can be tested when they are run on various infrastructure landscape types. For example, landscapes can be categorized into different types based on the infrastructure provider used for the landscape, the geographic location of the landscape, the type or size of clusters deployed at the landscape, etc. In some cases, some landscape types may be associated with issues that would not be identified if the component were run on another landscape type (e.g., a different landscape associated with a different infrastructure provider and / or hosting a different type of cluster). The distribution of components (e.g., services or products) across landscape types can be performed based on distribution logic that relies on distributing the entities to be tested to specific landscape types for each given time period when scheduling tests. For example, a selector for a landscape type for each service or product can be defined for a time period such as a month, day, week, set number of hours, etc.

[0024] This embodiment supports the automatic detection of the changes in the files of software projects and the adjustment of test strategies, and defines fewer or more tests depending on the content of the changes of files. The selection logic for testing the specific parts of software projects can allow the system of changes to be accurately tested, without the need to run the test of the unchanged parts of the software projects, which is more effective and saves costs. Test execution can be dispersed in the different landscape types that support accurate testing while reducing test costs. For example, using the test framework as described in the present disclosure can support that test workload is distributed to different infrastructure providers (for example, infrastructure as a service, such as hyperscaler or other infrastructure providers). For example, most microservices do not have any embodiment specific to hyperscaler, and can be tested on any cloud infrastructure (such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP) etc.) provided by a given provider. This distribution logic supports the effective workload distribution of the microservices on different infrastructure providers. This may cause the cost and time of test operation to be effectively distributed, while still covering all providers in database infrastructure testing.

[0025] In some instances, a landscape consisting of stateless microservices may offer the option of including new versions of existing microservices that deviate only minimally from currently active services. For example, a web service receiving traffic could be modified to generate a new version. For this new version, 1% of traffic could be routed to the new version of the web service, while 99% of traffic would remain with the current stable version of the web service. If this split in traffic causes alarms, increases failure rates, or any other issues are detected, the new version of the system can be terminated. In this scenario, 1% of requests might fail, but if clients retry their requests to the service, they will receive valid responses from the current stable version of the web service. Defining such a new version of a web service and using it in production scenarios can be an inexpensive test setup without the additional hardware costs of testing. However, such a test scenario may not be suitable for stateful services, such as a DBMS with a persistent backend. In such cases, if 1% of database instances are used with the new microservice version, and if that one instance fails and compromises the persistence of the remaining 1%, data could be irreversibly lost.

[0026] In some instances, when testing stateful services, testing techniques can be performed before any actual production use of the service, and a smaller version of the entire production system can be created. Such duplication of the production system can be expensive, and therefore techniques for defining a portion of the production landscape relevant to testing can be used so that only those aspects of the system relevant to testing can be defined in the hardware setup (size and number) and / or test execution (e.g., usage time). Such a hardware setup can be defined to allow for estimation of performance changes. The cost will vary based on the size of the setup, the number of instances, and the time used to execute the tests (and maintain the setup). The larger the required hardware that needs to be used, or the longer the time required for testing, the higher the cost that the test setup will incur. By using the techniques of the present disclosure, a hardware setup is determined that can be used for testing and can provide accurate test evaluation results while minimizing the cost of test execution.

[0027] In software development, the application development life cycle includes various phases, where the development and testing phases may be iteratively performed two or more times to verify and potentially improve the quality of the final product. In some cases, Figure 1 1 is a block diagram depicting an example computer-implemented system 100 that can perform embodiments of the present disclosure according to embodiments of the present disclosure. In the depicted example, example system 100 includes client device 102, client device 104, network 110, computing environment 106, and computing environment 108. Computing environment 106 and / or 108 can be cloud computing environments that may include one or more server devices and databases (e.g., processors and memory). In the depicted example, user 114 interacts with client device 102, and user 116 interacts with client device 104.

[0028] In some examples, client device 102 and / or client device 104 can communicate with environment 106 and / or environment 108 via network 110. Client device 102 can include any suitable type of computing device, such as a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), a cellular phone, a network appliance, a camera, a smartphone, an Enhanced General Packet Radio Service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or a suitable combination of any two or more of these devices or other data processing devices. In some implementations, network 110 can include a large computer network, such as a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a telephone network (e.g., PSTN), or a suitable combination thereof, that connects any number of communication devices, mobile computing devices, fixed computing devices, and server systems.

[0029] In some implementations, environment 106 (and / or environment 108) may include at least one server and at least one data store 120. Figure 1 In the example of FIG, environment 106 is intended to represent various forms of servers, including but not limited to web servers, application servers, proxy servers, network servers, and / or server pools. Typically, a server system accepts requests for application services and provides such services to any number of client devices (e.g., client device 102 via network 110).

[0030] According to embodiments of the present disclosure, and as described above, environment 106 and / or environment 108 can host applications and databases running on a host infrastructure. In some instances, environment 106 can include multiple cluster nodes, which can represent physical or virtual machines that can represent a cluster of web nodes. The hosted applications and / or services can run on virtual machines (VMs) hosted on a cloud infrastructure.

[0031] In some cases, according to the present disclosure, environment 106 and / or environment 108 may provide infrastructure for hosting a data processing application that may be tested and further developed (e.g., where testing is defined as part of development, or the testing phase may provide input for further development tasks).

[0032] In some instances, environment 106 or 108 can host a test framework and can support the instantiation of test landscapes of various infrastructure landscape types, such as Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 3 and Figure 4 Those described.

[0033] Figure 2A is a block diagram of an example computing environment 200 including a testing framework 225 that implements rules for selecting tests for execution on services running in a cluster on a cloud infrastructure, according to an embodiment of the present disclosure.

[0034] Computing environment 200 includes infrastructure 205, which may include multiple landscapes that provide infrastructure as a service for running services in clusters. In some embodiments, a landscape, such as Landscape X 201, may be defined as an infrastructure landscape of a given type associated with an infrastructure provider, such as AWS, GCP, Microsoft Azure, etc. Multiple services, such as Service A, may run on Landscape X 201. For example, services on Landscape X 201 may run in a cluster, such as Cluster 210 (e.g., a Kubernetes cluster) that supports the deployment, scaling, and management of containerized applications. For example, Cluster 210 may allow containers to run across multiple machines and environments (e.g., virtual, physical, cloud-based, or on-premises). In some instances, clusters running on a landscape may not be restricted to a specific operating system and may share an operating system and run without restrictions. Computer environment 200 includes Landscape Z 202, which is a different infrastructure landscape than Landscape X 201. Landscape Z 202 may include clusters, each of which may be associated with a different stage of software development in a software project. In some embodiments, landscape Z 202 can define channels, where a channel such as channel 215 can be associated with a single software product (e.g., SAP HANA). A defined channel such as channel 215 can include multiple clusters. In some embodiments, when defining a test flow for a landscape (e.g., a particular landscape type), one or more channels can be defined, where each channel can be mapped to a different development phase, i.e., development, test, and production. A defined channel such as channel 215 can point to a corresponding landscape type, where functionality provided by a cluster can be tested on different infrastructures. For example, channel 215 can be a channel associated with the European region of a cloud-provided service, which can indicate the use of a landscape type associated with the European region for a temporary use case where testing can be performed.

[0035] In some cases, more landscapes can be defined on the infrastructure 205 where services can run in clusters and on channels based on custom definitions.

[0036] In some embodiments, infrastructure 205 is a setup involving the deployment, configuration, and management of deployment infrastructure services 220. In some instances, requests to instantiate new services at one or more landscapes on infrastructure 205 can be received and processed by deployment infrastructure services 220. In some embodiments, a testing framework 225 can be defined with respect to infrastructure 205 and software products that can run on infrastructure 205. In some embodiments, software products can be defined as components and services that can interact to provide resources to end users. Software products can be deployed based on compiling (or not compiling) source code stored at source code repository 230.

[0037] In some embodiments, a software project can be created at a source code repository 230 (e.g., Git), where the initial source code for the project can be stored and further subsequent code changes can be made. For example, when deployed and run at the infrastructure 205, changes can be made to the existing source code to modify the functionality of the software product. In some embodiments, when changes are made to the source code of the software project, one or more files can be changed. Changes made to the source code (and metadata) of the software project can be automatically recognized by the source code repository 230 to trigger a test workflow through the test framework 225. In some instances, the test framework 225 can include a test suite with pre-submitted tests associated with the software project created at the source code repository 230 and / or the software product (deployed software project) at the infrastructure 205.

[0038] In some implementations, the testing framework 225 may include a rule set 226 that defines rules for evaluating metadata (e.g., metadata 231) associated with changes to a portion of a software product. Figure 2B The logic of at least some test portions of the default test plan for the software product executed at the test landscape described in . The evaluation and determination of the tests and related portions of the software product can be performed at the test framework 225, such as Figure 3 The rule set 226 may include rules for mapping flags provided as metadata of changes associated with a software product to corresponding actions or selections of tests associated with the corresponding flags (and changes).

[0039] Figure 2B is a block diagram of an example computing environment in which selected tests for services are performed in a temporary landscape according to an embodiment of the present disclosure.

[0040] Figure 2B The computer environment can be basically similar to Figure 2A environment, where the infrastructure 240 may be similar to Figure 2A Infrastructure 205. In some instances, Landscape X 241 can be defined as a landscape type such as AWS, where clusters are defined to run microservices on a hyperscaler.

[0041] In some implementations, source code files can be modified at source code repository 250 (to generate modified files 251), where upon receiving the modifications, source code repository 250 can trigger a testing process, where a testing framework can identify metadata associated with the modifications made at source code repository 250 to cause files 215 to be tested using a rule set (similar to or substantially similar to Figure 2A and Figure 3 Based on the evaluation of the metadata, it can be determined that service A 245 is the service associated with the modified file 251 and that a set of tests 260 is related to the modification. The set of tests 260 can be defined as being related to testing, as described below with respect to Figure 3 As described. A test set 260 is executed on a test landscape instantiated as a temporary landscape 265 to match Landscape X 241 (e.g., to match the type of landscape type or the configuration or constraints of the infrastructure). The identified service A 245 can be run on temporary landscape X 265, which can execute the set of tests 260, and the test results can be obtained on the test framework 255. In some embodiments, temporary landscape X 265 can include the service in its version before the change was made (i.e., matching Landscape X 241), where temporary landscape X 265 can be updated to include the change (e.g., the change from modified file 251). In other embodiments, temporary landscape X 265 can be created directly with the new service version that includes the change from modified file 251, such that temporary landscape X 265 will be an updated version of landscape 241 given the change in modified file 251. In some instances, temporary landscape X 265 can further differ in size from landscape 241.

[0042] Figure 2C is a block diagram of an example computing environment in which the load of a test is distributed for execution on different types of landscapes in a temporary test landscape, according to an embodiment of the present disclosure. Figure 2C The computing environment includes Figure 2A Infrastructure 205 and Figure 2B The infrastructure 240 describes the different landscape types of landscapes.

[0043] In some embodiments, Figure 2C The computing environment includes Figure 2A and / or Figure 2B The infrastructure in which the test framework 291 may include Figure 2A 、 Figure 2B and Figure 3 The test suite and rule set are substantially similar to the test suite and rule set. The test framework 291 may include landscape distribution logic 290. The distribution logic 290 may correspond to the Figure 4Method 400 is provided to determine logic for selecting a landscape type to execute a given test for a given service or component. Distribution of tests for services across various types of infrastructure landscapes and over time can be performed in a manner that takes into account infrastructure load (e.g., preventing overloading of a particular type of infrastructure landscape) and conserves resources for test execution while maintaining accuracy of test results over time.

[0044] In some embodiments, based on the initiation of test execution (e.g., based on identification of a change in source code of a software product that includes one or more microservices), a set of services 275 to be tested can be determined, where the set of services is evaluated by the testing framework to determine where to distribute the set of services 275 so that those services are instantiated in the test landscape to execute the test. In some embodiments, Figure 2C The test framework 291 can be basically similar to Figure 2A and 2B The testing framework 291 may include logic for distributing services to be tested across different types of landscapes, such as Figure 4 In some embodiments, the landscape distribution logic 290 of the testing framework 291 may be configured to perform Figure 4 Method 400 of the landscape distribution logic 290. The landscape distribution logic 290 may include a hash 295 to store a distribution of services from the services 275 identified as being to be tested that are mapped with a landscape of landscape type. The distribution may be performed over discrete time periods, such as daily, weekly, or monthly, where tests to be performed over a given time period for a given service (or iteration of a schedule with a time period) may be selected to be associated with a given landscape type. The selection of the landscape type may be done based on a predefined probability for selecting each of the different landscape types supported for executing the test. The selection of the landscape type for each service may be performed by a selector 292 as part of the landscape distribution logic 290 of the testing framework.

[0045] In some embodiments, a mechanism for selecting a landscape type for a specific time period on a schedule for executing tests can be implemented to reduce testing costs by providing the ability to define sets of flags for different types of landscapes, where a flag can be selected for each time period, for example, one flag can be selected each month for a specific service or product. The selection of landscape type can be consistent for each individual month and service, where the selection of landscape type can be tied to the service name (e.g., individually or as a group of services) and / or other identifiers for a given service. In some cases, if multiple services are changed in a single pull request with changes received from a source code repository, each channel associated with the corresponding landscape type in the temporary landscape instantiated for testing will be used.

[0046] In some embodiments, a landscape type from a set of available landscape types can be selected for each service, such that each service can be mapped to a different landscape type. This mapping between landscape types and services can be rotated over a defined period (such as a month or other scheduling period). The selection of landscape types for services can be distributed taking into account the distribution ratio between different infrastructure providers. For example, landscapes associated with lower costs (e.g., cheaper infrastructure in the context of resources to be instantiated and / or maintained) can be tasked with performing more tests than higher-cost landscapes.

[0047] In some implementations, a service may define additional flags to indicate that certain infrastructure landscape types should be excluded from selection for the service. For example, a landscape type such as GCP may be defined as the only infrastructure provider that can be selected for a service configured as a GCP-only service, thus excluding any other landscape types that could be used for testing.

[0048] In some cases, channels may be defined in a similar manner to different types of landscapes and may be used to define test flows having temporary channels that substantially correspond to temporary landscapes as discussed herein.

[0049] For example, if services A, B, and C need to be verified to function as expected on landscape types X, Y, and Z, testing can be configured to include testing of service A on landscape X, service B on landscape Y, and service C on landscape Z. The configuration for testing can be defined for a set time period. For subsequent time periods (e.g., months), the mapping between services and landscapes can be modified (or reshuffled). In some examples, a service can be tested under a particular landscape type as long as there are three services running under that landscape type that are generally independent of that particular landscape type (e.g., services not defined as landscape-specific). By defining the mapping (or distribution) of services to landscape types, the number of tests performed can be reduced. For example, by defining that a service can run on a landscape with three other independent landscape services, the number of tests required to complete the test scope of "service x landscape" is reduced from nine services to "one per service" for three services. The ratio of services mapped to landscapes can be adjusted. For example, 50% of services can be verified on a less expensive landscape, while another landscape can be tested for only 25% of the other services, and a third landscape can be tested for the remaining 25% of the services. There can be other combinations and distributions of tests among multiple landscapes and distribution percentages can be predefined for the tests.

[0050] In some embodiments, a landscape selection 285 can be made for the service 275, and the service can be run on the selected landscape type in the temporary test landscape 280. The landscape can be instantiated and configured based on the selection made at 285. In some embodiments, a software product can be associated with the service to be tested, where, based on the selection, the software product 1 can be executed on landscape type X, while in production mode, the software product 1 can be run on landscape type Y. Landscape selection 285 can be made over time and on iterations scheduled for execution of tests of the software project's services (e.g., source code, metadata, configuration associated with the development project).

[0051] In response to the landscape selection, a collection of infrastructure landscape types can be created. Each of the collection of infrastructure landscape types, run as a temporary test landscape, can be configured to run a corresponding product from the set of products to be tested. In some instances, a product from the collection can include one or more of the services 275 to be tested, wherein all services 275 to be tested can be associated with one or more products.

[0052] In some embodiments, once the tests are executed and show successful execution of the corresponding services (e.g., meeting test output criteria), the set of tested products or services can be deployed, each in a production channel of the infrastructure landscape type on the cloud infrastructure landscape, on the corresponding infrastructure landscape type on the cloud infrastructure landscape 270. In some cases, the corresponding infrastructure landscape type selected for each product for testing can be different from or the same as the infrastructure landscape type on the cloud infrastructure landscape where the product is deployed in production mode.

[0053] Figure 3 is a flow chart of a method 300 for selecting a test to be executed on a service running in a cluster on a cloud infrastructure according to an embodiment of the present disclosure. In some embodiments, the method 300 may be performed in a manner similar to that described in relation to Figure 2A 、 Figure 2B and Figure 2C The selection of tests for execution may be performed at a testing framework that implements a set of rules to evaluate code changes and identify relevant tests for execution. The testing framework may be substantially similar to Figure 2A Testing framework 225.

[0054] At 310, metadata for a set of changed files is obtained. The changed files are associated with a software product, such as a service running in a cluster (e.g., Figure 2A The changed files are archived and will be deployed in a software product that can be provided to end users in production mode. The changed files can be stored in a source code repository (e.g., In some implementations, the source code repository may be a version control system that tracks changes to files and can be used to coordinate work between users (e.g., developers) who collaborate on and change files (including the source code for a software project) during the software development phase.

[0055] In some examples, in response to detecting a change to the source code, metadata is obtained at 310. The detection of the change can be performed by a source code repository. In some examples, the source code repository can provide metadata in response to receiving a change to the source code and / or metadata associated with the software product.

[0056] At 320, metadata of the set of changed files and content of at least one of the changed files are analyzed based on the rule set. In some implementations, the rule set may be substantially similar to Figure 2A A set of rules 226 is provided. Metadata is evaluated to determine a subset of the tests of the default test plan to be executed. In some embodiments, a test plan can be defined for a software product and / or for a service (including multiple services) that is part of a software product defined as a solution. A test plan can be defined for each software product and may not be specifically divided for individual services, but rather configured as a test suite to be executed. In some cases, when some source files of a software product change, only a subset of tests may need to be executed to target the file changes, thereby reducing test execution in a timely manner and reducing the load on the test infrastructure. In some instances, the evaluation of metadata may also include a request to obtain the content of at least some of the changed files to evaluate the source code based on the rule set. In some cases, the evaluation of metadata and the determination of the tests to be executed may be provided to a user in order to obtain user instructions for including additional flags or modifying the test plan. For example, user instructions may be received as additional input that can rewrite the rule set 226 used for the evaluation of metadata.

[0057] In some embodiments, the rule set may be as follows: Figure 2A In some examples, the code can be defined as described above and can include rules for skipping or performing testing of portions of a software product based on flags in changed portions of source code in a set of changed files. In some examples, changes to the source code can be annotated with flags that can be iteratively evaluated to determine how to execute tests associated with those changes. In some examples, flags included in the source code can be added while the source code is being changed by the developer.

[0058] At 330, a subset of tests is executed on a test landscape running the set of software components associated with the set of changed files. In some embodiments, executing the subset of tests may include creating a test landscape, which is a temporary landscape instantiated for test execution. A set of instances of software components associated with the changed files in the software project can be provided and run on the temporary test landscape. A test landscape can be defined as a configuration based on landscape-specific settings. When a test is to be executed, the landscape-specific settings can be checked to determine whether one or more tests can be executed on a particular landscape. In this case, it can be determined that the test cannot be executed on a particular test landscape, and the test landscape can be skipped for the particular test and another landscape can be used.

[0059] In some embodiments, the tests performed include unit tests that may be performed on portions of the software components associated with the change. Additionally, in some cases, integration tests may be performed, for example, in parallel, for integration between at least two software components. In some cases, the tests performed (unit tests and / or integration tests) may be part of a subset of tests defined based on a rule set and may be part of a default test plan defined for a product running on the cloud infrastructure.

[0060] In some embodiments, when a test is to be executed, a landscape type can be determined for executing the test on components in the set of software components. The landscape type can be selected from a set of landscape types defined for testing. Each landscape type corresponds to a type of infrastructure used to run the software component. In some examples, the test landscape is a cloud test landscape configured to host the software component as a service.

[0061] At 340, test results are obtained from the executed tests. In some embodiments, the tests can be executed on the first product while the first product is running in a landscape that can be a selected infrastructure landscape type. The selection of the landscape type can be performed taking into account the load distribution of the tests across various landscape types (e.g., different infrastructures that provide resources for running a service in a cloud environment). In some instances, the landscape types can be associated with different infrastructure providers and different technical details for executing the service. In some cases, it can be determined that the tests are to be executed in a landscape environment that includes landscapes of at least two different infrastructure types. In those cases, the components of the set of software components can be tested in parallel on the at least two landscapes. In some embodiments, after executing a subset of the tests on the test landscape, the test landscape can be deleted because the space can be used for other test landscapes for subsequent testing.

[0062] In some embodiments, the obtained test results can be evaluated to determine the performance of the first product. In some cases, it can be determined that the test has failed. In these cases, it is necessary to continue the test execution. In some cases, based on the test results, further changes can be made to the source code or configuration of the first product. The new changes can be used to redeploy a new version of the first product to run new tests and obtain results that can show successful execution. In some embodiments, the test execution and development tasks for improving the functionality can be iteratively performed until the test results meet the state of successful execution (for example, a threshold percentage (for example, 90%, 95% or 100%) of the tests executed can produce successful results).

[0063] At 350 , a determination is made to deploy a first product in production mode on an infrastructure landscape type (eg, a type based on a product definition, a user specification definition, or otherwise obtained).

[0064] Figure 4 is a flow chart of a method 400 for testing load distribution according to an embodiment of the present disclosure. In some embodiments, the method 400 may be performed in a manner similar to that described in relation to Figure 2A 、 Figure 2B and Figure 2C When testing is to be performed on a component of a software product, method 400 may be performed, wherein the test is determined for a specific change made in the source code associated with the component. In some embodiments, the method may be performed as described with respect to Figure 3 The test selection approach described above determines the tests to be performed and the relevant components of the tests.

[0065] At 410, a set of products to be tested is determined. The products to be tested may include, for example, Figure 2A 、 Figure 2B and Figure 2C The described cloud service software products. In some instances, each product in the collection includes a set of microservices. The collection of microservices can be deployed on a cloud infrastructure landscape such as Figure 2A Infrastructure 200, Figure 2B Infrastructure 240 or Figure 2C Infrastructure 270.

[0066] In some embodiments, a product associated with the test can be determined based on a received change to the source code of a collection of microservices. In some instances, a product can be defined as comprising one or more microservices that can be deployed to run in an integrated manner and provide resources to end users. In some embodiments, the collection of microservices can be deployed at a production landscape infrastructure of a given infrastructure landscape type in a collection of infrastructure landscape types. The collection of microservices can be tested when the collection of microservices is deployed with its previous state and each service is updated to the changed state as defined in the change. In some instances, the update may only be related to a portion of the microservices. The change can be made in the source code of the collection of microservices, which can change the performance of the product collection when providing services deployed on the cloud infrastructure landscape.

[0067] You can Figure 2A and Figure 3 The portion of the microservice associated with the change can be determined as described. Tests related to the portion of the microservice can be determined based on a set of rules for evaluating metadata associated with the change, such as those described with respect to Figure 2A 、 Figure 2B and Figure 3 described.

[0068] At 420, a set of infrastructure landscape types is determined for test execution. Types are determined to be associated with products running on the infrastructure. The set of landscape types can be configured with a probability of selection for each corresponding product. For example, each type can be associated with a predefined probability of selection for each product, where some types may not be available for one or more products (e.g., not supported for execution). For example, if there are two types of landscapes, such as Figure 2C For example, if there are two landscapes, landscape X and landscape Y, each type can be associated with a 50% chance of selection, which can provide an even distribution between the tests on the two types of landscapes, or each type can be associated with a different percentage, which defines a different chance of selection of the landscape type for a given test execution. In some embodiments, the defined selection probabilities can be used during scheduling of tests within a time period (such as monthly), and products can be associated with a particular landscape type based on selection of a corresponding set of landscape types, where the distribution of tests for the product across the set of types can still correspond to the probability of the configuration being applied each month. Other definitions of time periods other than months (e.g., days, weeks, 50 hours, etc.) are within the scope of this embodiment.

[0069] At 430 , steps 440 and 450 are performed for each iteration of the schedule of iterations of test execution for a corresponding product in the product set within a time period (eg, a month).

[0070] At 440, a corresponding infrastructure landscape type is selected from the set of infrastructure landscape types (based on a predefined probability of selection) for hosting a corresponding product in the set of products. The hosted product on the corresponding landscape type can be tested by running the corresponding set of tests during each iteration. The selected infrastructure landscape type can be bound to the corresponding product for the corresponding test iteration.

[0071] At 450, tests from the test set are executed at running instances of corresponding products. Tests are executed on a first product from the product set when the first product is run on the selected infrastructure landscape type according to selection based on a predefined probability of selection of the infrastructure landscape type.

[0072] In some embodiments, test results from tests executed on the iterative sequence may be obtained. The test results are used to determine whether to deploy the first product on the infrastructure landscape type in production mode.

[0073] According to an embodiment of the present disclosure, a selection set of landscape types can be defined for different products, where for each product there is a set in which each type is defined as having a corresponding probability of selection associated with the product. In some cases, such a selection set can be defined not only for a product, but also for a product when executing at a specific type of channel, for example, associated with a development phase, testing, production, etc. In some embodiments, a specific landscape type can be configured to be irrelevant (or in a blacklist) for selection of a given product, and therefore such a landscape type may not be included in the corresponding set for the product or may be included and set with a probability of 0%. Therefore, a bucket is a list of choices from which an option needs to be selected. There is a chance for each choice.

[0074] Figure 5 is a schematic diagram of an example computer system 500 that can be used to implement embodiments of the present disclosure. For example, computer system 500 can be included in any or all of the server components discussed herein. Computer system 500 includes a processor 510, memory 520, storage device 530, and input / output device 550. Components 510, 520, 530, and 550 are interconnected using a system bus 550. Processor 510 is capable of processing instructions for execution within computer system 500. In some embodiments, processor 510 is a single-threaded processor. In some embodiments, processor 510 is a multi-threaded processor. Processor 510 is capable of processing instructions stored in memory 520 or on storage device 530 to display graphical information for a user interface on input / output device 550.

[0075] Memory 520 stores information within computer system 500. In some embodiments, memory 520 is a computer-readable medium. In some embodiments, memory 520 is a volatile memory unit. In some embodiments, memory 520 is a non-volatile memory unit. Storage device 530 can provide mass storage for system 500. In some embodiments, storage device 530 is a computer-readable medium. In some embodiments, storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device. Input / output device 550 provides input / output operations for computer system 500. In some embodiments, input / output device 550 includes a keyboard and / or a pointing device. In some embodiments, input / output device 550 includes a display unit for displaying a graphical user interface.

[0076] The described features can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier (e.g., in a machine-readable storage device for execution by a programmable processor), and the method operations can be performed by the programmable processor executing the program of instructions to perform the functions of the described embodiments by operating on input data and generating output. The described features can advantageously be implemented in one or more computer programs executable on a programmable system, the programmable system comprising at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform an activity or produce a result. A computer program can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0077] By way of example, suitable processors for executing a program of instructions include both general-purpose and special-purpose microprocessors, as well as the sole processor or one of multiple processors of any type of computer. Typically, the processor will receive instructions and data from read-only memory or random access memory, or both. Elements of a computer may include a processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer may also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as SPROMs, EEPROMs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated into an ASIC (application-specific integrated circuit).

[0078] To provide for interaction with a user, these features may be implemented on a computer having a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer.

[0079] These features can be implemented in a computer system that includes a back-end component (such as a data server), or includes a middleware component (such as an application server or an internet server), or includes a front-end component (such as a client computer with a graphical user interface or an internet browser), or any combination thereof. The components of the system can be connected by any form or medium of digital data communication (such as a communication network). Examples of communication networks include, for example, a LAN, a WAN, and the computers and networks that form the internet.

[0080] A computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a network, such as the one described. The relationship between client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0081] Additionally, the logic flows depicted in the accompanying drawings do not require the particular order shown, or sequential order, to achieve the desired results. Additionally, other operations may be provided, or operations may be eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

[0082] A number of embodiments of the present disclosure have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the appended claims.

[0083] In view of the embodiments of the above-mentioned subject matter, the present application discloses the following list of examples, wherein one feature of an individual example or more than one feature combination of the examples and optionally combined with one or more features of one or more additional examples are additional examples that also fall within the disclosure of the present application.

[0084] Example

[0085] Although the present application is defined in the appended claims, it will be appreciated that the invention may also (alternatively) be defined in accordance with the following embodiments:

[0086] A testing framework that supports precise selection of microservices testing in cloud environments

[0087] Example 1. A computer-implemented method comprising:

[0088] Obtaining metadata for a set of changed files, wherein the set of changed files is to be deployed in a software product and stored at a source code repository;

[0089] analyzing metadata of the set of changed files and content of at least one of the changed files based on the set of rules to determine a subset of tests of the default test plan to execute; and

[0090] The subset of tests is executed at a test landscape running the set of software components associated with the set of changed files.

[0091] Example 2. The method of example 1, wherein obtaining metadata for the set of changed files is performed in response to detecting a change by a source code repository, wherein obtaining comprises:

[0092] analyzing metadata obtained from a source code repository, wherein the source code repository provides metadata in response to receiving changes to source code and / or metadata associated with a software product; and

[0093] The method comprises:

[0094] Content of at least one of the changed files is obtained to evaluate the change in the source code based on the set of rules.

[0095] Example 3. The method of any of the preceding examples, comprising:

[0096] A rule set is defined that includes rules for skipping or performing testing of portions of the software product based on identifying flags in changed portions of source code in the set of changed files.

[0097] Example 4. The method of any preceding example, wherein performing the subset of the tests comprises:

[0098] Create a test landscape as a temporary landscape; and

[0099] A collection of instances of a collection of software components associated with the collection of changed files is provided.

[0100] Example 5. The method of any preceding example, wherein performing the subset of the tests further comprises:

[0101] The test landscape is updated by updating the set of instances by performing changes associated with the set of changed files to place the set of instances in an updated state after the changes.

[0102] Example 6. The method of any preceding example, wherein after executing a subset of tests at the test landscape, the test landscape is deleted.

[0103] Example 7. The method of any preceding example, wherein performing a subset of the tests comprises:

[0104] Perform unit testing on parts of software components, and

[0105] Perform integration testing in parallel between at least two of the software components,

[0106] Unit tests and integration tests are identified as part of the subset of tests based on a set of rules.

[0107] Example 8. The method of any preceding example, wherein performing the subset of the tests comprises:

[0108] A landscape type is determined for executing the test on a component in the set of software components, wherein the set of landscape types is defined for the test, wherein each of the landscape types corresponds to a type of infrastructure for running the software component.

[0109] Example 9. The method of any preceding example, wherein the test landscape is a cloud test landscape configured to host software components as a service.

[0110] Example 10. The method of any preceding example, wherein the test landscape comprises at least two landscapes of different infrastructure types, wherein components of the set of software components are tested in parallel on the at least two landscapes.

[0111] Example 11. A system comprising:

[0112] one or more processors; and

[0113] One or more computer-readable memories coupled to the one or more processors and having instructions stored thereon, the instructions executable by the one or more processors to perform the method according to any of Examples 1 to 10.

[0114] Example 12. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of Examples 1 to 10.

[0115] Test load distribution across different landscape types in cloud infrastructure

[0116] Example 1. A computer-implemented method comprising:

[0117] Determining a set of products to be tested, wherein each product includes a set of microservices, wherein the set of microservices is deployed on a cloud infrastructure scenario;

[0118] determining a set of infrastructure landscape types for test execution of each respective product in the set of products, wherein each type is associated with a predefined probability of being selected from each set corresponding to each product; and

[0119] For each iteration in the iteration schedule of the test execution for the corresponding product in the product set during the time period:

[0120] selecting, based on a predefined selection probability, a corresponding infrastructure landscape type from a corresponding set of infrastructure landscape types for hosting each product from the set of products to run the corresponding set of tests during an iteration, wherein the selected infrastructure landscape type is bound to the corresponding product for the corresponding test iteration, and

[0121] Tests from the test set are executed, wherein the tests are executed on corresponding products from the product set when the products are run on the selected infrastructure landscape type according to a selection based on a predefined selection probability for the infrastructure landscape type.

[0122] Example 2. The method of Example 1, comprising:

[0123] A set of products to be tested is determined based on changes in source code of the received set of microservices, wherein the set of microservices is deployed at a production landscape infrastructure of a given infrastructure landscape type in the set of infrastructure landscape types.

[0124] Example 3. A method as described in any of the preceding examples, wherein the collection of products includes services associated with a change in source code of the collection of microservices, wherein the change alters performance of the collection of products when provided as a service deployed on a cloud infrastructure landscape.

[0125] Example 4. The method according to any of the preceding implementation examples further includes:

[0126] in response to executing a test on the first product while the first product is running on the selected infrastructure landscape type, obtaining test results from the execution;

[0127] evaluating the test results to determine the performance of the first product; and

[0128] In response to determining that the test has failed, performing an update of the first product based on modifications defined in accordance with the failed test, and continuing to execute the test of the first product along with the update while running the test again on the selected infrastructure landscape type until new test results show successful execution.

[0129] Example 5. The method of any of the preceding examples, wherein a collection of products is deployed on corresponding collections of different infrastructure landscape types on the cloud infrastructure landscape in a collection of corresponding production channels of the corresponding infrastructure landscape types on the cloud infrastructure landscape, and wherein the corresponding infrastructure landscape type selected for each product for testing is different from the infrastructure landscape type on the cloud infrastructure landscape on which the product is deployed in production mode.

[0130] Example 6. The method of any preceding example, wherein during each iteration of the schedule:

[0131] Create a collection of infrastructure landscape types; and

[0132] Each of the collections of infrastructure landscape types is updated to run a corresponding product from the collection of products to be tested.

[0133] Example 7. The method according to any of the preceding examples further includes:

[0134] Test results are obtained from the executed tests through a series of iterations, wherein the test results are used to determine whether to deploy the first product in a production mode on the infrastructure landscape type.

[0135] Example 8. A system comprising:

[0136] one or more processors; and

[0137] One or more computer-readable memories, coupled to the one or more processors and having instructions stored thereon, the instructions executable by the one or more processors to perform the method according to any one of Examples 1 to 7.

[0138] Example 9. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of Examples 1 to 7.

Claims

1. A computer-implemented method, the method comprising: Obtaining metadata of a set of changed files, wherein the set of changed files is to be deployed in a software product and is stored at a source code repository; analyzing metadata of the set of changed files and content of at least one of the changed files based on the set of rules to determine a subset of tests of the default test plan to be executed; and The subset of tests is executed at a test landscape running the set of software components associated with the set of changed files.

2. The method according to claim 1, wherein: Obtaining metadata of the set of changed files is performed in response to detecting a change by a source code repository, wherein obtaining comprises: analyzing metadata obtained from a source code repository, wherein the source code repository provides metadata in response to receiving changes in source code and / or metadata associated with a software product; and Wherein, the method comprises: Content of at least one of the changed files is obtained to evaluate the changes in the source code based on the set of rules.

3. The method according to claim 1, comprising: A rule set is defined including rules for skipping or performing testing of portions of the software product based on identifying flags in changed portions of source code in the set of changed files.

4. The method according to claim 1, wherein: A subset of the tests performed include: Create the test landscape as a temporary landscape; and A collection of instances of a collection of software components associated with the collection of changed files is provided.

5. The method according to claim 4, wherein: A subset of the tests performed include: The test landscape is updated by updating the set of instances by executing changes associated with the set of changed files to place the set of instances in an updated state after the changes.

6. The method according to claim 1, wherein: After executing the subset of tests at the test landscape, the test landscape is deleted.

7. The method according to claim 1, wherein: A subset of the tests performed include: Perform unit testing on parts of software components, and Perform integration testing in parallel between at least two of the software components, The unit test and the integration test are identified as part of a subset of tests based on a set of rules.

8. The method according to claim 1, wherein: A subset of the tests performed include: A landscape type is determined for executing a test on a component in a set of software components, wherein a set of landscape types is defined for the test, wherein each of the landscape types corresponds to a type of infrastructure for running a software component.

9. The method according to claim 1, wherein: The test landscape is a cloud test landscape configured to host software components as a service.

10. The method according to claim 1, wherein: The test landscape includes at least two landscapes of different infrastructure types, wherein components in the set of software components are tested in parallel on the at least two landscapes.

11. A system comprising: one or more processors; as well as one or more computer-readable memories coupled to the one or more processors and having instructions stored thereon, the instructions executable by the one or more processors to perform operations comprising: Obtaining metadata of a set of changed files, wherein the set of changed files is to be deployed in a software product and is stored at a source code repository; analyzing metadata of the set of changed files and content of at least one of the changed files based on the set of rules to determine a subset of tests of the default test plan to be executed; and The subset of tests is executed at a test landscape running the set of software components associated with the set of changed files.

12. The system according to claim 11, wherein: Obtaining metadata of the set of changed files is performed in response to detecting a change by a source code repository, wherein obtaining comprises: analyzing metadata obtained from a source code repository, wherein the source code repository provides metadata in response to receiving changes in source code and / or metadata associated with a software product; and The one or more computer-readable memories further store stored instructions, which, when executed by the one or more processors, perform operations, including: Content of at least one of the changed files is obtained to evaluate the changes in the source code based on the set of rules.

13. The system according to claim 11, wherein: The one or more computer-readable memories also store stored instructions that, when executed by the one or more processors, perform operations comprising: A rule set is defined including rules for skipping or performing testing of portions of the software product based on identifying flags in changed portions of source code in the set of changed files.

14. The system according to claim 11, wherein: A subset of the tests performed include: Create the test landscape as a temporary landscape; and A collection of instances of a collection of software components associated with the collection of changed files is provided.

15. The system of claim 14, wherein: A subset of the tests performed include: The test landscape is updated by updating the set of instances by executing changes associated with the set of changed files to place the set of instances in an updated state after the changes.

16. A non-transitory computer readable medium coupled to one or more processors and having instructions stored thereon, the instructions, when executed by the one or more processors, causing the one or more processors to perform operations comprising: Obtaining metadata of a set of changed files, wherein the set of changed files is to be deployed in a software product and is stored at a source code repository; analyzing metadata of the set of changed files and content of at least one of the changed files based on the set of rules to determine a subset of tests of the default test plan to be executed; and The subset of tests is executed at a test landscape running the set of software components associated with the set of changed files.

17. The non-transitory computer readable medium of claim 16, wherein: Obtaining metadata of the set of changed files is performed in response to detecting a change by a source code repository, wherein obtaining comprises: analyzing metadata obtained from a source code repository, wherein the source code repository provides metadata in response to receiving changes in source code and / or metadata associated with a software product; and Wherein, the non-transitory computer-readable medium further stores stored instructions, which, when executed by the one or more processors, perform operations, the operations comprising: Content of at least one of the changed files is obtained to evaluate the changes in the source code based on the set of rules.

18. The non-transitory computer-readable medium of claim 16, comprising instructions stored thereon that, when executed by the one or more processors, perform operations comprising: A rule set is defined including rules for skipping or performing testing of portions of the software product based on identifying flags in changed portions of source code in the set of changed files.

19. The non-transitory computer readable medium of claim 16, wherein: A subset of the tests performed include: Create the test landscape as a temporary landscape; and A collection of instances of a collection of software components associated with the collection of changed files is provided.

20. The non-transitory computer readable medium of claim 19, wherein: A subset of the tests performed include: The test landscape is updated by updating the set of instances by executing changes associated with the set of changed files to place the set of instances in an updated state after the changes.

Images