Intelligent DevOps insight recommendation across software applications and landscapes

Abstract

The invention relates to intelligent DevOps insight recommendation across software applications and landscapes. In an embodiment of a computer-implemented method, in order to create extracted data records, an extraction filter is instructed to extract relevant data records from log messages for two operations of a software pipeline. To create a difference record using the extracted data record, a difference filter is instructed to compare and identify message differences between the two runs, where the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been extracted. A recommendation engine is instructed to execute a machine learning model trained with the difference records. A recommendation engine is invoked to analyze the difference record to obtain a fault indicator. It is determined that a fault resulting in the fault indicator has been corrected in a later run of the software pipeline. A change is identified in a configuration or version of the software application associated with the correction. A fault-indicator-solution combination is generated.

Description

Background Art

[0001] Software Development and Operations (DevOps) is a software engineering approach that integrates software development teams (Dev) and information technology operations teams (Ops) through collaboration and shared responsibilities. Teams work together throughout the lifecycle of design, development, and production support. DevOps benefits can include, among others, more stable production environments, shorter development cycles, faster release cycles, reduced defects, and availability of metrics. In current DevOps, problems, solutions, and actions in one landscape of the delivery pipeline are not communicated to other landscapes to prevent the same problems from occurring there. Summary of the invention

[0002] This disclosure describes insights recommendations for intelligent software development and information technology operations (DevOps) across software applications and landscapes.

[0003] In an embodiment, a computer-implemented method includes: instructing an extraction filter to extract relevant data records from log messages of two runs of a software pipeline in order to create an extracted data record; instructing a difference filter to compare and identify message differences between the two runs of the software pipeline in order to create a difference record using the extracted data record, wherein the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been taken; instructing a recommendation engine to perform machine learning model training using the difference record; calling the recommendation engine using the difference record to analyze the difference record to obtain a fault indicator; determining, based on a later run of the software pipeline, that a fault that caused the fault indicator has been corrected; identifying changes in the configuration or version of the software application associated with the correction; and generating a fault-indicator-solution combination.

[0004] The described subject matter may be implemented using a computer-implemented method; a non-transitory computer-readable medium storing computer-readable instructions for performing the computer-implemented method; and a computer-implemented system comprising one or more computer memory devices interoperably coupled with one or more computers and having a tangible, non-transitory machine-readable medium storing instructions that, when executed by the one or more computers, perform the computer-implemented method / computer-readable instructions stored on the non-transitory computer-readable medium.

[0005] The subject matter described in this specification can be implemented to achieve one or more of the following advantages. First, the described method allows analysis of configuration changes and log messages of software product build, test, and deployment pipelines, as well as production and test logs and monitoring alerts that exceed standard metrics (such as error message analysis). Second, the described method allows the system to derive indicators of production failures and identify such indicators in other cases (for example, findings from one product can be used to identify problems in other products). Third, the described method can identify potential problems in pipeline operations before deployment, which allows developers to analyze potential problems before they occur based on insights from other teams on the same situation. Fourth, the described method results in fewer problems with the deployment of new versions or changed configurations of software products, and increases product availability, enhances customer satisfaction, and reduces the operational workload of development teams in DevOps environments.

[0006] The details of one or more embodiments of the subject matter of the present specification are set forth in the detailed description, claims, and drawings. Other features, aspects, and advantages of the subject matter will become apparent to those of ordinary skill in the art from the detailed description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram showing log comparison of pipeline runs between "Test" and "Prod" (or "production") according to an embodiment of the present disclosure.

[0008] Figure 2 is a diagram illustrating identification of “fault-indicator-solution” according to an embodiment of the present disclosure.

[0009] Figure 3 is a diagram showing application of a “fault indicator” according to an embodiment of the present disclosure.

[0010] Figure 4A is a table showing three fault indicator sets according to an embodiment of the present disclosure.

[0011] Figure 4B is a table showing parameter sets for machine learning (ML) training according to an embodiment of the present disclosure.

[0012] Figure 5 is a block diagram illustrating components of the described method according to an embodiment of the present disclosure.

[0013] Figure 6 is a flow chart illustrating an example of a computer-implemented method for intelligent software development and information technology operations (DevOps) insight recommendations across software applications and landscapes according to an embodiment of the present disclosure.

[0014] Figure 7 is a block diagram illustrating an example of a computer-implemented system for providing computing functions associated with the described algorithms, methods, functions, processes, procedures, and programs according to an embodiment of the present disclosure.

[0015] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0016] The following detailed description describes insights recommendations for intelligent software development and information technology operations (DevOps) across software applications and landscapes, and is presented to enable any technician in the field to make and use the disclosed subject matter in the context of one or more specific implementations. Various modifications, transformations, and substitutions may be made to the disclosed embodiments without departing from the scope of the present disclosure, and these modifications, transformations, and substitutions will be obvious to those of ordinary skill in the art, and the general principles defined may be applied to other embodiments and applications. In some cases, one or more technical details that are unnecessary for obtaining an understanding of the described subject matter and are within the skill of those of ordinary skill in the art may be omitted so as not to obscure one or more described embodiments. The present disclosure is not intended to be limited to the embodiments described or shown, but is in accordance with the widest scope consistent with the described principles and features.

[0017] Software Development and Operations (DevOps) is a software engineering approach that integrates software development teams (Dev) and information technology operations teams (Ops) through collaboration and shared responsibility. Teams work together throughout the entire lifecycle of design, development, and production support. Among other things, the benefits of DevOps can include a more stable production environment, shorter development cycles, faster release cycles, reduced defects, and availability of metrics.

[0018] Describes the development of software applications, typically cloud-based applications built with a microservices architecture and modern continuous integration and continuous delivery (CI / CD) pipelines. Typically, such software development teams run a DevOps approach—that is, the software development team not only develops the software application, but also runs the software application in a production environment and is responsible for information technology operations. Such a CI / CD pipeline typically includes a staging (or test) software landscape and one or more production software landscapes.

[0019] Typically, CI / CD delivery pipelines are designed to deliver small changes frequently (i.e., the difference between two software deployments is fairly small in terms of version changes, configuration changes, and / or code changes). Each change is tested in testing and propagated to production if no errors are found. Teams are not only required to iterate quickly (to enable rapid reaction to any issues found in production), but also to optimize the availability of services for other things like service level agreements (SLAs). For example, a software application provided as a service should not be offline for more than a contractually agreed percentage of time (usually, availability is in the range of 99.7% or higher). Therefore, it is critical to avoid violating SLA contracts by interrupting efficiently running applications with new deployments.

[0020] Software development teams are therefore faced with the dilemma of “delivering fast” but still “not breaking anything”. Code and configuration changes are tested before delivery to production. But such testing can only cover a certain set of scenarios as there are always gaps. Often, services used in production face higher loads, data volumes, more connected users, and more heterogeneous landscapes than the test landscape can cover due to logistics, time constraints, costs, etc. Therefore, any additional insights into potential issues / failures and identifying potential problems before they can occur are highly relevant. The described approach elaborates and exploits insights that can be derived from one software landscape and software application, and can be transcribed to other software landscapes and software applications using similar building blocks, libraries, etc. In other words, DevOps efficiency can be improved by identifying problems in one software landscape of a CI / CD pipeline and recommending actions for another software landscape to prevent the same / similar problems from happening again. In addition, the described approach can identify problems with one software application and recommend actions for another software application with the same intent.

[0021] In traditional DevOps approaches, problems and associated solutions found in one software application can only be transferred to other software application development efforts if different software development teams know their commonalities and communicate issues between each other. In other words, one software development team may find a solution to a problem that arises in their production software landscape, but the software development team may not know that another software development team has the same configuration and use of software development building blocks; therefore, one or more insights are lost to the other software development team. Alternatively, the software development team may know about another software development team remotely, but be too busy solving their own problems to communicate with the other software development team in a timely manner, which means that precious time between identifying the problem and implementing the solution may be lost.

[0022] Often, problems are not detected during testing, but occur in the production software landscape due to different usage characteristics. This is because usage between test and production can vary—typically with higher loads, larger data volumes, and wider usage scopes found in the production software landscape when compared to the test software landscape.

[0023] For many components, warning messages are usually issued if the situation approaches a critical limit, and error messages are issued when a critical limit is exceeded (e.g., for buffer usage, cache flush rate, or size of a queue, several critical thresholds are defined, at which warning or error messages are written before the software system terminates at a certain point with a fatal message). Looking at different software landscapes, there may be situations where a load created in a test software landscape, which only causes warning messages, turns into a fatal error when faced with a much larger load found in a production software landscape. Therefore, even warning messages in tests may need to be considered as critical situations, if one knows the correlation between a certain warning message and what it will turn into under increased load in a production software landscape.

[0024] Testing software landscapes is resource intensive / expensive and cannot realistically cover all possible software application integration variations and usage options. Specific software applications are often used for testing because the total resource expenditure for testing increases the total cost of the service and can reduce or eliminate the competitiveness of the service. Therefore, decisions are always made about what balance should be achieved between what to test and what not to test.

[0025] The untestable situation is closely monitored to enable rapid reaction in case the situation falls into an error condition. Early indicators of impending problems are embedded in the production software landscape (e.g., escalating warning and error messages are logged, or more fine-grained threshold monitoring is established). However, this situation is only monitored in the production software landscape and may eventually lead to a failure that could have been prevented if detected earlier or targeted at other similar software applications.

[0026] The described method creates a system that can detect failures in a production software landscape, associate the detected failures with one or more indicators in a build or test, and use the occurrence of one or more indicators found in other landscapes to alert software developers that the one or more indicators may cause failures in other landscapes.

[0027] The described process can be extended to also identify solution indicators for removing faults in the production software landscape. When a fault indicator is found in the landscape, the software developer can then not only be alerted that a production software landscape fault may occur, but can also be informed of potential solutions to mitigate / eliminate the potential production software landscape fault.

[0028] The described method relies on existing fault reporting systems based on logging and external monitoring. To identify potential fault indicators and solution indicators, the described method prepares information in two steps: First, extract data records that are likely to contain relevant information (using "extraction filters"). In some implementations, the data records are in the form of log messages or configuration parameters that contain information about deployed versions and components, error messages, and status information of executed tasks. Second, compare the differences of one pipeline run to an earlier pipeline run (using "difference filters"). In some embodiments, the difference filter compares two runs of the pipeline and identifies different messages. In this way, the described method can derive, for example, a "failure-indicator-failure-solution-indicator" data set.

[0029] While a single data set may be used to scan pipeline runs (particularly pipeline runs of other software applications), multiple data sets may be used to prepare machine learning (ML) models. In some implementations, a superset of identified failure indicator and solution indicator parameters may be read from a set of pipeline runs of different products. The ML model may be trained with the parameters and failures as labeled data to provide a model that may be queried with a set of parameters to assess whether the set of parameters may cause future failures. In some implementations, the parameters may include changes to build configurations, changes to software versions used (both directly and indirectly consumed), log messages from builds and runs, and / or values ​​extracted from such log messages.

[0030] Using the described method, a certain set of problematic changes and critical situations in build and test can be identified before the pipeline updates the production software landscape and can potentially cause failures. Software developers can be proactively alerted to check their software landscape and adjust - potentially even using the solution recommendations provided by the method. The described method allows insights from one software application to be transferred to other software applications.

[0031] Identifying faults

[0032] Identification of system failures can be based on existing monitoring infrastructure; however, extensions can add broader and more detailed analysis to better detect early warning indicators. Failures can be identified as alerts in log messages, crash dump records, service downtime, other service degradations, and / or other alert events in the production software landscape. For failures, it is necessary to identify the software application versions used and pipeline runs that led to the configuration deployment ("version vector of building blocks used").

[0033] Identifying Fault Indicators

[0034] The described method then searches for potential failure indicators. The focus is on failure indicators in the "version vector of used building blocks" - the exact pipeline run that caused the software application version to fail. In the pipeline run, the focus is on the changed configuration and software application version, certain log messages and parameters in the log messages, and (logged) failure detection in the test software landscape, similar to the failure detection in the production software landscape.

[0035] The data record is extracted (using the "Extract Filter") and is likely to contain relevant information. This is a very small subset of the total available information for a Pipeline run. On this subset, the focus is on what has changed compared to the previous run (e.g. a changed configuration parameter, a new repository version of a parameter, or the severity of a log message) using the "Difference Filter".

[0036] Configuration / Version

[0037] The described method is provided with access to the software application build, software application configuration, and the used software application version in the pipeline configuration. The described method can then identify changes in the data set between two pipeline runs. Comparing the pipeline run that caused the production software application deployment to fail with the previous pipeline run can be used to determine the changes that may be the cause of the failure.

[0038] The identified records are considered “failure indicators” (in “configuration / version”). Since in current DevOps approaches with automated CI / CD pipelines, the changes between two deployments are small, several changed records (configuration / version) will be detected, providing a strong signal.

[0039] Go to Figure 1 , Figure 1 is a diagram 100 showing a log comparison of pipeline runs between "test" and "production" according to an embodiment of the present disclosure.

[0040] Comparing log messages from pipeline runs

[0041] The described method is provided with information about logged or otherwise identified (eg, by monitoring) faults in the production software landscape and related logged / monitored messages from a test run of the software application during a test step of a pipeline run. Figure 1 "Build 5.3" and "Build 5.4" are shown at 102 and 104, respectively. When a "failure" is identified in production (106) as a result of a pipeline run, the filtered log of the pipeline run is analyzed for differences between that pipeline run (i.e., build 5.4, 104) and the previous pipeline run (i.e., build 5.3, 102):

[0042] ■ At 108, the log of the test steps of the pipeline run is compared with the log of the test steps of the previous pipeline run.

[0043] ■Similarly, builds and deployments to test logs can be compared with earlier runs of the pipeline.

[0044] The described method scans the filtered log for records that changed in a failed run (ie, build 5.4, 104) compared to records in a previous successful run (ie, build 5.3, 102). Such identified records are considered "failure indicators".

[0045] Compare test and production ("prod") log messages.

[0046] Additionally, the described method can compare log messages (e.g., deployment and runtime) from two software landscapes (i.e., test and production) of the same application (e.g., Build 5.4 104). Log messages with the same log message id are compared between the two landscapes (test and production) within the log files of a product with the same version ("version vector of used building blocks") in test and production.

[0047] As an example of two scenarios being searched: high severity in a production software landscape, lower severity in a test software landscape:

[0048] ■ The message is evaluated as an "error" in the production software landscape and is detected as evaluated as a "warning" in the test software landscape.

[0049] ■ The message is evaluated as "Fatal" in the production software landscape and is detected as evaluated as "Warning" or "Error" in the test software landscape.

[0050] As part of the described method, some failure indicators may be identified where the test severity may be increased (110). For example, in the previous example, a message detected as being rated as a "warning" or "error" in the test software landscape may be "increased" to also being rated as "fatal" in the test software landscape.

[0051] Additionally, in some implementations, certain log messages can be scanned that contain statements about limited resources (e.g., CPU or memory - search for keywords in the logs). As an example, key parameters written to the logs (e.g., "% of memory consumed") can also be compared between landscapes:

[0052] ■Such a message may be reported as "info" for example if usage is below 80% in a test.

[0053] ■ As opposed to a "warning" for a "90% memory consumption" or even a low memory situation in production.

[0054] Other differences in log statements related to application health between the compared landscapes may be more complex, but still detectable applying modern natural language analysis algorithms.

[0055] In the described approach, the comparison of test and production log data can allow generating insights for developers in at least two ways:

[0056] 1. Use data analytics so that developers can review the data and evaluate whether changes to, for example, log message severity, threshold levels, or test coverage should be applied:

[0057] ■ Generate input for developers with selective log messages that differ between Test and Prod and have, for example, a severity such as "Error in Production" and "Warning in Test".

[0058] ■Extract relevant parameters from log messages.

[0059] ■ Calculate the distribution of these parameters over the runs of the pipeline (not only the erroneous runs / runs that caused production failures).

[0060] 2. Train the ML model using the selected log messages and extracted parameters (112):

[0061] ■ In some implementations, the ML model can be queried to evaluate test logs for indicators that production may fail if a particular configuration is deployed to production (e.g., after analyzing the test logs of build 5.4 104 before deploying to production).

[0062] Fault-Fault-Indicator-Set

[0063] With the previously described method, two sets of failure occurrences are obtained with identification indicators (“Failure-Indicator-Set”) for the application (Ax):

[0064] ■Fault-Indicator-Set-CV(Ax):

[0065] ■ Errors or fatal messages in the landscape and configuration / version changes in the application build.

[0066] ■Fault-Indicator-Set-Log (Ax):

[0067] ■ Error or fatal messages in production landscape and warning or error messages in test landscape.

[0068] These two collections compare test and production deployments of your app with the same version vectors of the building blocks used.

[0069] Identify indicators in other apps

[0070] The fault-indicator combination can be used to monitor other applications of the same indicator in the AI ​​landscape and CI / CD pipelines.

[0071] ■If the same indicator is found in the changes or logs (indicator_i) of the test landscape of another application Ai (i.e. the same change to the configuration / version or the same log record), it is considered to be identified that the indicator that caused the failure at application Ax has been found at Ai, and this indicator can cause the same failure at Ai.

[0072] ■ The software development team at Ai can now be informed that the performed change (indicator_i) resulted in a failure in another application Ax. In this way, investigative and / or mitigation actions can be taken to try to avoid the possible failure.

[0073] Identify solutions

[0074] Go to Figure 2 , Figure 2 is a diagram 200 illustrating identification of a "fault-indicator-solution" according to an embodiment of the present disclosure.

[0075] Continuing with the previous discussion, when the developer has corrected the failure and the original error message is no longer in the log, each of the two failure indicator sets can be compared to a later run of application Ax (a later version vector of the building block used).

[0076] ■ A deployment of a CI / CD run may be identified after which the error message (“failure”) no longer occurs. For example, at 202, a failure is detected at a run of a pipeline with version vector 1.3 (ie, build 1.3,203).

[0077] ■ Corresponding changes during the build (configuration / version) of the same CI / CD may be identified (if the failure has been resolved by changing the configuration or version). For example, at 204, an "indicator" is identified as a difference in the build version or configuration run earlier compared to version vector 1.3. Note that version vector 1.4 (i.e., build 1.4, 205) also indicates an "application error" as in version vector 1.3, build 1.3, 203.

[0078] ■ If such a change in (configuration / version) is detected, this is a strong indicator that the original failure was resolved: the change is a “resolution” indicator. For example, at 206 and version vector 1.5 (ie, build 1.5, 207), the failure no longer occurs.

[0079] ■ Thus, one sets a failure indicator at Ax, and a solution is given at Ax at a later point in time: a "failure-indicator-solution" combination. For example, at 208, the difference in version / configuration compared to version vector 1.3 (ie, build 1.3,203) is identified as a solution.

[0080] Identify indicators in other applications and recommend solutions

[0081] Go to Figure 3 , Figure 3 is a diagram 300 showing application of a “Fault Indicator” according to an embodiment of the present disclosure.

[0082] The failure-indicator-solution combination can be used to monitor the software landscape and CI / CD pipelines of other applications Ai for the same failure indicators. Figure 2 ), not only can the fault indicator be reported (as described previously), but a solution can also be recommended.

[0083] ■ If the same indicator (Indicator_x) is found in a change or test landscape of another application Ai (Indicator_i) (same change to (configuration / version) or same log record), then the indicator (Indicator_x) that caused the failure at application Ax has been found at Ai and may cause the same failure at Ai. For example, at 302, with respect to build 5.6, 303 identifies the indicator from the previous determination.

[0084] ■ The development team of Ai can now be informed that the change (Indicator_i) caused a failure in another application Ax. For example, at 304, the pipeline for build 5.6, 303 is not run, but build 5.4, 306 continues to run.

[0085] ■ In addition, the development team may be notified of the fault-indicator-solution combination with a description of the change to its application (Indicator_i) and a description of the change and log of the fault-indicator-solution combination for Ax. For example, at 308, a solution is identified based on the indicator and proposed to the developer.

[0086] ■The development team can then analyze to determine if the change that caused the other application to fail is a risk, and ultimately utilize the solution found by Ax's development team to correct / mitigate possible future failures. For example, in build 5.7 (310), the development team can incorporate the change (solution) to correct / mitigate the identified indicator at 302 and run build 5.7 (310).

[0087] These techniques can be used in small settings with few pipelines and products. Another possible use case is to react quickly to newly discovered failure-incident combinations, where time will be lost until the new events are incorporated into the new ML model through training.

[0088] ML model training

[0089] To train the ML model, data records are collected based on the analysis runs to find failure indicators and failure-indicator-solution sets. If a data record is found in the context of any failure indicator or failure-indicator-solution set, the data record is obtained to filter the input. The idea is to collect a superset of data records for all pipeline runs that appear as "failures", "indicators", and "solutions" in each analysis.

[0090] Go to Figure 4A , Figure 4A 4 is a table 400a showing three fault indicator sets according to an embodiment of the present disclosure. Figure 4A middle:

[0091] ■Fault Indicator Solution Set 1 (402a):

[0092] - Failure: Message with ID 112 "Insufficient Memory" and severity "Critical" at production (408a).

[0093] - Indicator: Changed version of library (410a) 'magic-parkman-player' used.

[0094] -Solution: Added the setting of parameter "heap-space" in the configuration file "container-pacman-player.json".

[0095] ■Fault Indicator Solution Set 2 (404a):

[0096] - Failure: Message with ID 112 "Insufficient Memory" and severity "Critical" at production (412a).

[0097] - Indicator: Message with ID 123 "Memory Usage" and severity "Warning" at test (414a).

[0098] -Solution: Added the setting of parameter "heap-space" in the configuration file "container-pacman-player.json".

[0099] ■Fault Indicator Solution Set 3 (406a):

[0100] - Failure: Message with ID 99 "Message Queue Overflow" and severity "CRITICAL" at production (416a).

[0101] - Indicator: Message with ID 99 "Message queue overflow" and severity "Warning" at test (418a).

[0102] -Solution: Added the setting of parameter "queue-size" in the configuration file "player-queue.json".

[0103] Therefore, the superset of data records obtained from each pipeline run will be:

[0104] ■Production:

[0105] -Message with ID 112 "Insufficient Memory" and severity "Critical" at production.

[0106] - A message with ID 99 "Message Queue Overflow" and severity "Critical" at production.

[0107] ■Test:

[0108] -Message with ID 123 "Memory Usage" and severity "Warning" at test.

[0109] -Message with ID 99 "Message queue overflow" and severity "Warning" at test.

[0110] ■Configuration / Version:

[0111] - Changed the version of the used library "Magic-Pac-Man-Player".

[0112] - Changed the setting of parameter "heap-space" in configuration file "container-pacman-player.json".

[0113] - Changed the setting of parameter "queue-size" in the configuration file "player-queue.json".

[0114] Go to Figure 4B , Figure 4B 4 is a table 400b showing a parameter set for machine learning (ML) training according to an embodiment of the present disclosure.

[0115] For each pipeline run (e.g., at least Figure 4A ), collect more records than used in a single "failure indicator" set, but every parameter that appears in the "failure indicator" set is then included in the dataset. This results in a dataset of n parameters and a label "ok" or "failed" for each pipeline run. Figure 4B In the text format, 'W', 'I', and 'E' stand for "Warning", "Information", and "Error", respectively.

[0116] ■ If any of the data records indicative of a failure are part of a pipeline run data set, the set is marked as "failed". For example, the parameter sets used for ML training 402b, 404b and 406b correspond to Figure 4A Fault indicator sets 402a, 404a and 406a in.

[0117] ■ And if these parameters are not found (eg, the parameter set used for ML training 408b), then "ok".

[0118] The dataset of parameters and labels is passed to the ML training.

[0119] If the ML model is created with an appropriate level of accuracy, it can be used during a pipeline run:

[0120] ■From the build step, the parameters that change can be extracted.

[0121] ■ From the test run steps, extract the identified log messages.

[0122] ■Parameters are passed to the ML model to derive whether production was predicted (with a certain accuracy and error) to fail as it runs.

[0123] ■ If the ML model returns an increased probability that the production run will fail, the deployment to production step is paused and the developer team is notified of the reported probability.

[0124] Regarding training data, once parameters or log messages have been identified as relevant, they will be included in the dataset fed to ML. Figure 4B , the "x" present in the three columns to the right of the pipeline run column of the row designated by 406b indicates that there was a case where error message ID 99 was associated (i.e., "Configuration Library Version", "Configuration Heap Space", and "Configuration Queue Size"), and therefore the presence / value is included in the dataset given to ML. In most build instances, the message will be empty (i.e., not present) or the value will be a good value, and if this is paired with "No Problem", ML can learn the relationship. When the message appears in a dataset (or a configuration parameter is changed) and this is paired with a failure (or a solution after a failure), then the correct data is included in the learning, and ML will then detect the relationship.

[0125] Go to Figure 5 , Figure 5 is a block diagram 500 illustrating components of the described method according to an embodiment of the present disclosure.

[0126] Developer 502 uses the described methods to find insights and control the pipeline.

[0127] Extraction filters 504 may be configured to extract certain parameters and log messages from various records 506 - at a high level - product configurations, build versions, log messages, and monitoring data (e.g., alerts) of test runs and production systems. In particular, and in some embodiments, 506 may include Configuration / Version, Test Logs, Test Monitoring, Prod Logs, and Prod Monitoring.

[0128] The difference filter 508 may calculate the difference of the extracted records compared to an earlier run of the pipeline.

[0129] The recommender 510 (e.g., a recommendation software engine / application) can run a model training 512 process using data from the difference filter 508. The recommender 510 can re-run the model training 512 process in a loop (e.g., after a specified number of pipeline runs). The recommender 510 can query the ML model 514 with a set of parameters. The developer 502 can query the recommender 510 for insights about the ML model 514 or the data passed through the extraction filter 504 and the difference filter 508.

[0130] The pipeline runtime 516 can use the extraction filter 504 and the difference filter 508 to read a set of parameters and log messages from the product configuration, build version and / or log messages of the test run (e.g., 506). The pipeline runtime 516 can calculate the difference between the extracted records when compared with an earlier run of the pipeline (e.g., using the pipeline run history 518). The pipeline runtime 516 can call the recommender 510 with a set of parameters obtained using the extraction filter 501 / difference filter 508. The recommender 510 can stop the pipeline run and notify the developer (e.g., on the development team) about the expected problems with the current pipeline run.

[0131] To enhance understanding, an example pipeline using a new library version that requires more memory during runtime may include:

[0132] The new library version requires more memory and needs to be configured in the runtime settings. If the developer is not aware of this, the library version change may lead to error messages in the production system (for example, "out of memory" errors (oom messages) may occur, in the log or detected by runtime monitoring).

[0133] If this failure is observed using the described method, the method will check the changes in the build and test logs and will identify the library version change. If other teams also change the library version, the system can identify the change and point to the potential problem of the out of memory condition.

[0134] In some implementations, this example can be expanded. For example, there may be a memory warning in test that already indicates a memory problem, but is easily overseen because it is only logged as a warning and does not cause an error (because "fatal" errors only occur under higher loads in production). Memory errors in production and warnings in test can be identified, and developers can be notified of the correlation. The developer can then decide to modify the memory threshold or severity of the messages in test for earlier alerts.

[0135] As a further example of extension, in a subsequent run of the pipeline (after the library version has changed), the developer extends the memory configuration of the runtime settings. This pipeline run deployment no longer runs the wrong production version. The system identifies that the change in configuration solved the problem in production. The configuration change is identified as a "solution", so other teams can be notified not only about the problem of the increased library version, but also that the memory configuration change solved the problem for another team.

[0136] Figure 6is a flow chart illustrating an example of a computer-implemented method 600 for intelligent DevOps recommendations for insights across software applications and landscapes according to an embodiment of the present disclosure. For clarity of presentation, the following description generally describes the method 600 in the context of other figures in the present description. However, it should be understood that the method 600 can be performed, for example, by any system, environment, software and hardware or a combination of systems, environments, software and hardware, as appropriate. In some embodiments, the various steps of the method 600 can be run in parallel, combined, looped, or in any order.

[0137] At 602, to create an extracted data record, an extraction filter is instructed to extract relevant data records from log messages of two runs of the software pipeline. In some implementations, the relevant data records include deployed software application versions and components, error messages, and state information of the extracted tasks. In some embodiments, the software pipeline includes deployment, testing, and production. Method 600 proceeds from 602 to 604.

[0138] At 604, a difference filter is instructed to compare and identify message differences between two runs of the software pipeline in order to create a difference record using the extracted data record, wherein the difference record is modified using the tagged data state information of the pipeline run from which the record has been taken. From 604, method 600 proceeds to 606.

[0139] At 606, the recommendation engine is instructed to perform machine learning model training using the difference record. In some implementations, the recommendation engine can be instructed to re-perform machine learning model training after a certain number of software pipeline runs. Method 600 proceeds from 606 to 608.

[0140] At 608 , using the difference record, a recommendation engine is invoked to analyze the difference record to obtain a fault-indicator. From 608 , method 600 proceeds to 610 .

[0141] At 610 , based on a later run of the software pipeline, it is determined that the fault that caused the fault-indicator has been corrected. From 610 , method 600 proceeds to 612 .

[0142] At 612 , a change in the configuration or version of the software application associated with the correction is identified. From 612 , method 600 proceeds to 614 .

[0143] At 614, a fault-indicator-solution combination is generated. In some embodiments, upon detecting the same fault-indicator in different software pipeline executions, a notification of the same fault-indicator is generated, and the fault-indicator-solution combination is used to recommend a recommended solution for the same fault-indicator. In some embodiments, using the fault-indicator, the landscape and software pipelines of other software applications can be monitored for the same fault-indicator. Upon detecting the same fault indicator in the landscape and software pipelines of other software applications, a notification of the same fault indicator is generated, and the fault-indicator-solution combination is used to recommend a recommended solution for the same fault indicator. After 614, the method 600 can stop.

[0144] Figure 7 700 is a block diagram illustrating an example of a computer-implemented system 700 for providing computing functions associated with the described algorithms, methods, functions, processes, flows, and procedures according to an embodiment of the present disclosure. In the illustrated embodiment, the computer-implemented system 700 includes a computer 702 and a network 730.

[0145] The illustrated computer 702 is intended to encompass any computing device, such as a server, a desktop computer, a laptop / notebook computer, a wireless data port, a smart phone, a personal data assistant (PDA), a tablet computer, one or more processors within these devices, or a combination of computing devices, including physical or virtual instances of computing devices, or a combination of physical or virtual instances of computing devices. In addition, the computer 702 may include an input device, such as a keypad, a keyboard, or a touch screen, or a combination of input devices that can accept user information, and an output device that transmits information associated with the operation of the computer 702 on a graphical user interface (UI) (or GUI) or other UI, including digital data, visuals, audio, another type of information, or a combination of information types.

[0146] The computer 702 can act as, for example, a client, a network component, a server or a database or another persistent role in a distributed computing system, or a combination of roles for performing the subject matter described in the present disclosure. The illustrated computer 702 is communicatively coupled to a network 730. In some embodiments, one or more components of the computer 702 can be configured to operate within an environment or combination of environments, including cloud computing, local, or global.

[0147] At a high level, the computer 702 is an electronic computing device operable to receive, transmit, process, store or manage data and information associated with the described subject matter. According to some embodiments, the computer 702 may also include or be communicatively coupled to a server, such as an application server, an email server, a web server, a cache server or a streaming data server or a combination of servers.

[0148] Computer 702 may receive the request over network 730 (e.g., from a client software application executing on another computer 702) and respond to the received request by processing the received request using the software application or a combination of software applications. In addition, the request may be sent to computer 702 from an internal user (e.g., from a command console or through another internal access method), an external or third party, or other entity, person, system, or computer.

[0149] Each component of the computer 702 can communicate using the system bus 703. In some embodiments, any or all components of the computer 702 (including hardware, software, or a combination of hardware and software) can be interfaced through the system bus 703 using an application programming interface (API) 712, a service layer 713, or a combination of an API 712 and a service layer 713. The API 712 may include specifications for routines, data structures, and object classes. The API 712 may be independent of or dependent on a computer language, and refers to a complete interface, a single function, or even a set of APIs. The service layer 713 provides software services to the computer 702 or other components (whether or not shown) that are communicatively coupled to the computer 702. The functions of the computer 702 can be accessed by all service consumers using the service layer 713. Software services (such as those provided by the service layer 713) provide reusable defined functions through defined interfaces. For example, the interface may be software written in a computing language (e.g., JAVA or C++) or a combination of computing languages, and provides data in a specific format (e.g., extensible markup language (XML)) or a combination of formats. Although shown as an integrated component of the computer 702, alternative implementations may show the API 712 or service layer 713 as a separate component relative to or communicatively coupled to other components of the computer 702 (whether shown or not). Furthermore, any or all portions of the API 712 or service layer 713 may be implemented as a sub-module or sub-module of another software module, enterprise application, or hardware module without departing from the scope of the present disclosure.

[0150] The computer 702 includes an interface 704. Although shown as a single interface 704, two or more interfaces 704 may be used according to the specific needs, expectations, or specific implementations of the computer 702. The interface 704 is used by the computer 702 to communicate with another computing system (whether or not shown) that is communicatively linked to the network 730 in a distributed environment. Typically, the interface 704 is operable to communicate with the network 730 and includes logic encoded in software, hardware, or a combination of software and hardware. More specifically, the interface 704 may include software that supports one or more communication protocols associated with the communication, so that the network 730 or the hardware of the interface 704 is operable to transmit physical signals inside and outside the computer 702 shown.

[0151] The computer 702 includes a processor 705. Although shown as a single processor 705, two or more processors 705 may be used depending on the particular needs, desires, or particular implementations of the computer 702. In general, the processor 705 executes instructions and manipulates data to perform the operations of the computer 702 and any algorithms, methods, functions, procedures, flows, and processes as described in this disclosure.

[0152] The computer 702 also includes a database 706, which can store data for the computer 702, another component (whether or not shown) that is communicatively linked to the network 730, or a combination of the computer 702 and another component. For example, the database 706 can be an in-memory or conventional database that stores data consistent with the present disclosure. In some embodiments, the database 706 can be a combination of two or more different database types (e.g., a hybrid in-memory and conventional database) according to the specific needs, expectations, or specific implementation of the computer 702 and the functions described. Although shown as a single database 706, two or more databases of similar or different types can be used according to the specific needs, expectations, or specific implementation of the computer 702 and the functions described. Although the database 706 is shown as an integrated component of the computer 702, in alternative embodiments, the database 706 can be external to the computer 702. The database 706 can maintain and operate on at least any data type mentioned or any data type consistent with the present disclosure.

[0153] The computer 702 also includes a memory 707, which can hold data for the computer 702, another component or components (whether or not shown) that are communicatively linked to the network 730, or a combination of the computer 702 and another component. The memory 707 can store any data consistent with the present disclosure. In some embodiments, the memory 707 can be a combination of two or more different types of memory (e.g., a combination of semiconductor and magnetic storage devices) according to the specific needs, expectations, or specific implementations of the computer 702 and the functions described. Although shown as a single memory 707, two or more memories 707 or similar or different types can be used according to the specific needs, expectations, or specific implementations of the computer 702 and the functions described. Although the memory 707 is shown as an integrated component of the computer 702, in alternative embodiments, the memory 707 can be external to the computer 702.

[0154] The application 708 is an algorithmic software engine that provides functionality according to the specific needs, expectations, or specific implementation of the computer 702, particularly with respect to the functionality described in the present disclosure. For example, the application 708 can be used as one or more components, modules, or applications. In addition, although shown as a single application 708, the application 708 can be implemented as multiple applications 708 on the computer 702. In addition, although shown as integrated with the computer 702, in alternative embodiments, the application 708 can be external to the computer 702.

[0155] The computer 702 may also include a power supply 714. The power supply 714 may include a rechargeable or non-rechargeable battery, which may be configured to be user-replaceable or non-user-replaceable. In some embodiments, the power supply 714 may include power conversion or management circuitry (including recharging, standby, or another power management function). In some embodiments, the power supply 714 may include a power plug to allow the computer 702 to be plugged into a wall outlet or another power source, for example, to power the computer 702 or to recharge a rechargeable battery.

[0156] There may be any number of computers 702 associated with or external to the computer system containing computer 702, each of which communicates via network 730. In addition, the terms "client," "user," or other appropriate terms may be used interchangeably as appropriate without departing from the scope of the present disclosure. In addition, the present disclosure contemplates that many users may use one computer 702, or that one user may use multiple computers 702.

[0157] Implementations of the described subject matter may include one or more features alone or in combination.

[0158] For example, in a first embodiment, a computer-implemented method includes: instructing an extraction filter to extract relevant data records from log messages of two runs of a software pipeline to create an extracted data record; instructing a difference filter to compare and identify message differences between the two runs of the software pipeline to create a difference record using the extracted data record, wherein the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been taken; instructing a recommendation engine to perform machine learning model training using the difference record; calling the recommendation engine using the difference record to analyze the difference record to obtain a fault indicator; determining, based on a later run of the software pipeline, that the fault that caused the fault indicator has been corrected; identifying a change in the configuration or version of the software application associated with the correction; and generating a fault-indicator-solution combination. The foregoing and other described embodiments may each optionally include one or more of the following features:

[0159] The first feature, which may be combined with any of the following features, wherein the relevant data records include deployed software application versions and components, error messages, and status information of extracted tasks.

[0160] The second feature, which can be combined with any of the previous or following features, is that the software pipeline includes deployment, testing and production.

[0161] A third feature, which may be combined with any previous or following feature, comprises: generating a notification of the same fault indicator when the same fault indicator in different software pipeline executions is detected; and recommending a recommended solution for the same fault indicator using the fault indicator-solution combination.

[0162] A fourth feature, which may be combined with any previous or following feature, comprises: using the fault indicator to monitor the landscape and software pipeline of other software applications to obtain the same fault indicator.

[0163] A fifth feature, which may be combined with any previous or following feature, comprises: generating a notification of the same fault indicator when the same fault indicator is detected in the landscape and software pipeline of other software applications; and recommending a recommended solution for the same fault indicator using the fault indicator-solution combination.

[0164] A sixth feature, which may be combined with any of the previous or following features, comprises: instructing the recommendation engine to re-execute the machine learning model training after a specified number of software pipeline runs.

[0165] In a second embodiment, a non-transitory computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations includes: instructing an extraction filter to extract relevant data records from log messages of two runs of a software pipeline in order to create an extracted data record; instructing a difference filter to compare and identify message differences between the two runs of the software pipeline in order to create a difference record using the extracted data record, wherein the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been taken; instructing a recommendation engine to perform machine learning model training using the difference record; calling the recommendation engine using the difference record to analyze the difference record to obtain a fault indicator; determining, based on a later run of the software pipeline, that the fault that caused the fault indicator has been corrected; identifying changes in the configuration or version of the software application associated with the correction; and generating a fault-indicator-solution combination.

[0166] The foregoing and other described embodiments may each optionally include one or more of the following features:

[0167] The first feature, which may be combined with any of the following features, wherein the relevant data records include deployed software application versions and components, error messages, and status information of extracted tasks.

[0168] The second feature, which can be combined with any of the previous or following features, is that the software pipeline includes deployment, testing and production.

[0169] A third feature, which may be combined with any previous or following feature, comprises: generating a notification of the same fault indicator when the same fault indicator in different software pipeline executions is detected; and recommending a recommended solution for the same fault indicator using the fault indicator-solution combination.

[0170] A fourth feature, which may be combined with any previous or following feature, comprises: using the fault indicator to monitor the landscape and software pipeline of other software applications to obtain the same fault indicator.

[0171] A fifth feature, which may be combined with any previous or following feature, comprises: generating a notification of the same fault indicator when the same fault indicator is detected in the landscape and software pipeline of other software applications; and recommending a recommended solution for the same fault indicator using a fault-indicator-solution combination.

[0172] A sixth feature, which may be combined with any of the previous or following features, comprises: instructing the recommendation engine to re-execute the machine learning model training after a specified number of software pipeline runs.

[0173] In a third embodiment, a computer-implemented system comprises: one or more computers; and one or more computer memory devices, the one or more computer memory devices being interoperably coupled to the one or more computers and having a tangible, non-transitory machine-readable medium storing one or more instructions, the one or more instructions performing one or more operations when executed by the one or more computers, the one or more operations comprising: instructing an extraction filter to extract relevant data records from log messages of two runs of a software pipeline in order to create an extracted data record; instructing a difference filter to compare and identify message differences between the two runs of the software pipeline in order to create a difference record using the extracted data record, wherein the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been taken; instructing a recommendation engine to perform machine learning model training using the difference record; calling the recommendation engine using the difference record to analyze the difference record to obtain a fault indicator; determining, based on a later run of the software pipeline, that the fault that caused the fault indicator has been corrected; identifying changes in the configuration or version of the software application associated with the correction; and generating a fault-indicator-solution combination.

[0174] The foregoing and other described embodiments may each optionally include one or more of the following features:

[0175] The first feature, which may be combined with any of the following features, wherein the relevant data records include deployed software application versions and components, error messages, and status information of extracted tasks.

[0176] The second feature, which can be combined with any of the previous or following features, is that the software pipeline includes deployment, testing and production.

[0177] A third feature, which may be combined with any previous or following feature, comprises: generating a notification of the same fault indicator when the same fault indicator in different software pipeline executions is detected; and recommending a recommended solution for the same fault indicator using the fault indicator-solution combination.

[0178] A fourth feature, which may be combined with any previous or following feature, comprises: using the fault indicator to monitor the landscape and software pipeline of other software applications to obtain the same fault indicator.

[0179] A fifth feature, which may be combined with any previous or following feature, comprises: generating a notification of the same fault indicator when the same fault indicator is detected in the landscape and software pipeline of other software applications; and recommending a recommended solution for the same fault indicator using the fault indicator-solution combination.

[0180] A sixth feature, which may be combined with any of the previous or following features, comprises: instructing the recommendation engine to re-execute the machine learning model training after a specified number of software pipeline runs.

[0181] The subject matter and implementation of the functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The software implementation of the described subject matter may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory computer-readable medium, for execution by a computer or computer-implemented system or for controlling the operation of a computer or computer-implemented system. Alternatively or additionally, the program instructions may be encoded in / on an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a receiver device for execution by a computer or computer-implemented system. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of computer storage media. Configuring one or more computers means that one or more computers have installed hardware, firmware, or software (or a combination of hardware, firmware, and software) so that when the software is executed by one or more computers, specific computing operations are performed. However, a computer storage medium is not a propagated signal.

[0182] The terms "real-time," "real-time," "real-time," "fast real-time (RFT)," "near real-time (NRT)," "quasi real-time," or similar terms (as understood by one of ordinary skill in the art) refer to an action and a response being close in time such that an individual perceives the action and the response to occur substantially simultaneously. For example, the time difference in response to displaying data (or initiating display) following an individual's action to access the data may be less than 1 millisecond (ms), less than 1 second (s), or less than 5s. Although the requested data need not be displayed (or initiated to be displayed) instantaneously, the requested data is displayed (or initiated to be displayed) without any intentional delay, taking into account the processing limitations of the described computing systems and, for example, the time required to collect, accurately measure, analyze, process, store, or transmit the data.

[0183] The term "data processing apparatus", "computer", "computing device" or "electronic computer device" (or equivalent terms understood by a person of ordinary skill in the art) refers to data processing hardware, and includes all kinds of devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The computer may also be or further include a dedicated logic circuit, such as a central processing unit (CPU), a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In some embodiments, the computer or computer-implemented system or dedicated logic circuit (or a combination of a computer or computer-implemented system and dedicated logic circuit) may be hardware-based or software-based (or a combination of hardware-based and software-based). The computer may optionally include code that creates an execution environment for a computer program, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system or a combination of execution environments. The present disclosure contemplates the use of a computer or computer-implemented system with an operating system (e.g., LINUX, UNIX, WINDOWS, MAC OS, ANDROID or IOS) or a combination of operating systems.

[0184] A computer program (which may also be referred to or described as a program, software, software application, unit, module, software module, script, code or other component) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including, for example, as a stand-alone program, module, component or subroutine, for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines or code portions. A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0185] Although portions of the programs shown in the various figures may be shown as separate components, such as units or modules, that use various objects, methods, or other processes to implement the described features and functions, the programs may alternatively include multiple sub-units, sub-modules, third-party services, components, libraries, and other components as appropriate. Conversely, the features and functions of the various components may be appropriately combined into a single component. The thresholds used to make computational determinations may be determined statically, dynamically, or both statically and dynamically.

[0186] The described methods, processes, or logic flows represent one or more examples of functionality consistent with the present disclosure, and are not intended to limit the present disclosure to the implementations described or shown, but are in accordance with the widest scope consistent with the described principles and features. The described methods, processes, or logic flows may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output data. The methods, processes, or logic flows may also be performed by a special-purpose logic circuit (e.g., a CPU, FPGA, or ASIC), and the computer may also be implemented as a special-purpose logic circuit (e.g., a CPU, FPGA, or ASIC).

[0187] A computer for executing a computer program may be based on a general or special purpose microprocessor, both or another type of CPU. Typically, the CPU will receive instructions and data from a memory and write instructions and data to the memory. The basic elements of a computer are a CPU for executing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (such as disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to one or more mass storage devices (such as disks, magneto-optical disks, or optical disks) for storing data, receive data from it or transmit data to it or both. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable memory storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0188] Non-transitory computer-readable media for storing computer program instructions and data may include all forms of permanent / non-permanent or volatile / non-volatile memory, media and memory devices, including, for example, semiconductor memory devices, such as random access memory (RAM), read-only memory (ROM), phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) and flash memory devices; magnetic devices, such as magnetic tapes, cassettes, cartridges, internal / removable disks; magneto-optical disks; and optical memory devices, such as digital versatile / video disk (DVD), compact disk (CD)-ROM, DVD+ / -R, DVD-RAM, DVD-ROM, high definition / density (HD)-DVD and Blu-ray / Blu-ray disc (BD), as well as other optical memory technologies. The memory may store various objects or data, including caches, classes, frameworks, applications, modules, backup data, jobs, web pages, web page templates, data structures, database tables, repositories storing dynamic information, or other appropriate information including any parameters, variables, algorithms, instructions, rules, constraints, or references. In addition, the memory may include other appropriate data, such as logs, policies, security or access data or report files. The processor and memory may be supplemented by or incorporated into special purpose logic circuitry.

[0189] To provide interaction with a user, implementations of the subject matter may include: The descriptions in this specification may be implemented on a computer having a display device (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), or a plasma monitor) for displaying information to a user, and a keyboard and a pointing device (e.g., a mouse, a trackball, or a trackpad) through which a user can provide input to a computer. A touch screen may also be used to provide input to a computer, such as a pressure-sensitive tablet computer surface or a multi-point touch screen using capacitive or electrical sensing. Other types of devices may be used to interact with a user. For example, the feedback provided to a user may be any form of sensory feedback (such as visual, auditory, tactile, or a combination of feedback types). Input from a user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a client computing device used by a user (e.g., by sending a web page to a web browser in response to a request received from a web browser on a user's mobile computing device).

[0190] The term "graphical user interface (GUI)" may be used in the singular or plural to describe one or more graphical user interfaces and each display of a particular graphical user interface. Thus, a GUI may represent any graphical user interface, including but not limited to a web browser, a touch screen, or a command line interface (CLI) that processes information and efficiently presents the information results to a user. Typically, a GUI may include a plurality of user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, drop-down lists, and buttons. These and other UI elements may be related to or represent the functionality of a web browser.

[0191] Implementations of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with implementations of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by wired or wireless digital data communication (or a combination of data communications) of any form or medium, such as a communication network. Examples of communication networks include a local area network (LAN), a radio access network (RAN), a metropolitan area network (MAN), a wide area network (WAN), a global interoperability for microwave access (WIMAX), a wireless local area network (WLAN) using, for example, 802.11x or other protocols, all or part of the Internet, another communication network, or a combination of communication networks. The communication network may communicate using, for example, Internet Protocol (IP) packets, frame relay frames, asynchronous transfer mode (ATM) cells, voice, video, data, or other information between network nodes.

[0192] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0193] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any inventive concept or the scope that can be claimed, but rather as descriptions of features that can be specific to a particular implementation of a particular inventive concept. Certain features described in this specification in the context of separate implementations may also be implemented in a single implementation in combination. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any sub-combination in a plurality. In addition, although the previously described features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0194] Specific embodiments of the subject matter have been described. Other embodiments, variations, and permutations of the described embodiments are within the scope of the appended claims and will be apparent to those skilled in the art. Although operations are depicted in a particular order in the drawings or claims, this should not be construed as requiring that the operations be performed in the particular order shown or in sequence, or that all of the illustrated operations (some operations may be considered optional) be performed, to achieve the desired results. In certain circumstances, multitasking or parallel processing (or a combination of multitasking and parallel processing) may be advantageous and is performed where deemed appropriate.

[0195] The previously described embodiments of the separation or integration of various system modules and components in this document should not be understood as requiring such separation or integration in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0196] Therefore, the previously described example embodiments do not define or constrain the present disclosure. Other changes, substitutions, and alterations are also possible without departing from the scope of the present disclosure.

[0197] Furthermore, any claimed embodiment is considered applicable to at least one computer-implemented method; a non-transitory computer-readable medium storing computer-readable instructions for performing the computer-implemented method; and a computer system comprising a computer memory interoperably coupled to a hardware processor configured to perform the computer-implemented method or the instructions stored on the non-transitory computer-readable medium.

Claims

1. A computer-implemented method comprising: To create the extracted data records, instructing the extraction filter to extract relevant data records from log messages of two runs of the software pipeline; instructing a difference filter to compare and identify message differences between two runs of a software pipeline in order to create a difference record using the extracted data record, wherein the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been taken; Instructing a recommendation engine to perform machine learning model training using the difference records; calling the recommendation engine using the difference record to analyze the difference record to obtain a fault indicator; determining, based on a later execution of the software pipeline, that the fault causing the fault indicator has been corrected; identifying a change in configuration or version of a software application associated with the correction; and Generates a fault-indicator-solution combination.

2. The computer-implemented method of claim 1 , wherein: The relevant data records include deployed software application versions and components, error messages, and status information of executed tasks.

3. The computer-implemented method of claim 1 , wherein: The software pipeline includes deployment, testing and production.

4. The computer-implemented method of claim 1 , comprising: upon detecting identical fault indicators in different software pipeline executions, generating a notification of the identical fault indicators; as well as The fault-indicator-solution combination is used to recommend a recommended solution for the same fault indicator.

5. The computer-implemented method of claim 1 , comprising: The fault indicators are used to monitor the landscape of other software applications and software pipelines for the same fault indicators.

6. The computer-implemented method of claim 5, comprising: upon detecting the same fault indicator in the landscape and software pipeline of other software applications, generating a notification of the same fault indicator; as well as The fault-indicator-solution combination is used to recommend a recommended solution for the same fault indicator.

7. The computer-implemented method of claim 1 , comprising: Instruct the recommendation engine to re-execute the machine learning model training after a specified number of software pipeline runs.

8. A non-transitory computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations comprising: To create the extracted data records, instructing the extraction filter to extract relevant data records from log messages of two runs of the software pipeline; instructing a difference filter to compare and identify message differences between two runs of a software pipeline in order to create a difference record using the extracted data record, wherein the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been taken; Instructing a recommendation engine to perform machine learning model training using the difference records; calling the recommendation engine using the difference record to analyze the difference record to obtain a fault indicator; determining, based on a later execution of the software pipeline, that the fault causing the fault indicator has been corrected; identifying a change in configuration or version of a software application associated with the correction; and Generates a fault-indicator-solution combination.

9. The non-transitory computer readable medium of claim 8, wherein: The relevant data records include deployed software application versions and components, error messages, and status information of extracted tasks.

10. The non-transitory computer readable medium of claim 8, wherein: The software pipeline includes deployment, testing and production.

11. The non-transitory computer-readable medium of claim 8, comprising: upon detecting identical fault indicators in different software pipeline executions, generating a notification of the identical fault indicators; as well as The fault-indicator-solution combination is used to recommend a recommended solution for the same fault indicator.

12. The non-transitory computer-readable medium of claim 8, comprising: The fault indicators are used to monitor the landscape of other software applications and software pipelines for the same fault indicators.

13. The non-transitory computer readable medium of claim 12, comprising: upon detecting the same fault indicator in the landscape and software pipeline of other software applications, generating a notification of the same fault indicator; as well as The fault-indicator-solution combination is used to recommend a recommended solution for the same fault indicator.

14. The non-transitory computer-readable medium of claim 8, comprising: Instruct the recommendation engine to re-execute the machine learning model training after a specified number of software pipeline runs.

15. A computer-implemented system comprising: one or more computers; as well as One or more computer memory devices interoperably coupled with the one or more computers and having a tangible, non-transitory, machine-readable medium storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising: To create the extracted data records, instructing the extraction filter to extract relevant data records from log messages of two runs of the software pipeline; instructing a difference filter to compare and identify message differences between two runs of a software pipeline in order to create a difference record using the extracted data record, wherein the difference record is modified using tagged data state information of the software pipeline run from which the extracted data record has been taken; Instructing a recommendation engine to perform machine learning model training using the difference records; calling the recommendation engine using the difference record to analyze the difference record to obtain a fault indicator; determining, based on a later execution of the software pipeline, that the fault causing the fault indicator has been corrected; identifying a change in configuration or version of a software application associated with the correction; and Generates a fault-indicator-solution combination.

16. The computer-implemented system of claim 15, wherein: The relevant data records include deployed software application versions and components, error messages, and status information of extracted tasks.

17. The computer-implemented system of claim 15, wherein: The software pipeline includes deployment, testing and production.

18. The computer-implemented system of claim 15, comprising: upon detecting identical fault indicators in different software pipeline executions, generating a notification of the identical fault indicators; as well as The fault-indicator-solution combinations are used to recommend recommended solutions for the same fault indicators.

19. The computer-implemented system of claim 15, comprising: The fault indicators are used to monitor the landscape of other software applications and software pipelines for the same fault indicators.

20. The computer-implemented system of claim 19, comprising: upon detecting the same fault indicator in the landscape and software pipeline of other software applications, generating a notification of the same fault indicator; and using the fault-indicator-solution combination to recommend a recommended solution for the same fault indicator.

Images