A system and a method for analysing log data for operational and security anomalies

Abstract

A system (100) for analyzing log data for operational and security anomalies is provided. A preparation module (120) prepares log data by determining the necessary volume, evaluating suitability, and categorizing it. A pattern discovery module (125) identifies structures in the log data using a plurality of methods. A patterned-message structure module ( 130) creates a structured format for each log message and computes hashes for data integrity. A processing module (135) stores and validates the structured logs. A message pattern curation module (140) customizes patterned messages and offers suggestions. A log message grouping module (145) organizes log messages into groups based on patterns and binding variables. An error identification module (150) detects and categorizes errors, highlighting operational and security issues. Finally, a log storage module (155) stores configurations, static log data, and variable content efficiently.

Description

[0001] A SYSTEM AND A METHOD FOR ANALYSING LOG DATA FOR OPERATIONAL AND SECURITY ANOMALIES

[0002] EARLIEST PRIORITY DATE:

[0003] This Application claims priority from a Complete patent application filed in India having Patent Application No. 202441066677, filed on September 03, 2024, and titled “A SYSTEM AND A METHOD FOR ANALYSING LOG DATA FOR OPERATIONAL AND SECURITY ANOMALIES”.

[0004] FIELD OF INVENTION

[0005] Embodiments of the present disclosure relate to the field of data analysis, and more particularly, a system and a method for analyzing log data for operational and security anomalies.

[0006] BACKGROUND

[0007] Log data refers to records of all events that occur within a system, software, or network device. The log data captures a wide range of information, including errors, operational issues, and general status updates. Log files, which are generated by software, contain detailed information about the operations, activities, and usage patterns of an application, server, or IT system. The log files serve as a historical record of all processes, events, and messages, often accompanied by additional descriptive data such as timestamps. Timestamps are crucial for understanding not only what occurred within the system but also precisely when these events took place. The log files provide a detailed and accessible record of system information that would otherwise be difficult to compile, offering valuable insights into the performance and compliance of applications and systems. Organizations use log data for various purposes, including identifying and troubleshooting errors, improving operations, enhancing efficiency, understanding user behaviour, strengthening security and the like.

[0008] However, several challenges are associated with managing and analysing log data. Modern IT systems generate vast quantities of log data, as they record every activity within the system. This results in a rapid accumulation of log files, both in number and size, making it increasingly difficult to analyse and interpret the data effectively. Traditional log analysis methods, which often rely on manual inspection, can quickly become overwhelming without proper procedures in place. Further, different systems generate log data in various formats includes structured, semi-structured, or unstructured further complicating the analysis process.

[0009] Another challenge is the time and cost associated with issue resolution. Identifying and resolving issues within a system can be both time-consuming and costly, particularly when dealing with large volumes of log data. Traditional log analysis methods often lack the precision needed for accurate problem detection. Moreover, many existing systems lack the capability to provide near-real-time functional and operational health statistics and alerts, delaying the necessary actions to address emerging issues. Additional difficulties include the complexity of pattern discovery and curation, log grouping and anomaly detection, and efficient log storage, all of which are critical to effective log data management.

[0010] Hence, there is a need for an improved system and a method for analyzing log data for operational and security anomalies which addresses the aforementioned issue(s).

[0011] OBJECTIVES OF THE INVENTION

[0012] The primary objective of the invention is to develop a system for analyzing log data that offers suggestions for potential solutions and tracks various processing details, including request sources, response times, trends, number of active users and the like. Another objective of the invention is to generate comprehensive functional and operational health statistics for applications in real-time, enabling quick identification of issues and timely decision-making.

[0013] Yet another objective of the invention to develop a system for identifying patterns within log data, including predefined global patterns and various types of exceptions, using techniques like a static-first approach, a variable-first approach, and machine learning based approach for pattern relevance.

[0014] Yet another objective of the invention is to create a structured format for log messages by assigning pattern structures, computing hashes (SHA2) for static and full messages and storing these in a Structured Data Store (SCDS) with capabilities for further validation and review.

[0015] Yet another objective of the invention is to provide optimized storage and search capabilities for log data by categorizing static and variable content, enabling full-text search and efficient storage solutions.

[0016] BRIEF DESCRIPTION

[0017] In accordance with an embodiment of the present disclosure, a system for analyzing log data for operational and security anomalies is provided. The system includes a processing subsystem hosted on a server. The processing subsystem is configured to execute on a network to control bidirectional communications among a plurality of modules. The processing subsystem includes a preparation module. The preparation module is configured to prepare a log data for analysis by determining a required volume of log data needed, evaluating suitability of the log data from an environment test based on variance, and categorizing the log data. The processing subsystem includes a pattern discovery module operatively coupled to the preparation module. The pattern discovery module is configured to detect structures and characteristics of log messages pre-processed by the preparation module using a plurality of methods. The pattern discovery module is configured to employ a predefined global pattern on the log data. The predefined global pattern includes common log formats, date patterns, IP address patterns, uniform resource locator patterns, uniform resource identifier patterns, numerical patterns, alphanumeric, and field enclosures. The pattern discovery module is configured to identify a plurality of exception patterns in log data by applying a general-purpose pattern. The pattern discovery module is configured to employ at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages. The processing subsystem includes a patterned- message structure module operatively coupled to the pattern discovery module. The patterned-message structure module is configured to create a structured format for each log message. The patterned-message structure module is configured to assign an Application ID to each log message to identify the source application from which the log message originated. The patterned-message structure module is configured to define a pattern structure for log messages. The pattern structure includes pattern ID, pattern representing processing logic, and custom key names. The patterned-message structure module is configured to assign a Patterned-Message ID to each structured log messages. The patterned-message structure module is configured to maintain a list of pattern IDs, each corresponding to a matched pattern within the log messages. The patterned-message structure module is configured to compute a Secure Hash Algorithm 2 hash of the Static Message Part to ensure data integrity and facilitate quick comparison. The patterned-message structure module is configured to compute the Secure Hash Algorithm 2 hash of entire log message, comprising the static and patterned parts. The patterned-message structure module is configured to retain a Pattern ID for a binding variable within the log message. The processing subsystem includes a processing module operatively coupled to the patterned-message structure module. The processing module is configured to process each log message formatted and structured by the patterned-message structure module. The processing module is configured to store the patterned-message structure in a structured data store. The stored structure maintains a validation status that indicate whether the log message requires validation and review. The processing module is configured to track a plurality of details on the processing. The plurality of details includes number of requests, sources of the requests, response times, trends, and number of active users. The processing module is configured to generate comprehensive functional and operational health statistics for the application in real-time. The processing subsystem includes a message pattern curation module operatively coupled to the processing module. The message pattern curation module is configured to curate and customize a plurality of patterned messages. The message pattern curation module is configured to display the plurality of patterned messages with colour coding, to highlight variables and likely variables. The message pattern curation module is configured to offer a plurality of suggestions for potential solutions based on the analysis. The processing subsystem includes a log message grouping module operatively coupled to the message pattern curation module. The log message grouping module is configured to organize the log messages into structured groups based on identified patterns and binding variables. The log message grouping module is configured to discover binding variables and identify variables that span across multiple messages and link together. The log message grouping module is configured to collect messages in sequence. The log message grouping module is configured to apply patterns to categorize the collected messages. The log message grouping module is configured to identify and analyse repeating patterns within message groups. The log message grouping module is configured to allow experts to review, adjust, and confirm groupings. The log message grouping module is configured to store message group with Message Group ID. The processing subsystem includes an error identification module operatively coupled to the log message grouping module. The error identification module is configured to identify and categorize error sequences within the log messages of an application. The error indicates operational and security anomalies. The error identification module is configured to collect list of phrases that indicate errors in the application. The error identification module is configured to check each Group-Pattern for error phrases. The processing subsystem includes a log storage module operatively coupled to the error identification module. The log storage module is configured to store configurations, static log data and variable log content. The static log data stored a single time.

[0018] In accordance with another embodiment of the present disclosure, a method for analyzing log data for operational and security anomalies is provided. The method includes preparing, by a preparation module, a log data for analysis by determining a required volume of log data needed, evaluating suitability of the log data from an environment test based on variance, and categorizing the log data. The method includes detecting, by a pattern discovery module, structures and characteristics of log messages pre-processed by the preparation module using a plurality of methods. The method includes employing, by the pattern discovery module, a predefined global pattern on the log data. The predefined global pattern includes common log formats, date patterns, IP address patterns, uniform resource locator patterns, uniform resource identifier patterns, numerical patterns, alphanumeric, and field enclosures. The method includes identifying, by the pattern discovery module, a plurality of exception patterns in log data by applying a general-purpose pattern. The method includes employing, by the pattern discovery module, at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages. The method includes creating, by a patterned-message structure module, a structured format for each log message. The method includes assigning, by the patterned-message structure module, an Application ID to each log message to identify the source application from which the log message originated. The method includes defining, by the patterned-message structure module, a pattern structure for log messages. The pattern structure includes pattern ID, pattern representing processing logic, and custom key names. The method includes assigning, by the patterned-message structure module, a Patterned-Message ID to each structured log messages. The method includes maintaining, by the patterned- message structure module, a list of pattern IDs, each corresponding to a matched pattern within the log messages. The method includes computing, by the patterned- message structure module, a Secure Hash Algorithm 2 hash of the Static Message Part to ensure data integrity and facilitate quick comparison. The method includes computing, by the patterned-message structure module, the Secure Hash Algorithm 2 hash of entire log message, comprising the static and patterned parts. The method includes retaining, by the patterned-message structure module, a Pattern ID for a binding variable within the log message. The method includes processing, by a processing module, each log message formatted and structured by the patterned- message structure module. The method includes storing, by the processing module, the patterned-message structure in a structured data store. The stored structure maintains a validation status that indicate whether the log message requires validation and review. The method includes tracking, by the processing module, a plurality of details on the processing. The plurality of details includes number of requests, sources of the requests, response times, trends, and number of active users. The method includes generating, by the processing module, comprehensive functional and operational health statistics for the application in real-time. The method includes curating, by a message pattern curation module, and customize a plurality of patterned messages. The method includes displaying, by the message pattern curation module, the plurality of patterned messages with colour coding, to highlight variables and likely variables. The method includes offering, by the message pattern curation module, a plurality of suggestions for potential solutions based on the analysis. The method includes organizing, by a log message grouping module, the log messages into structured groups based on identified patterns and binding variables. The method includes discovering, by the log message grouping module, binding variables and identify variables that span across multiple messages and link together. The method includes collecting, by the log message grouping module, messages in sequence. The method includes applying, by the log message grouping module, patterns to categorize the collected messages. The method includes identifying and analyzing, by the log message grouping module, repeating patterns within message groups. The method includes allowing, by the log message grouping module, experts to review, adjust, and confirm groupings. The method includes storing, by the log message grouping module, message group with message group ID. The method includes identifying and categorizing, by an error identification module, error sequences within the log messages of an application. The error indicates operational and security anomalies. The method includes collecting, by the error identification module, list of phrases that indicate errors in the application. The method includes checking, by the error identification module, each Group-Pattern for error phrases. The method includes storing, by a log storage module, configurations, static log data and variable log content. The static log data stored a single time.

[0019] To further clarify the advantages and features of the present disclosure, a more particular description of the disclosure will follow by reference to specific embodiments thereof, which are illustrated in the appended figures. It is to be appreciated that these figures depict only typical embodiments of the disclosure and are therefore not to be considered limiting in scope. The disclosure will be described and explained with additional specificity and detail with the appended figures.

[0020] BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The disclosure will be described and explained with additional specificity and detail with the accompanying figures in which:

[0022] FIG. 1 is a block diagram representation of a system for analyzing log data for operational and security anomalies in accordance with an embodiment of the present disclosure;

[0023] FIG. 2 is a block diagram of a computer or a server in accordance with an embodiment of the present disclosure;

[0024] FIG. 3(a) illustrates a flow chart representing the steps involved in a method for analyzing log data for operational and security anomalies in accordance with an embodiment of the present disclosure; FIG. 3(b) illustrates continued steps of the method of FIG. 3(a) in accordance with an embodiment of the present disclosure.

[0025] FIG. 3(c) illustrates continued steps of the method of FIG. 3(b) in accordance with an embodiment of the present disclosure;

[0026] FIG. 3(d) illustrates continued steps of the method of FIG. 3(c) in accordance with an embodiment of the present disclosure; and

[0027] FIG. 3(e) illustrates continued steps of the method of FIG. 3(d) in accordance with an embodiment of the present disclosure.

[0028] Further, those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and may not have necessarily been drawn to scale. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the figures by conventional symbols, and the figures may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the figures with details that will be readily apparent to those skilled in the art having the benefit of the description herein.

[0029] DETAILED DESCRIPTION

[0030] For the purpose of promoting an understanding of the principles of the disclosure, reference will now be made to the embodiment illustrated in the figures and specific language will be used to describe them. It will nevertheless be understood that no limitation of the scope of the disclosure is thereby intended. Such alterations and further modifications in the illustrated system, and such further applications of the principles of the disclosure as would normally occur to those skilled in the art are to be construed as being within the scope of the present disclosure. The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such a process or method. Similarly, one or more devices or subsystems or elements or structures or components preceded by "comprises... a" does not, without more constraints, preclude the existence of other devices, sub-systems, elements, structures, components, additional devices, additional sub-systems, additional elements, additional structures or additional components. Appearances of the phrase "in an embodiment", "in another embodiment" and similar language throughout this specification may, but not necessarily do, all refer to the same embodiment.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this disclosure belongs. The system, methods, and examples provided herein are only illustrative and not intended to be limiting.

[0032] In the following specification and the claims, reference will be made to a number of terms, which shall be defined to have the following meanings. The singular forms “a”, “an”, and “the” include plural references unless the context clearly dictates otherwise.

[0033] Embodiments of the present disclosure relate to a system for analyzing log data for operational and security anomalies. The system includes a processing subsystem hosted on a server. The processing subsystem is configured to execute on a network to control bidirectional communications among a plurality of modules. The processing subsystem includes a preparation module. The preparation module is configured to prepare a log data for analysis by determining a required volume of log data needed, evaluating suitability of the log data from an environment test based on variance, and categorizing the log data. The processing subsystem includes a pattern discovery module operatively coupled to the preparation module. The pattern discovery module is configured to detect structures and characteristics of log messages pre-processed by the preparation module using a plurality of methods. The pattern discovery module is configured to employ a predefined global pattern on the log data. The predefined global pattern includes common log formats, date patterns, IP address patterns, uniform resource locator patterns, uniform resource identifier patterns, numerical patterns, alphanumeric, and field enclosures. The pattern discovery module is configured to identify a plurality of exception patterns in log data by applying a general-purpose pattern. The pattern discovery module is configured to employ at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages. The processing subsystem includes a patterned-message structure module operatively coupled to the pattern discovery module. The patterned-message structure module is configured to create a structured format for each log message. The patterned-message structure module is configured to assign an Application ID to each log message to identify the source application from which the log message originated. The patterned-message structure module is configured to define a pattern structure for log messages. The pattern structure includes pattern ID, pattern representing processing logic, and custom key names. The patterned-message structure module is configured to assign a Patterned-Message ID to each structured log messages. The patterned- message structure module is configured to maintain a list of pattern IDs, each corresponding to a matched pattern within the log messages. The patterned-message structure module is configured to compute a Secure Hash Algorithm 2 hash of the Static Message Part to ensure data integrity and facilitate quick comparison. The patterned- message structure module is configured to compute the Secure Hash Algorithm 2 hash of entire log message, comprising the static and patterned parts. The patterned-message structure module is configured to retain a Pattern ID for a binding variable within the log message. The processing subsystem includes a processing module operatively coupled to the patterned-message structure module. The processing module is configured to process each log message formatted and structured by the patterned- message structure module. The processing module is configured to store the patterned- message structure in a structured data store. The stored structure maintains a validation status that indicate whether the log message requires validation and review. The processing module is configured to track a plurality of details on the processing. The plurality of details includes number of requests, sources of the requests, response times, trends, and number of active users. The processing module is configured to generate comprehensive functional and operational health statistics for the application in realtime. The processing subsystem includes a message pattern curation module operatively coupled to the processing module. The message pattern curation module is configured to curate and customize a plurality of patterned messages. The message pattern curation module is configured to display the plurality of patterned messages with colour coding, to highlight variables and likely variables. The message pattern curation module is configured to offer a plurality of suggestions for potential solutions based on the analysis. The processing subsystem includes a log message grouping module operatively coupled to the message pattern curation module. The log message grouping module is configured to organize the log messages into structured groups based on identified patterns and binding variables. The log message grouping module is configured to discover binding variables and identify variables that span across multiple messages and link together. The log message grouping module is configured to collect messages in sequence. The log message grouping module is configured to apply patterns to categorize the collected messages. The log message grouping module is configured to identify and analyze repeating patterns within message groups. The log message grouping module is configured to allow experts to review, adjust, and confirm groupings. The log message grouping module is configured to store message group with Message Group ID. The processing subsystem includes an error identification module operatively coupled to the log message grouping module. The error identification module is configured to identify and categorize error sequences within the log messages of an application. The error indicates operational and security anomalies. The error identification module is configured to collect list of phrases that indicate errors in the application. The error identification module is configured to check each Group-Pattern for error phrases. The processing subsystem includes a log storage module operatively coupled to the error identification module. The log storage module is configured to store configurations, static log data and variable log content. The static log data stored a single time.

[0034] FIG. 1 is a block diagram of a system (100) for analyzing log data for operational and security anomalies in accordance with an embodiment of the present disclosure. The system (100) includes a processing subsystem (105) hosted on a server (108). In one embodiment, the server (108) may include a cloud-based server. In another embodiment, parts of the server (108) may be a local server coupled to a user device (not shown in FIG.l). The processing subsystem (105) is configured to execute on a network (115) to control bidirectional communications among a plurality of modules. In one example, the network (115) may be a private or public local area network (LAN) or Wide Area Network (WAN), such as the Internet. In another embodiment, the network (115) may include both wired and wireless communications according to one or more standards and / or via one or more transport mediums. In one example, the network (115) may include wireless communications according to one of the 802.11 or Bluetooth specification sets, or another standard or proprietary wireless communication protocol. In yet another embodiment, the network (115) may also include communications over a terrestrial cellular network, including, a global system for mobile communications (GSM), code division multiple access (CDMA), and / or enhanced data for global evolution (EDGE) network.

[0035] The processing subsystem (105) includes a preparation module (120). The preparation module (120) is configured to prepare log data to ensure that the log data is ready for further processing and analysis. The preparation process involves several key tasks. Initially, the preparation module (120) determines the amount of log data that will be sufficient for learning and analysis and assesses how much data is necessary to effectively capture the characteristics and patterns within the log data. Further, the preparation module (120) decides how the log data will be ingested into the system (100). There are generally two main methods for log ingestion: ingesting as a set of stand-alone files and ingesting via a client and an ingestion API from a running system. The preparation module (120) also evaluates suitability of the log data from an environment test based on variance. Finally, the preparation module (120) categorizes the log data with a particular focus on understanding error and exception logs.

[0036] The processing subsystem (105) includes an ingestion module operatively coupled to the preparation module (120). The ingestion module is configured to input the log data from a plurality of sources for delivering to the preparation module (120). The log data includes but is not limited to application logs, operational logs, security logs and the like. The plurality of sources includes existing log files, log files generated in real-time by the application. The ingestion module supports two primary methods for gathering log data: ingesting as the set of stand-alone files and ingesting via the client and the ingestion API from the running system. Ingesting as the set of stand-alone files involves identifies all relevant multi-line patterns within the log files to ensure each log message is accurately recognized and processed. Once these patterns are identified, each log message is ingested as though it were received through the ingestion API service. Ingesting via the client and the ingestion API support real time ingestion. The client continuously monitors the application’s log file, reading each log message as it is updated. Since this approach captures logs as they are generated, there is no need for identifying individual log messages. This approach includes a robust API capable of ingesting one or more log messages simultaneously.

[0037] The processing subsystem (105) includes a pattern discovery module (125) operatively coupled to the preparation module (120). The pattern discovery module (125) is configured to detect structures and characteristics of log messages pre-processed by the preparation module (120) using a plurality of methods. The plurality of methods includes the predefined global pattern, the static first approach, the variable first approach and the machine learning based approach. The pattern discovery module (125) begins by employing a predefined global pattern on the log data. The predefined global pattern is a comprehensive set of well-known patterns used in log analysis, which includes, but is not limited to common log formats, date patterns, IP address patterns, uniform resource locator (URL) patterns, uniform resource identifier (URI) patterns, numerical patterns, alphanumeric, field enclosures and the like. The common log formats include Apache WS pattern. IP address patterns include patterns for both IPv4 and IPv6 addresses, which are essential for tracking network-related events and activities. The URL and URI patterns identify and extract URLs and URIs from log messages. The alphanumeric patterns identify alphanumeric from log messages. The field enclosures pattern identifying fields inside enclosures like (), { }, [], and the like.

[0038] The pattern discovery module (125) is configured to identify a plurality of exception patterns in log data by applying a general-purpose pattern. The general-purpose pattern is particularly useful for multi-line exceptions. The general-purpose pattern serves as a first filter to capture exception-related information. Once the general-purpose pattern is applied, the pattern discovery module (125) lists all exception names found in the log data and gets the corresponding exception messages. The exception messages, along with their static components, are then used as identifiers for further pattern discovery using advanced approaches. The output of the general-purpose pattern includes a list of exception names and a list of exception messages, providing a detailed understanding of the exception patterns present in the log data.

[0039] The pattern discovery module (125) is configured to employ at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages.

[0040] In the static first approach, the pattern discovery module (125) tokenizes log messages based on whitespace characters. Further, tokens are categorized as static, variable, or likely variable. A token is categorized as static, if all its characters are of alphabetic characters or one that is found in a well-known English dictionary, such as the Apache Lucene dictionary. Consecutive non-static tokens are merged, and predefined global patterns are applied recursively to identify static and variable components. Tokens matching the predefined pattern are marked as variables, while those that do not match any patterns are marked as likely variables. The final output includes a list of static tokens, a list of variables (with corresponding patterns), and a list of likely variable tokens, all maintained in their original order.

[0041] In the variable first approach, the pattern discovery module (125) first applies the predefined patterns to the log message in order of relevance, marking segments that match with a specific pattern name, such as $$PATTERN_<NAME>$$. The message is then tokenized based on whitespace characters, with tokens being classified as variables if they match a pattern. Further, those that have matched a pattern, if found to be static mark as static else mark as likely variable. Finally, providing an output similar to the static-first approach.

[0042] In the machine learning based approach, the pattern discovery module (125) extracts only the static tokens from the log message and generates a Message Key by hashing a static part. The log messages are grouped based on either the same Message Key or a similarity score that meets a predefined threshold based on an ensemble of similarity algorithms. Within the groups, the pattern discovery module (125) applies clustering algorithms to group all similar phrases, followed by applying different patterns to each cluster and selecting the top patterns that match. By default, the top matching pattern is selected. The output of the machine learning based approach includes a list of the top recommended patterns. At the end of processes in the pattern discovery module (125), a list of ordered tokens that are marked as static, variable or likely variable are generated.

[0043] The processing subsystem (105) includes a patterned-message structure module (130) operatively coupled to the pattern discovery module (125). The patterned-message structure module (130) is configured to create a structured format for each log message. For each log message, the patterned-message structure module (130) assigns an Application ID and defines a pattern structure for log messages. The pattern structure includes pattern ID representing ID of the pattern, pattern representing processing logic typically a regex or a Class and custom key names. Each structured log message is given a unique Patterned-Message ID (PMID) and maintains a list of Pattern IDs, capturing all patterns matched within the message. The patterned-message structure module (130) also separates the static message part (SMP), which consists of the log content without any variable parts. To ensure data integrity, the patterned-message structure module (130) computes a Secure Hash Algorithm (SHA) 2 hash of the static message part (SMPD) and another SHA2 hash of entire log message, comprising the static and patterned parts (PMD). The patterned-message structure module (130) is configured to retain a Pattern ID for a binding variable within the log message. The binding variable is a variable within the log messages that can be used to link or group together multiple related log entries.

[0044] The processing subsystem (105) includes a processing module (135) operatively coupled to the patterned-message structure module (130). The processing module (135) is configured to process each log message formatted and structured by the patterned- message structure module (130). The processing starts with the list of tokens that are marked as static, variable and likely variable and the matching log messages. The processing module (135) is configured to store the patterned-message structure in a structured data store (SCDS). The stored structure maintains a validation status that indicates whether the log message requires validation and review. The processing module (135) is configured to track a plurality of details on the processing. The plurality of details includes number of requests, sources of the requests, response times, trends, and number of active users. The processing module (135) is configured to generate comprehensive functional and operational health statistics for the application in real-time. The processing subsystem (105) includes a message pattern curation module (140) operatively coupled to the processing module (135). The message pattern curation module (140) is configured to curate and customize a plurality of patterned messages. The message pattern curation module (140) is configured to display the plurality of patterned messages with color coding, to highlight variables and likely variables. Experts are then able to review and refine the identification of variables, marking them as valid, or reclassifying likely variables as either static or variable. Additionally, the message pattern curation module (140) allows for the customization of variable names, such as labeling an IP address as "ClientIP" or a URL as " So-and-So- Service URL." In cases, if there are multiple patterns match a variable, the expert can select the most appropriate one, or if necessary, suggest a different set of patterns and retrigger the pattern discovery. Once this curation process is complete, the message pattern curation module (140) finalizes the patterned messages as reviewed and confirmed, with updates to the patterned messages being versioned for future reference and learning, message pattern curation module (140) is configured to offer a plurality of suggestions for potential solutions based on the analysis.

[0045] The processing subsystem (105) includes a log message grouping module (145) operatively coupled to the message pattern curation module (140). The log message grouping module (145) is configured to organize the log messages into structured groups based on identified patterns and binding variables. The log message grouping module (145) is configured to discover binding variables and identify variables that span across multiple messages and link together. The log message grouping module (145) is configured to collect messages in sequence. The log message grouping module (145) is configured to apply patterns to categorize the collected messages. The log message grouping module (145) is configured to identify and analyze repeating patterns within message groups. Lor each group, determine how often these repeating patterns occur, which helps in identifying common behaviors or issues within the system (100). Develop a Group-Pattern for each set of repeating sequences, associating them with their respective occurrence counts. For each group pattern also find a subset or superset of Group-Pattern that have T% overlap. For every group-pattern maintain a similar Group-Pattern that’s either a subset or superset. Further, present a random set of “grouped” log messages and highlight each of the Group-Patterns with different color coding. The log message grouping module (145) is configured to allow experts to review, adjust, and confirm groupings. The log message grouping module (145) is configured to store message groups with Message Group ID(MGID).

[0046] The processing subsystem (105) includes an error identification module (150) operatively coupled to the log message grouping module (145). The error identification module (150) is configured to identify and categorize error sequences within the log messages of an application. The error indicates operational and security anomalies. The error identification module (150) is configured to collect list of phrases that indicate errors in the application. The error identification module ( 150) is configured to check each group-pattern for error phrases. As part of the curation process, each group-pattern presents similar group-patterns that are marked as error-patterns. Experts can then review and confirm both the similarity and whether the identified error pattern is correctly associated with the group-pattern under evaluation.

[0047] The processing subsystem (105) includes a log storage module (155) is operatively coupled to the error identification module (150). The log storage module (155) is configured to store configurations, static log data and variable log content. The static log data stored a single time

[0048] It must be noted that the store configurations, static log data and variable log content are stored in their respective storage systems: the config store, static log store, and variable log store. The config store holds configuration data, including identified patterns. The static log store is designed for storing static content with full-text search capabilities. The variable log store manages storage for variable content with capability to store and search similar kinds of data in an optimal manner. Since the log data is time stamped data, the variable log store is also optimized for time-based storage and searches. This data is further partitioned with time.

[0049] It must be noted that, for real time log data, an agent such as Logstash is usually employed on the application server to collect the logs as they are created. The Logstash sends logs immediately to the server (108), where each log message is assigned a sequence and placed onto a message bus for further processing. This allows for immediate analysis, pattern matching, and storage. The message received on the bus is further processed based on the discovered message-patterns. If a message does not match any known patterns, it is forwarded "as is" for further learning. Messages that do match a pattern are sent for group processing based on the binding variable. Messages that are received for a single binding variable are first sequenced. Group- Pattern is applied to identify the right message group. Messages that do not match any group pattern are diverted “as is” for Learning. Messages that match a Group-Pattern are then forwarded for storage. Static content of the log messages is stored only once along with Message-Pattern. This significantly reduces the overall storage requirement. All variables along with their custom names are extracted out of the Group message and stored in a Variable Log Store. Static content is indexed by Application Log Type.

[0050] It must be noted that, in addition to standard configuration, each module described in the specification, including the preparation module, pattern discovery module, patterned-message structure module, processing module, a message pattern curation module, a log message grouping module, an error identification module, and a log storage module has the ability to self-learn. The self-learning capabilities in modules can be implemented using various technologies including machine learning, deep learning, natural language processing and the like.

[0051] It is to be noted that the system may comprise, but is not limited to, a mobile phone, desktop computer, portable digital assistant (PDA), smart phone, tablet, ultra-book, netbook, laptop, multi-processor system, microprocessor-based or programmable consumer electronic system, or any other communication device that a user may use. In some embodiments, the system may comprise a display module (not shown) to display information (for example, in the form of user interfaces). In further embodiments, the system may comprise one or more of touch screens, accelerometers, gyroscopes, cameras, microphones, global positioning system (GPS) devices, and so forth.

[0052] In one embodiment, the various functional components of the system may reside on a single computer, or they may be distributed across several computers in various arrangements. The various components of the system may, furthermore, access one or more databases, and each of the various components of the system may be in communication with one another. Further, while the components of FIG. 1 are discussed in the singular sense, it will be appreciated that in other embodiments multiple instances of the components may be employed.

[0053] FIG. 2 is a block diagram of a computer or a server in accordance with an embodiment of the present disclosure. The server (200) includes processor(s) (230), and memory (210) operatively coupled to the bus (220). The processor(s) (230), as used herein, means any type of computational circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing microprocessor, a reduced instruction set computing microprocessor, a very long instruction word microprocessor, an explicitly parallel instruction computing microprocessor, a digital signal processor, or any other type of processing circuit, or a combination thereof.

[0054] The memory (210) includes several subsystems stored in the form of executable program which instructs the processor (230) to perform the method steps illustrated in FIG. 1. The memory (210) includes a processing subsystem (105) of FIG.l. The processing subsystem (105) further has following modules: a preparation module (120), a pattern discovery module (125), a patterned-message structure module (130), a processing module (135), a message pattern curation module (140), a log message grouping module (145), an error identification module (150), and a log storage module (155).

[0055] In accordance with an embodiment of the present disclosure, a system (100) for analyzing log data for operational and security anomalies is provided. The system (100) includes a processing subsystem (105) hosted on a server (108). The processing subsystem (105) is configured to execute on a network to control bidirectional communications among a plurality of modules. The processing subsystem (105) includes a preparation module (120). The preparation module (120) is configured to prepare a log data for analysis by determining a required volume of log data needed, evaluating suitability of the log data from an environment test based on variance, and categorizing the log data. The processing subsystem (105) includes a pattern discovery module (125) operatively coupled to the preparation module (120). The pattern discovery module (125) is configured to detect structures and characteristics of log messages pre-processed by the preparation module (120) using a plurality of methods. The pattern discovery module (125) is configured to employ a predefined global pattern on the log data. The predefined global pattern includes common log formats, date patterns, IP address patterns, uniform resource locator patterns, uniform resource identifier patterns, numerical patterns, alphanumeric, and field enclosures. The pattern discovery module (125) is configured to identify a plurality of exception patterns in log data by applying a general-purpose pattern. The pattern discovery module (125) is configured to employ at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages. The processing subsystem (105) includes a patterned-message structure module (130) operatively coupled to the pattern discovery module (125). The patterned-message structure module (130) is configured to create a structured format for each log message. The patterned-message structure module ( 130) is configured to assign an Application ID to each log message to identify the source application from which the log message originated. The patterned-message structure module (130) is configured to define a pattern structure for log messages. The pattern structure includes pattern ID, pattern representing processing logic, and custom key names. The patterned-message structure module (130) is configured to assign a Patterned-Message ID to each structured log messages. The patterned-message structure module (130) is configured to maintain a list of pattern IDs, each corresponding to a matched pattern within the log messages. The patterned- message structure module (130) is configured to compute a Secure Hash Algorithm 2 hash of the Static Message Part to ensure data integrity and facilitate quick comparison. The patterned-message structure module (130) is configured to compute the Secure Hash Algorithm 2 hash of entire log message, comprising the static and patterned parts. The patterned-message structure module ( 130) is configured to retain a Pattern ID for a binding variable within the log message. The processing subsystem (105) includes a processing module (135) operatively coupled to the patterned-message structure module (130). The processing module (135) is configured to process each log message formatted and structured by the patterned-message structure module (130). The processing module (135) is configured to store the patterned-message structure in a structured data store. The stored structure maintains a validation status that indicate whether the log message requires validation and review. The processing module (135) is configured to track a plurality of details on the processing. The plurality of details includes number of requests, sources of the requests, response times, trends, and number of active users. The processing module (135) is configured to generate comprehensive functional and operational health statistics for the application in realtime. The processing subsystem (105) includes a message pattern curation module (140) operatively coupled to the processing module (135). The message pattern curation module (140) is configured to curate and customize a plurality of patterned messages. The message pattern curation module (140) is configured to display the plurality of patterned messages with colour coding, to highlight variables and likely variables. The message pattern curation module (140) is configured to offer a plurality of suggestions for potential solutions based on the analysis. The processing subsystem (105) includes a log message grouping module (145) operatively coupled to the message pattern curation module (140). The log message grouping module (145) is configured to organize the log messages into structured groups based on identified patterns and binding variables. The log message grouping module (145) is configured to discover binding variables and identify variables that span across multiple messages and link together. The log message grouping module (145) is configured to collect messages in sequence. The log message grouping module (145) is configured to apply patterns to categorize the collected messages. The log message grouping module (145) is configured to identify and analyse repeating patterns within message groups. The log message grouping module (145) is configured to allow experts to review, adjust, and confirm groupings. The log message grouping module (145) is configured to store message group with Message Group ID. The processing subsystem (105) includes an error identification module (150) operatively coupled to the log message grouping module (145). The error identification module (150) is configured to identify and categorize error sequences within the log messages of an application. The error indicates operational and security anomalies. The error identification module (150) is configured to collect list of phrases that indicate errors in the application. The error identification module (150) is configured to check each Group-Pattern for error phrases. The processing subsystem (105) includes a log storage module (155) is operatively coupled to the error identification module (150). The log storage module (155) is configured to store configurations, static log data and variable log content. The static log data stored a single time.

[0056] The bus (220) as used herein refers to internal memory channels or computer network that is used to connect computer components and transfer data between them. The bus (220) includes a serial bus or a parallel bus, wherein the serial bus transmits data in bitserial format and the parallel bus transmits data across multiple wires. The bus (220) as used herein, may include but not limited to, a system bus, an internal bus, an external bus, an expansion bus, a frontside bus, a backside bus and the like. FIG. 3(a) illustrates a flow chart representing the steps involved in a method (300) for analyzing log data for operational and security anomalies in accordance with an embodiment of the present disclosure. FIG. 3(b) illustrates continued steps of the method (300) of FIG. 3(a) in accordance with an embodiment of the present disclosure. FIG. 3(c) illustrates continued steps of the method (300) of FIG. 3(b) in accordance with an embodiment of the present disclosure. FIG. 3(d) illustrates continued steps of the method (300) of FIG. 3(c) in accordance with an embodiment of the present disclosure. FIG. 3(e) illustrates continued steps of the method (300) of FIG. 3(d) in accordance with an embodiment of the present disclosure. The method (300) includes preparing, by a preparation module, a log data for analysis by determining a required volume of log data needed, evaluating suitability of the log data from an environment test based on variance, and categorizing the log data in step 305. Initially, the preparation module determines the amount of log data that will be sufficient for learning and analysis and assesses how much data is necessary to effectively capture the characteristics and patterns within the log data. Further, the preparation module decides how the log data will be ingested into the system. There are generally two main methods for log ingestion: ingesting as a set of stand-alone files and ingesting via a client and an ingestion API from a running system. The preparation module also evaluates suitability of the log data from an environment test based on variance. Finally, the preparation module categorizes the log data with a particular focus on understanding error and exception logs.

[0057] The method (300) also includes inputting by an ingestion module, the log data from a plurality of sources for delivering to the preparation module. The log data includes but is not limited to application logs, operational logs, security logs and the like. The plurality of sources includes existing log files, log files generated in real-time by the application. The ingestion module supports two primary methods for gathering log data: ingesting as the set of stand-alone files and ingesting via the client and the ingestion API from the running system. Ingesting as the set of stand-alone files involves identifies all relevant multi-line patterns within the log files to ensure each log message is accurately recognized and processed. Once these patterns are identified, each log message is ingested as though it were received through the ingestion API service. Ingesting via the client and the ingestion API support real time ingestion. The client continuously monitors the application’s log file, reading each log message as it is updated. Since this approach captures logs as they are generated, there is no need for identifying individual log messages. This approach includes a robust API capable of ingesting one or more log messages simultaneously.

[0058] The method (300) includes detecting, by a pattern discovery module, structures and characteristics of log messages pre-processed by the preparation module using a plurality of methods in step in step 310. The plurality of methods includes the predefined global pattern, the static first approach, the variable first approach and the machine learning based approach.

[0059] The method (300) includes employing, by the pattern discovery module, a predefined global pattern on the log data. The predefined global pattern includes common log formats, date patterns, IP address patterns, uniform resource locator patterns, uniform resource identifier patterns, numerical patterns, alphanumeric, and field enclosures in step 315. The common log formats include Apache WS pattern. IP address patterns include patterns for both IPv4 and IPv6 addresses, which are essential for tracking network-related events and activities. The URL and URI patterns identify and extract URLs and URIs from log messages. The alphanumeric patterns identify alphanumeric from log messages. The field enclosures pattern identifying fields inside enclosures like (), { }, [], and the like.

[0060] The method (300) includes identifying, by the pattern discovery module, a plurality of exception patterns in log data by applying a general-purpose pattern in step 320. The general-purpose pattern is particularly useful for multi-line exceptions. The general- purpose pattern serves as a first filter to capture exception-related information. Once the general-purpose pattern is applied, the pattern discovery module lists all exception names found in the log data and gets the corresponding exception messages. The exception messages, along with their static components, are then used as identifiers for further pattern discovery using advanced approaches. The output of the general- purpose pattern includes a list of exception names and a list of exception messages, providing a detailed understanding of the exception patterns present in the log data.

[0061] The method (300) includes employing, by the pattern discovery module, at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages in step 325. In the static first approach, the pattern discovery module tokenizing log messages based on whitespace characters. Further categorizing tokens as static, variable, or likely variable. A token is categorized as static, if all its characters are of alphabetic characters or one that is found in a well-known English dictionary, such as the Apache Lucene dictionary. Consecutive non-static tokens are merged, and predefined global patterns are applied recursively to identify static and variable components. Tokens matching the predefined pattern are marked as variables, while those that do not match any patterns are marked as likely variables. The final output includes a list of static tokens, a list of variables (with corresponding patterns), and a list of likely variable tokens, all maintained in their original order.

[0062] In the variable first approach, the pattern discovery module first applies the predefined patterns to the log message in order of relevance, marking segments that match with a specific pattern name, such as $$PATTERN_<NAME>$$. The message is then tokenized based on whitespace characters, with tokens being classified as variables if they match a pattern. Further, those that have matched a pattern, if found to be static mark as static else mark as likely variable. Finally, providing an output similar to the static-first approach.

[0063] In the machine learning based approach, the pattern discovery module extracts only the static tokens from the log message and generates a Message Key by hashing a static part. The log messages are grouped based on either the same Message Key or a similarity score that meets a predefined threshold based on an ensemble of similarity algorithms. Within the groups, the pattern discovery module applies clustering algorithms to group all similar phrases, followed by applying different patterns to each cluster and selecting the top patterns that match. By default, select the top matching pattern. The output of the machine learning based approach includes a list of the top recommended patterns. At the end of processes in the pattern discovery module, a list of ordered tokens that are marked as static, variable or likely variable are generated.

[0064] The method (300) includes creating, by a patterned-message structure module, a structured format for each log message in step 330.

[0065] The method (300) includes assigning, by the patterned-message structure module, an Application ID to each log message to identify the source application from which the log message originated in step 335.

[0066] The method (300) includes defining, by the patterned-message structure module, a pattern structure for log messages. The pattern structure includes pattern ID, pattern representing processing logic, and custom key names in step 340. The pattern representing processing logic typically a regex or a Class.

[0067] The method (300) includes assigning, by the patterned-message structure module, a Patterned-Message ID (PMID) to each structured log messages in step 345.

[0068] The method (300) includes maintaining, by the patterned-message structure module, a list of pattern IDs, each corresponding to a matched pattern within the log messages in step 350. The patterned-message structure module also separates the static message part (SMP), which consists of the log content without any variable parts. The method (300) includes computing, by the patterned-message structure module, a Secure Hash Algorithm (SHA) 2 hash of the Static Message Part to ensure data integrity and facilitate quick comparison in step 355.

[0069] The method (300) includes computing, by the patterned-message structure module, the Secure Hash Algorithm 2 hash of entire log message, comprising the static and patterned parts in step 360.

[0070] The method (300) includes retaining, by the patterned-message structure module, a Pattern ID for a binding variable within the log message in step 365. The binding variable is a variable within the log messages that can be used to link or group together multiple related log entries.

[0071] The method (300) includes processing, by a processing module, each log message formatted and structured by the patterned-message structure module in step 370. The processing starts with the list of tokens that are marked as static, variable and likely variable and the matching log messages.

[0072] The method (300) includes storing, by the processing module, the patterned-message structure in a structured data store (SCDS). The stored structure maintains a validation status that indicate whether the log message requires validation and is reviewed in step 375.

[0073] The method (300) includes tracking, by the processing module, a plurality of details on the processing. The plurality of details includes number of requests, sources of the requests, response times, trends, and number of active users in step 380.

[0074] The method (300) includes generating, by the processing module, comprehensive functional and operational health statistics for the application in real-time in step 385. The method (300) includes curating, by a message pattern curation module, and customize a plurality of patterned messages in step 390.

[0075] The method (300) includes displaying, by the message pattern curation module, the plurality of patterned messages with colour coding, to highlight variables and likely variables in step 395. Experts are then able to review and refine the identification of variables, marking them as valid, or reclassifying likely variables as either static or variable. Additionally, the message pattern curation module allows for the customization of variable names, such as labeling an IP address as "ClientIP" or a URL as " So-and-So-Service URL." In cases, if there are multiple patterns match a variable, the expert can select the most appropriate one, or if necessary, suggest a different set of patterns and retrigger the pattern discovery. Once this curation process is complete, the message pattern curation module finalizes the patterned messages as reviewed and confirmed, with updates to the patterned messages being versioned for future reference and learning.

[0076] The method (300) includes offering, by the message pattern curation module, a plurality of suggestions for potential solutions based on the analysis in step 400.

[0077] The method (300) includes organizing, by a log message grouping module, the log messages into structured groups based on identified patterns and binding variables in step 405.

[0078] The method (300) includes discovering, by the log message grouping module, binding variables and identify variables that span across multiple messages and link together in step 410.

[0079] The method (300) includes collecting, by the log message grouping module, messages in sequence in step 415. The method (300) includes applying, by the log message grouping module, patterns to categorize the collected messages in step 420.

[0080] The method (300) includes identifying and analyzing, by the log message grouping module, repeating patterns within message groups in step 425. For each group, determine how often these repeating patterns occur, which helps in identifying common behaviors or issues within the system. Develop a Group-Pattern for each set of repeating sequences, associating them with their respective occurrence counts. For each group pattern also find a subset or superset of Group-Pattern that have T% overlap. For every group-pattern maintain a similar Group-Pattern that’s either a subset or superset. Further, present a random set of “grouped” log messages and highlight each of the Group-Patterns with different color coding.

[0081] The method (300) includes allowing, by the log message grouping module, experts to review, adjust, and confirm groupings in step 430.

[0082] The method (300) includes storing, by the log message grouping module, message group with message group ID(MGID) in step 435.

[0083] The method (300) includes identifying and categorizing, by an error identification module, error sequences within the log messages of an application. The error indicates operational and security anomalies in step 440.

[0084] The method (300) includes collecting, by the error identification module, list of phrases that indicate errors in the application in step 445.

[0085] The method (300) includes checking, by the error identification module, each Group- Pattern for error phrases in step 450. As part of the curation process, each group-pattern presents similar group-patterns that are marked as error-patterns. Experts can then review and confirm both the similarity and whether the identified error pattern is correctly associated with the group-pattern under evaluation. The method (300) includes storing, by a log storage module, configurations, static log data and variable log content. The static log data stored a single time in step 455.

[0086] It must be noted that the store configurations, static log data and variable log content are stored in their respective storage systems: the config store, static log store, and variable log store. The config store holds configuration data, including identified patterns. The static log store is designed for storing static content with full-text search capabilities. The variable log store manages storage for variable content with capability to store and search similar kinds of data in an optimal manner. Since the log data is time stamped data, the variable log store is also optimized for time-based storage and searches. This data is further partitioned with time.

[0087] V arious embodiments of the system and method for analyzing log data for operational and security anomalies as described above offer a several advantages in log data analyzing. The system provides a comprehensive approach to managing vast quantities of log data by automating the identification of patterns, anomalies, and errors, thereby reducing the time and cost associated with issue resolution. By employing advanced pattern discovery techniques, including static-first, variable-first, and machine learning approaches, the system enhances the relevance of log analysis. Further, the system generates comprehensive functional and operational health statistics for applications in real-time, enabling quick identification of issues and timely decision-making. Furthermore, the system simplifies the curation, grouping, and storage of log data, making it easier for experts to review, customize, and validate patterns, leading to improved system performance, security and the like.

[0088] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components. The term “processor” or “processing subsystem” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit including hardware may also perform one or more of the techniques of this disclosure.

[0089] Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various techniques described in this disclosure. In addition, any of the described units, modules, or components may be implemented together or separately as discrete but interoperable logic devices. Depiction of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware, firmware, or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware, firmware, or software components, or integrated within common or separate hardware, firmware, or software components.

[0090] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the disclosure and are not intended to be restrictive thereof.

[0091] While specific language has been used to describe the disclosure, any limitations arising on account of the same are not intended. As would be apparent to a person skilled in the art, various working modifications may be made to the method in order to implement the inventive concept as taught herein.

[0092] The figures and the foregoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of processes described herein may be changed and are not limited to the manner described herein. Moreover, the actions of any flow diagram need not be implemented in the order shown; nor do all of the acts need to be necessarily performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of embodiments is by no means limited by these specific examples.

Claims

I CLAIM:

1. A system (100) for analyzing log data for operational and security anomalies: a processing subsystem (105) hosted on a server (108) and configured to execute on a network to control bidirectional communications among a plurality of modules, wherein the plurality of modules comprising: characterized in that, a preparation module (120) configured to prepare a log data for analysis by determining a required volume of log data needed, evaluating suitability of the log data from an environment test based on variance, and categorizing the log data; a pattern discovery module (125) operatively coupled to the preparation module (120), wherein the pattern discovery module (125) is configured to: detect structures and characteristics of log messages pre-processed by the preparation module (120) using a plurality of methods; employ a predefined global pattern on the log data, wherein the predefined global pattern comprises common log formats, date patterns, IP address patterns, uniform resource locator patterns, uniform resource identifier patterns, numerical patterns, alphanumeric, and field enclosures; identify a plurality of exception patterns in log data by applying a general-purpose pattern; and employ at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages;a patterned-message structure module (130) operatively coupled to the pattern discovery module (125), wherein the patterned-message structure module (130) is configured to: create a structured format for each log message; assign an application ID to each log message to identify the source application from which the log message originated; define a pattern structure for log messages, wherein the pattern structure comprises pattern ID, pattern representing processing logic, and custom key names; assign a patterned-message ID to each structured log messages; maintain a list of pattern IDs, each corresponding to a matched pattern within the log messages; compute a secure hash algorithm 2 hash of the static message part to ensure data integrity and facilitate quick comparison; compute the secure hash algorithm 2 hash of entire log message, comprising the static and patterned parts; and retain a pattern ID for a binding variable within the log message; a processing module (135) operatively coupled to the patterned-message structure module (130), wherein the processing module (135) is configured to: process each log message formatted and structured by the patterned- message structure module (130); store the patterned-message structure in a structured data store, wherein the stored structure maintains a validation status that indicate whether the log message requires validation and review;track a plurality of details on the processing, wherein the plurality of details comprises number of requests, sources of the requests, response times, trends, and number of active users; and generate comprehensive functional and operational health statistics for the application in real-time; a message pattern curation module (140) operatively coupled to the processing module (135), wherein the message pattern curation module (140) is configured to: curate and customize a plurality of patterned messages; display the plurality of patterned messages with colour coding, to highlight variables and likely variables; and offer a plurality of suggestions for potential solutions based on the analysis; a log message grouping module (145) operatively coupled to the message pattern curation module (140), wherein the log message grouping module (145) is configured to: organize the log messages into structured groups based on identified patterns and binding variables; discover binding variables and identify variables that span across multiple messages and link together; collect messages in sequence; apply patterns to categorize the collected messages; identify and analyze repeating patterns within message groups; allow experts to review, adjust, and confirm groupings; andstore message group with message group ID; an error identification module (150) operatively coupled to the log message grouping module (145), wherein the error identification module (150) is configured to: identify and categorize error sequences within the log messages of an application, wherein the error indicates operational and security anomalies; collect list of phrases that indicate errors in the application; and check each Group-Pattern for error phrases; and a log storage module (155) is operatively coupled to the error identification module (150), wherein the log storage module (155) is configured to store configurations, static log data and variable log content, wherein the static log data stored a single time.

2. The system (100) as claimed in claim 1, comprising an ingestion module operatively coupled to the preparation module (120), wherein the ingestion module is configured to input the log data from a plurality of sources for delivering to the preparation module (120), wherein the log data comprises application logs, operational logs, and security logs.

3. The system (100) as claimed in claim 2, wherein the plurality of sources comprises existing log files, log files generated in real-time by the application.

4. The system (100) as claimed in claim 1, wherein the plurality of exception patterns comprises list of exception names and list of exception messages.

5. The system (100) as claimed in claim 1, wherein the plurality of methods comprises the predefined global pattern, the static first approach, the variable first approach and the machine learning based approach.

6. The system (100) as claimed in claim 1, wherein the static first approach comprises: tokenizing the entire log message based on whitespace characters to segment the text into tokens; marking a token as static; merging consecutive non-static tokens to create larger variable segments; recursively applying the predefined global patterns based on the tokens that is matched; marking tokens that match a predefined pattern as variables; marking tokens that do not match the predefined pattern as likely variables; and ensuring an order of tokens is maintained.

7. The system (100) as claimed in claim 1, wherein the variable first approach comprises: applying the predefined patterns in order of relevance; marking segments that match the predefined pattern with a pattern name; tokenizing the log message based on whitespace characters; and categorizing tokens as variable, static, and likely variable.

8. The system (100) as claimed in claim 1, wherein the machine learning based approach comprises: extracting static tokens from the log messages;generating a message key through hashing; grouping the log messages based on the Message Key and similarity scores; applying an ensemble of clustering algorithms to identify and group similar phrases; applying the predefined patterns to clusters; selecting relevant patterns; and outputting a list of recommended patterns.

9. A method (300) for analyzing log data for operational and security anomalies: characterized in that, preparing, by a preparation module, a log data for analysis by determining a required volume of log data needed, evaluating suitability of the log data from an environment test based on variance, and categorizing the log data; (305) detecting, by a pattern discovery module, structures and characteristics of log messages pre-processed by the preparation module using a plurality of methods; (310) employing, by the pattern discovery module, a predefined global pattern on the log data, wherein the predefined global pattern comprises common log formats, date patterns, IP address patterns, uniform resource locator patterns, uniform resource identifier patterns, numerical patterns, alphanumeric, and field enclosures; (315) identifying, by the pattern discovery module, a plurality of exception patterns in log data by applying a general-purpose pattern; (320) employing, by the pattern discovery module, at least one of a static first approach, a variable first approach and a machine learning based approach on the log messages; (325) creating, by a patterned-message structure module, a structured format for each log message; (330)assigning, by the patterned-message structure module, an application ID to each log message to identify the source application from which the log message originated; (335) defining, by the patterned-message structure module, a pattern structure for log messages, wherein the pattern structure comprises pattern ID, pattern representing processing logic, and custom key names; (340) assigning, by the patterned-message structure module, a patterned-message ID to each structured log messages; (345) maintaining, by the patterned-message structure module, a list of pattern IDs, each corresponding to a matched pattern within the log messages; (350) computing, by the patterned-message structure module, a secure hash algorithm 2 hash of the static message part to ensure data integrity and facilitate quick comparison; (355) computing, by the patterned-message structure module, the Secure Hash Algorithm 2 hash of entire log message, comprising the static and patterned parts; (360) retaining, by the patterned-message structure module, a Pattern ID for a binding variable within the log message; (365) processing, by a processing module, each log message formatted and structured by the patterned-message structure module; (370) storing, by the processing module, the patterned-message structure in a structured data store, wherein the stored structure maintains a validation status that indicate whether the log message requires validation and review; (375) tracking, by the processing module, a plurality of details on the processing, wherein the plurality of details comprises number of requests, sources of the requests, response times, trends, and number of active users; (380)generating, by the processing module, comprehensive functional and operational health statistics for the application in real-time; (385) curating, by a message pattern curation module, and customize a plurality of patterned messages; (390) displaying, by the message pattern curation module, the plurality of patterned messages with colour coding, to highlight variables and likely variables; (395) offering, by the message pattern curation module, a plurality of suggestions for potential solutions based on the analysis; (400) organizing, by a log message grouping module, the log messages into structured groups based on identified patterns and binding variables; (405) discovering, by the log message grouping module, binding variables and identify variables that span across multiple messages and link together; (410) collecting, by the log message grouping module, messages in sequence; (415) applying, by the log message grouping module, patterns to categorize the collected messages; (420) identifying and analyzing, by the log message grouping module, repeating patterns within message groups; (425) allowing, by the log message grouping module, experts to review, adjust, and confirm groupings; (430) storing, by the log message grouping module, message group with message group ID; (435) identifying and categorizing, by an error identification module, error sequences within the log messages of an application, wherein the error indicates operational and security anomalies; (440)collecting, by the error identification module, list of phrases that indicate errors in the application; (445) checking, by the error identification module, each Group-Pattern for error phrases; and (450) storing, by a log storage module, configurations, static log data and variable log content, wherein the static log data stored a single time. (455)

Images